A low-altitude, slow-moving, small target tracking method integrating millimeter-wave radar and visual sensor

Abstract

The present invention belongs to the field of machine vision technology, and specifically relates to a low-altitude, slow, small target tracking method that integrates millimeter-wave radar and visual sensor. The method comprises: using millimeter-wave radar scanning to obtain target information, and filtering the obtained target information to obtain filtered target information; using a visual sensor to obtain an optical image as a search image, mapping the filtered target information to the optical image, and then generating a Gaussian image to obtain a heat map (Heat Map) containing target position information; constructing a twin network target tracking model, intercepting the first frame image with a millimeter-wave radar-marked target position in the optical image as a template image, fusing the search image and the radar heat map through a spatial attention mechanism, and using the twin network target tracking model to achieve stable tracking of the target; the present invention achieves reliable target tracking by fusing millimeter-wave radar and visual sensor data.

Description

Technical Field

[0001] The present invention belongs to the technical field of machine vision, and in particular relates to a low-altitude, slow-speed, small target tracking method integrating a millimeter-wave radar and a visual sensor. Background Art

[0002] Low-altitude, small, and slow targets are a general term for small aircraft and low-altitude floating objects that have the characteristics of flying at low altitudes, slow speeds, and being difficult to detect and discover. The low-altitude, small, and slow targets currently seen mainly include aviation models, civilian drones, and hot air balloons. Due to their small size, simple operation, low flying altitude, and many obstructions from the ground, low-altitude, and slow targets cannot be covered by the Air Force and large radar equipment. Preventing and handling the interference and sabotage of low-altitude, small, and slow targets has become a global problem for major security activities, which is particularly difficult to control, detect, and handle.

[0003] In the existing technology, radar monitoring and video monitoring are mainly used for the detection and tracking of low, small and slow targets. Among them, radar monitoring mainly uses millimeter wave radar detection. Millimeter wave radar has the advantages of all-weather, all-day, long-distance, and easy integration. It has the ability to detect, locate and track targets, but its recognition ability of targets is poor and it cannot obtain information such as the visual texture of the target; the video detection method mainly uses visual sensors to detect, identify and track targets. Visual sensors have high detection accuracy, high resolution, high visualization, and can obtain the shape, texture and environmental information of the target. It is easy to identify the target, but its ranging and speed measurement capabilities are poor, the field of view is small, and it is greatly affected by the environment. At the same time, it is not as sensitive and fast as radar. Therefore, how to organically combine the two, complement each other's advantages, and achieve efficient low, small and slow target detection is an urgent problem to be solved. Summary of the invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the present invention proposes a low-altitude, slow-speed, small target tracking method that integrates millimeter-wave radar and visual sensor, the method comprising: using millimeter-wave radar scanning to obtain target information, and filtering the obtained target information to obtain filtered target information; using a visual sensor to obtain an optical image of the target, and using the optical image as a search image; mapping the filtered target information to the optical image, and performing Gaussian filtering on the mapped optical image to obtain a radar heat map containing target position information; constructing a twin network target tracking model; intercepting the first frame image with the millimeter-wave radar marked target position in the optical image as a template image; inputting the search image, the radar heat map and the template image into the twin network target tracking model to track the target, and obtain the final information of the target.

[0005] Preferably, the target information acquired by the millimeter wave radar scanning includes the target's azimuth, pitch angle, distance, speed, and target type.

[0006] Preferably, the process of mapping the filtered target information to the optical image includes: constructing a mapping rule, wherein the mapping rule includes: converting millimeter-wave radar coordinates into world coordinates, converting world coordinates into camera coordinates, converting camera coordinates into image coordinates, and converting image coordinates into pixel coordinates; mapping the filtered target information into the optical image according to the mapping rule to obtain an optical image with the target information.

[0007] Furthermore, the process of converting the millimeter wave radar coordinates into the world coordinates includes: the origin of the millimeter wave radar coordinates is located at the center of the millimeter wave radar device, and the offset matrix of the position of the millimeter wave radar relative to the world coordinates is set to M r , the millimeter wave radar obtains the target P position information as Where r is the distance, θ is the pitch angle, represents the azimuth; the coordinates of the target P in the millimeter wave radar coordinate system are calculated by trigonometric functions (X r ,Y r ,Z r ); Convert the target position from the millimeter wave radar coordinates to the world coordinates (X w ,Y w ,Z w ).

[0008] Preferably, the process of converting the world coordinates into the camera coordinates includes: the origin of the camera coordinates is located at the center of the camera optical lens, and the offset matrix of the camera position relative to the world coordinates is assumed to be M c , the position of the target P in the world coordinate is (X w ,Y w ,Z w ), transform the target position from world coordinates to camera coordinates (X c ,Y c ,Z c ).

[0009] Preferably, the process of converting the camera coordinates into the image coordinates includes: obtaining the camera position coordinates (X c ,Y c ,Z c ), according to the triangle similarity principle, the camera position coordinates are converted into image coordinates. The conversion formula is:

[0010]

[0011] Where f represents the focal length of the camera lens, x represents the horizontal coordinate of the image coordinate system, and y represents the vertical coordinate of the image coordinate system. c ,Y c ,Z c ) represents the camera position coordinates.

[0012] Preferably, the process of converting the image coordinates into pixel coordinates includes: assuming that the unit length of each pixel on the x-axis and the y-axis of the image coordinate system is d x and d y , the pixel coordinates corresponding to the origin of the image coordinate system (u 0 ,v 0 ); construct the transformation relationship between a point (u, v) in the pixel coordinate system and a position (x, y) in the image coordinate system according to the set parameters; convert the image coordinates into pixel coordinates according to the transformation relationship, and the transformation relationship includes:

[0013]

[0014] Among them, d x Represents the unit length of the pixel on the x-axis of the image coordinate system, d y Represents the unit length of the pixel on the y-axis of the image coordinate system, (u 0 ,v 0 ) represents the pixel coordinates corresponding to the origin of the image coordinate system.

[0015] Preferably, the process of generating a Gaussian image of the millimeter-wave radar target information mapped onto the optical image includes: the target position coordinates are (u, v), and a heat map containing the target position information is calculated by a Gaussian kernel formula; the calculation expression is:

[0016]

[0017] Among them, x and y represent the coordinates of a point in the image, σ p is the target scale adaptive variance, σ p It is linearly proportional to r, where r represents the target distance. xy Represents the label value of each point in the heat map.

[0018] Preferably, the twin network target tracking model includes a feature extraction backbone network, which contains five convolutional layers and two pooling layers; wherein, the convolution kernel of the first convolutional layer is 11×11, and the step size is 2; the pooling window of the first pooling layer is 3×3, and the step size is 2; the convolution kernel of the second convolutional layer is 5×5, and the step size is 1; the pooling window of the second pooling layer is 3×3, and the step size is 2; the convolution kernel of the third convolutional layer is 3×3, and the step size is 1; the convolution kernel of the fourth convolutional layer is 3×3, and the step size is 1; the convolution kernel of the fifth convolutional layer is 3×3, and the step size is 1.

[0019] Preferably, the process of constructing a fusion twin network target tracking model and inputting the template image, the search image and the radar heat map into the twin network target tracking model for tracking includes:

[0020] Step 1: In the first frame of the visual sensor, take the target position obtained by the millimeter-wave radar as the center, cut out a 127×127 image as the template image z, and pass it through the backbone network Extract features and get template feature map

[0021] Step 2: Crop the image acquired by the subsequent visual sensor to a size of 255×255 as the search image x and pass it through the backbone network Extract features and obtain search feature graph

[0022] Step 3: Search feature map Perform global maximum pooling and global average pooling;

[0023] Step 4: Downsample the radar heat map, and concatenate the downsampled radar heat map and the pooled search feature map to obtain a first feature map of 22×22×3;

[0024] Step 5: Use a convolution layer to perform convolution processing on the first feature map, so that the number of channels of the first feature map is compressed to 1; use the spatial attention mechanism to process the first feature map after channel compression to obtain a feature map containing spatial attention information;

[0025] Step 6: Multiply the search feature map with the feature map containing spatial attention to obtain the enhanced search feature map

[0026] Step 7: Template feature map With enhanced search feature map Perform cross-correlation operation with to obtain the response graph;

[0027] Step 8: Upsample the response map and obtain the response value in the upsampled response map, where the position with the largest response value is the target position.

[0028] Beneficial effects of the present invention:

[0029] The present invention realizes reliable target tracking by fusing millimeter-wave radar and visual sensor data; the present invention obtains the initial position of the target through the millimeter-wave radar, fuses the acquired position information with the visual sensor image, and combines the spatial attention mechanism to enhance the stability and accuracy of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the low-altitude, slow-speed, small target detection method integrating millimeter-wave radar and visual sensor of the present invention;

[0031] Figure 2This is a flowchart of the present invention using a twin network target tracking model to fuse millimeter wave radar and visual sensor data for processing;

[0032] Figure 3 Schematic diagram of deformable convolution of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In order to solve the problem that a single device of millimeter-wave radar and visual sensor is difficult to efficiently track low, small and slow targets, the present invention provides a low, small and slow target tracking method that integrates millimeter-wave radar and visual sensor, and uses millimeter-wave radar data to assist visual sensor to achieve stable tracking of low, small and slow targets.

[0035] A low-altitude, slow-moving, small target detection method that integrates millimeter-wave radar and visual sensors, such as Figure 1 As shown, the method includes: using millimeter-wave radar scanning to acquire target information, and filtering the acquired target information to obtain filtered target information; using a visual sensor to acquire an optical image as a search image, mapping the filtered target information to the optical image, and then generating a Gaussian image to obtain a heat map containing target position information; constructing a twin network target tracking model, intercepting the first frame image with the millimeter-wave radar marked target position in the optical image as a template image, fusing the search image and the radar heat map through a spatial attention mechanism, and using the twin network target tracking model to achieve stable tracking of the target.

[0036] The target information obtained by millimeter-wave radar scanning includes the target's azimuth, pitch angle, distance, speed and target type.

[0037] The process of mapping target information using visual sensors includes:

[0038] Step 1: Construct the millimeter-wave radar coordinate system and the world coordinate system, obtain the position coordinates of the target in the millimeter-wave radar coordinate system according to the target information obtained by the millimeter-wave radar scanning, and convert the position coordinates in the millimeter-wave radar coordinate system into the world coordinate system;

[0039] Step 2: Construct the visual sensor coordinate system and transform the target position coordinates in the world coordinate system into the visual sensor coordinate system;

[0040] Step 3: Construct the image coordinate system and transform the target position coordinates in the visual sensor coordinate system into the image coordinate system;

[0041] Step 4: Construct a pixel coordinate system and convert the target position coordinates in the image coordinate system into the pixel coordinate system to obtain image information.

[0042] Specifically, a visual sensor is used to acquire an optical image, and the information of the millimeter-wave radar is mapped into the optical image. The mapping process includes:

[0043] Step 1: Convert the millimeter wave radar coordinates to world coordinates.

[0044] The origin of the millimeter-wave radar coordinates is located at the center of the millimeter-wave radar device. Let the offset matrix of the millimeter-wave radar position relative to the world coordinates be M r , the millimeter wave radar obtains the target P position information as Where r is the distance, θ is the pitch angle, Indicates azimuth;

[0045] The coordinates (X r ,Y r ,Z r ):

[0046]

[0047] Convert the target position from the millimeter wave radar coordinates to the world coordinates (X w ,Y w ,Z w ):

[0048]

[0049] Step 2: Convert world coordinates to camera coordinates.

[0050] The origin of the camera coordinates is located at the center of the camera optical lens. Let the offset matrix of the camera position relative to the world coordinates be M c , the position of the target P in the camera coordinate is (X c ,Y c ,Z c ), the position in the world coordinate is (X w ,Y w ,Z w ), there are the following relations:

[0051]

[0052] Step 3: Convert camera coordinates to image coordinates.

[0053] The image coordinate system xOy is a two-dimensional plane coordinate system, which is the generation position of the two-dimensional image. Its coordinate plane is parallel to the center plane of the camera lens, and the distance from the center plane of the lens is the focal length f of the lens.

[0054] According to the camera imaging principle, the position of a point P(x,y) in the image coordinate system in the camera coordinate system is (X c ,Y c ,Z c ), according to the triangular similarity principle:

[0055]

[0056] It can be expressed in matrix form as:

[0057]

[0058] Step 4: Convert image coordinates to pixel coordinates.

[0059] A photo is composed of pixels, and the pixel coordinate system is uO 0 The origin of v is usually in the upper left corner of the image, the u axis extends from the origin to the right along the image boundary and is parallel to the x axis of the image coordinate system, and the v axis extends downward and is parallel to the y axis of the image coordinate system. Let the unit length of each pixel on the x axis and y axis of the image coordinate system be d x and d y , the pixel coordinates corresponding to the origin of the image coordinate system (u 0 ,v 0 ), thereby establishing the transformation relationship between a point (u,v) in the pixel coordinate system and a position (x,y) in the image coordinate system:

[0060]

[0061] It can be expressed in matrix form as:

[0062]

[0063] Combining steps 2 to 4, we get the conversion model between pixel coordinates and world coordinates:

[0064]

[0065]

[0066] Step 5: Use world coordinates as a medium to map the millimeter-wave radar coordinates to pixel coordinates.

[0067] The process of generating a Gaussian image of the millimeter-wave radar target information mapped onto the optical image includes: the target position coordinates are (u, v), and a heat map containing the target position information is calculated by the Gaussian kernel formula; the calculation expression is:

[0068]

[0069] Among them, x and y represent the coordinates of a point in the image, and the target scale adaptive variance σ p The distance r is calculated by converting the millimeter-wave radar coordinates into world coordinates, σ p It is linearly proportional to r, label xy Represents the label value of each point in the heat map.

[0070] The twin network target tracking model includes a feature extraction backbone network, which contains five convolutional layers and two pooling layers; among them, the convolution kernel of the first convolutional layer is 11×11, and the step size is 2; the pooling window of the first pooling layer is 3×3, and the step size is 2; the convolution kernel of the second convolutional layer is 5×5, and the step size is 1; the pooling window of the second pooling layer is 3×3, and the step size is 2; the convolution kernel of the third convolutional layer is 3×3, and the step size is 1; the convolution kernel of the fourth convolutional layer is 3×3, and the step size is 1; the convolution kernel of the fifth convolutional layer is 3×3, and the step size is 1. Its structure is shown in the table:

[0071] layer Convolution kernel / pooling window Step Length Conv1 11×11 2 Pool1 3×3 2 Conv2 5×5 1 Pool2 3×3 2 Conv3 3×3 1 Conv4 3×3 1 Conv5 3×3 1

[0072] like Figure 2 As shown in the figure, a twin network target tracking model is constructed, which integrates visual sensor and millimeter wave radar data and uses the spatial attention mechanism to stably track the target. The specific process includes:

[0073] Step 1: In the first frame of the visual sensor, take the target position obtained by the millimeter-wave radar as the center, cut out a 127×127 image as the template image z, and pass it through the backbone network Extract features and get template feature map

[0074] Step 2: Crop the image acquired by the subsequent visual sensor to a size of 255×255 as the search image x and pass it through the backbone network Extract features and obtain search feature graph

[0075] Step 3: Search feature map Perform global maximum pooling and global average pooling, compress the number of channels, downsample the radar heat map, and then cascade the downsampled radar heat map and the pooled search feature map to obtain a 22×22×3 feature map. Then, use a layer of convolution to compress the number of channels to 1, and then activate it through the Sigmoid function to obtain a feature map containing spatial attention information.

[0076] Step 4: Multiply the search feature map with the feature map containing spatial attention to obtain the enhanced search feature map

[0077] Step 5: Convert the template feature map With enhanced search feature map Perform cross-correlation operation with to obtain the response graph.

[0078] The template feature map With enhanced search feature map The specific process of performing cross-correlation operations includes: Figure 3 As shown, firstly, the enhanced search feature map The purpose of deformable convolution is to enhance the network's ability to adapt to target deformation. The length and width of the convolution kernel are 6×6. The coordinate offset of the search feature map (OffsetMap, 64×22×22) is obtained through deformable convolution. The pixel index value in the image is added to the coordinate offset to obtain the offset pixel coordinate position. In this way, an offset is added to each convolution sampling point, so that the convolution kernel can adaptively adjust its own shape; then the pixel value of each coordinate point is calculated by bilinear interpolation, and back propagation is realized at the same time. Finally, the template feature map is the convolution kernel, and the enhanced search feature map after offset Perform cross-correlation operation to obtain the response diagram. The cross-correlation operation formula is:

[0079]

[0080] Among them, z represents the template image, x represents the search image, and * represents the convolution operation.

[0081] Step 6: Upsample the response map, where the position with the largest response value is the target position.

[0082] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A low-altitude, slow-speed, small target tracking method integrating millimeter-wave radar and visual sensor, characterized in that: The method includes: using millimeter wave radar scanning to obtain target information, and filtering the obtained target information to obtain filtered target information; using a visual sensor to obtain an optical image of the target, and using the optical image as a search image; mapping the filtered target information to the optical image, and performing Gaussian filtering on the mapped optical image to obtain a radar thermal map containing target position information; constructing a twin network target tracking model; intercepting the first frame image with a millimeter wave radar marked target position in the optical image as a template image; inputting the search image, the radar thermal map and the template image into the twin network target tracking model to track the target and obtain the final information of the target; The process of mapping the filtered target information to the optical image includes: constructing a mapping rule, wherein the mapping rule includes: converting the millimeter wave radar coordinates into world coordinates, converting the world coordinates into camera coordinates, converting the camera coordinates into image coordinates, and converting the image coordinates into pixel coordinates; mapping the filtered target information to the optical image according to the mapping rule, and then generating a Gaussian image to obtain a thermal map containing the target position information; The process of building a fusion twin network target tracking model and inputting the template image, search image and radar heat map into the twin network target tracking model for tracking includes: Step 1: In the first frame of the visual sensor, take the target position obtained by the millimeter-wave radar as the center, cut out a 127×127 image as the template image z, and pass it through the backbone network Extract features and get template feature map Step 2: Crop the image acquired by the subsequent visual sensor to a size of 255×255 as the search image x and pass it through the backbone network Extract features and obtain search feature graph Step 3: Search feature map Perform global maximum pooling and global average pooling; Step 4: Downsample the radar heat map, and concatenate the downsampled radar heat map and the pooled search feature map to obtain a first feature map of 22×22×3; Step 5: Use a convolution layer to perform convolution processing on the first feature map, so that the number of channels of the first feature map is compressed to 1; use the spatial attention mechanism to process the first feature map after channel compression to obtain a feature map containing spatial attention information; Step 6: Multiply the search feature map with the feature map containing spatial attention to obtain the enhanced search feature map Step 7: Convert the template feature map With enhanced search feature map Perform cross-correlation operation with to obtain the response graph; Step 8: Upsample the response map and obtain the response value in the upsampled response map, where the position with the largest response value is the target position.

2. According to the method for tracking a low-altitude, slow, small target by integrating a millimeter-wave radar and a visual sensor according to claim 1, it is characterized in that: The target information obtained by millimeter-wave radar scanning includes the target's azimuth, pitch angle, distance, speed and target type.

3. According to the method for tracking a low-altitude, slow, small target by integrating a millimeter-wave radar and a visual sensor according to claim 1, it is characterized in that: The process of converting the millimeter wave radar coordinates into the world coordinates includes: the origin of the millimeter wave radar coordinates is located at the center of the millimeter wave radar device, and the offset matrix of the millimeter wave radar position relative to the world coordinates is set to M r , the millimeter wave radar obtains the target P position information as Where r is the distance, θ is the pitch angle, represents the azimuth; the coordinates of the target P in the millimeter wave radar coordinate system are calculated by trigonometric functions (X r ,Y r ,Z r ); Convert the target position from the millimeter wave radar coordinates to the world coordinates (X w ,Y w ,Z w ).

4. The low-altitude, slow-speed, small target tracking method integrating millimeter-wave radar and visual sensor according to claim 1 is characterized in that: The process of converting world coordinates to camera coordinates includes: the origin of the camera coordinates is located at the center of the camera optical lens, and the offset matrix of the camera position relative to the world coordinates is set to M c , the position of the target P in the world coordinate is (X w ,Y w ,Z w ), transform the target position from world coordinates to camera coordinates (X c ,Y c ,Z c ).

5. The low-altitude, slow-speed, small target tracking method integrating millimeter-wave radar and visual sensor according to claim 1 is characterized in that: The process of converting camera coordinates into image coordinates includes: obtaining the camera position coordinates (X c ,Y c ,Z c ), according to the triangle similarity principle, the camera position coordinates are converted into image coordinates. The conversion formula is: Where f represents the focal length of the camera lens, x represents the horizontal coordinate of the image coordinate system, and y represents the vertical coordinate of the image coordinate system. c ,Y c ,Z c ) represents the camera position coordinates.

6. The low-altitude, slow-speed, small target tracking method integrating millimeter-wave radar and visual sensor according to claim 1 is characterized in that: The process of converting image coordinates into pixel coordinates includes: assuming that the unit length of each pixel on the x-axis and y-axis of the image coordinate system is d x and d y , the pixel coordinates (u0, v0) corresponding to the origin of the image coordinate system; construct the transformation relationship between a point (u, v) in the pixel coordinate system and the position (x, y) in the image coordinate system according to the set parameters; convert the image coordinates into pixel coordinates according to the transformation relationship, and the transformation relationship includes: Among them, d x Represents the unit length of the pixel on the x-axis of the image coordinate system, d y It represents the unit length of the pixel on the y-axis of the image coordinate system, and (u0, v0) represents the pixel coordinates corresponding to the origin of the image coordinate system.

7. The low-altitude, slow-speed, small-sized target tracking method integrating millimeter-wave radar and visual sensor according to claim 3 is characterized in that: The process of generating a Gaussian image of the millimeter-wave radar target information mapped to the optical image includes: the target position coordinates are (u, v), and the Gaussian kernel formula is used Calculate and generate a heat map containing target location information; where x and y represent the coordinates of a point in the image, and the target scale adaptive variance σ p Calculated by the distance r, σ p It is linearly proportional to r, label xy Represents the label value of each point in the heat map.

8. The low-altitude, slow-speed, small-sized target tracking method integrating millimeter-wave radar and visual sensor according to claim 1 is characterized in that: The Siamese network target tracking model includes a feature extraction backbone network The network contains five convolutional layers and two pooling layers; among them, the convolution kernel of the first convolutional layer is 11×11, and the step size is 2; the pooling window of the first pooling layer is 3×3, and the step size is 2; the convolution kernel of the second convolutional layer is 5×5, and the step size is 1; the pooling window of the second pooling layer is 3×3, and the step size is 2; the convolution kernel of the third convolutional layer is 3×3, and the step size is 1; the convolution kernel of the fourth convolutional layer is 3×3, and the step size is 1; the convolution kernel of the fifth convolutional layer is 3×3, and the step size is 1.

Images