Water surface target tracking method based on 4D millimeter wave radar and unmanned aerial vehicle remote sensing image fusion, electronic equipment and storage medium
By fusing 4D millimeter-wave radar and UAV remote sensing images, and using a Gaussian mixture model and an improved Faster-RCNN network, the problems of high computing power requirements, high latency, and low fault tolerance in 4D millimeter-wave radar target tracking in complex environments were solved, achieving stable surface target tracking.
Patent Information
- Application Number
- CN202510812375.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-18
- Publication Date
- 2025-10-03
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In existing technologies, 4D millimeter-wave radar is prone to false alarms or missed detections in complex environments. It has high computing power requirements, high latency, low fault tolerance, a single data source, and poor anti-interference capabilities, resulting in poor target tracking effects.
Combining 4D millimeter-wave radar and UAV remote sensing images, through alignment and filling processing, using a Gaussian mixture model for dynamic foreground extraction and noise filtering, optimizing the foreground image, combining with an improved Faster-RCNN network for target recognition, eliminating background point clouds, and fusing the tracked object motion data for target tracking.
It achieves stable tracking of surface targets on low-computing power devices, reduces latency, improves fault tolerance and anti-interference capabilities, optimizes data source utilization, and improves target recognition speed and accuracy.
Smart Images

Figure CN120742302A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target tracking technology, and in particular to a water surface target tracking method, electronic equipment, and storage medium that integrates 4D millimeter-wave radar and unmanned aerial vehicle remote sensing images. Background Art
[0002] Traditional millimeter-wave radar can only provide information on distance, direction, and speed, because it cannot identify the height of the target, which makes it easy to have false alarms or missed detections in complex environments. 4D millimeter-wave radar uses a multi-input and multi-output antenna array, combined with advanced beamforming technology, to accurately measure the elevation angle of the target, thereby achieving more comprehensive four-dimensional perception. The existing technology simultaneously predicts the detection frames of three frames before and after, and obtains the trajectory segment of the current target that lasts for three frames. Then, the distance intersection and union ratio score of the current trajectory segment and the trajectory tracking result of the previous moment is calculated, and the greedy algorithm is used to match and splice the current split trajectory segment with the historical trajectory segment to obtain the final trajectory tracking result at each moment. This method has the application potential to cope with multi-target tracking in sparse point clouds, and can still obtain stable tracking results in sparse point cloud sequence input.
[0003] However, existing technologies require a lot of computing power and are not suitable for devices with low computing power. In addition, too many model parameters lead to high tracking delays. When tracking targets, existing methods often directly use point cloud data for tracking. There is a lack of recognition process for the target object, which makes tracking failure easy to occur and has a low fault tolerance rate. In addition, the use of point cloud data alone is less effective. In order to meet the tracking requirements, the point cloud data needs to be processed in a complex way, which increases the computing power demand and delays. The utilization of the distance, direction and other data provided by the point cloud is low. In addition, there is too much noise in the outdoor environment. Directly using the collected data will lead to unstable model accuracy and poor tracking effect. Summary of the Invention
[0004] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a surface target tracking method, electronic equipment and storage medium that integrates 4D millimeter-wave radar and UAV remote sensing images to solve the problems of high computing power requirements, high latency, low fault tolerance, single data source and poor anti-interference ability in the existing technology.
[0005] To achieve the above object, the present invention provides the following solutions:
[0006] A surface target tracking method based on 4D millimeter-wave radar and UAV remote sensing image fusion, comprising:
[0007] The 4D millimeter-wave radar and multispectral camera attached to the data acquisition drone are used to collect the point cloud and remote sensing images of the target area.
[0008] Performing alignment and filling processing on the remote sensing images according to the historical movement data of the acquisition drone to obtain a virtual static acquisition image set;
[0009] Performing dynamic foreground extraction on the virtual static collected image set using the Gaussian mixture model to obtain an original foreground image;
[0010] performing noise filtering and contour optimization on the original foreground image to obtain an optimized foreground image;
[0011] Extracting radiation characteristics of the foreground area in the optimized foreground image within the remote sensing image to obtain foreground attribute data;
[0012] The optimized foreground image and the foreground attribute data are fused to obtain an image to be identified, and the target to be tracked is identified in a target recognition model pre-trained by an improved Faster-RCNN network to obtain a target area image and an image segmentation mask corresponding to the target area image;
[0013] Using the image segmentation mask to remove background point clouds from the point cloud to be analyzed to obtain an optimized point cloud;
[0014] Extracting movement data of each point cloud in the optimized point cloud to obtain motion data of the tracked object; the motion data of the tracked object includes: distance data, speed data, orientation data, and angular velocity data;
[0015] fusing the optimized point cloud with the motion data of the tracked object to obtain current tracked object state data, and using a pre-trained target position prediction model to predict the current tracked object state data to obtain predicted movement position data;
[0016] The target movement information of the collection drone is calculated using the position difference between the predicted movement position data and the current tracking object state data to obtain a flight control instruction, and the flight control instruction is used to control the collection drone to track the target to be tracked.
[0017] Preferably, a 4D millimeter wave radar and a multispectral camera attached to a data acquisition drone are used to collect a point cloud to be analyzed and a remote sensing image of the target area, including:
[0018] Fixing the 4D millimeter-wave radar and the multispectral camera on the bottom of the data acquisition drone;
[0019] Performing spatial coordinate system calibration and time synchronization calibration on the 4D millimeter-wave radar and the multispectral camera;
[0020] Setting the radar scanning range and point cloud density of the 4D millimeter-wave radar according to the range of the target area;
[0021] Preset multiple groups of exposure times for the multispectral camera, score the extraction effects of the original foreground image at different exposure times to obtain a foreground extraction score, and update the exposure time corresponding to the maximum value of the foreground extraction score to the multispectral camera; the foreground extraction score is calculated as follows: S = α1·P + α2·E; where S is the foreground extraction score; α1 and α2 are the first weight and the second weight, respectively; P is the foreground pixel accuracy; and E is the edge integrity under an ideal edge reference.
[0022] Preferably, the remote sensing images are aligned and filled according to the historical movement data of the acquisition drone to obtain a virtual static acquisition image set, including:
[0023] Determine the foreground image to be extracted;
[0024] According to the frame setting value, a plurality of frames before the foreground image to be extracted and a plurality of frames after the foreground image to be extracted are sequentially extracted to obtain a frame image set to be analyzed; the calculation formula of the frame setting value includes: n f =round(β·N) and n b =Nn f ; Where, N = min (T, F + B); n f 、n b are the number of frames extracted in the forward direction and the number of frames extracted in the backward direction, respectively; round(·) indicates the rounding operation; β is the division ratio; N is the actual number of frames extracted; min(·) indicates the minimum value operation; T is the preset number threshold; F indicates the number of images with a repetition rate greater than 80% in the forward adjacent frames; B indicates the number of images with a repetition rate greater than 80% in the backward adjacent frames;
[0025] Setting a mapping matrix for each frame image in the frame image set to be analyzed according to the historical movement data, and performing mapping processing on each frame image in the frame image set to be analyzed using the mapping matrix to obtain an original mapping frame image set;
[0026] Performing similarity matching on pixel points within a critical range of the left boundary and the right boundary of each frame image in the original mapping frame image set and the foreground image to be extracted to obtain an optimized mapping frame image set;
[0027] Each frame image in the optimized mapping frame image set is cropped according to the window of the foreground image to be extracted to obtain a cropped frame image set, and the missing area of each frame image in the cropped frame image set is filled with the foreground image to be extracted to obtain the virtual static acquisition image set.
[0028] Preferably, using the Gaussian mixture model to perform dynamic foreground extraction on the virtual static collected image set to obtain an original foreground image includes:
[0029] Calculating the matching degree between the pixel points of each frame image in the virtual static captured image set and each Gaussian distribution in the foreground image to be extracted to obtain matching degree data;
[0030] Using a preset Gaussian threshold to filter foreground pixels in the matching data to obtain a foreground mask image;
[0031] The foreground mask image is sequentially subjected to erosion processing, median filtering, and dilation processing to obtain the original foreground image.
[0032] Preferably, performing noise filtering and contour optimization on the original foreground image to obtain an optimized foreground image includes:
[0033] Determining adjacent foreground pixels in the original foreground image as being in the same area, and setting an area consisting of all foreground pixels in the same area as the foreground area;
[0034] Calculating the distance between any two foreground areas to obtain a regional pixel distance;
[0035] Eliminating the foreground area where the pixel distances of the area are all greater than a preset pixel distance threshold to obtain a denoised foreground image;
[0036] Contour data of each foreground area in the denoised foreground image is extracted, and the contour data is smoothed using a Canny operator to obtain the optimized foreground image.
[0037] Preferably, the optimized foreground image and the foreground attribute data are fused to obtain an image to be identified, and the target to be tracked is identified in a target recognition model pre-trained with an improved Faster-RCNN network to obtain a target area image and an image segmentation mask corresponding to the target area image, including:
[0038] Pre-collecting a plurality of groups of images to be identified and recognition results corresponding to the images to be identified to obtain a first training set;
[0039] Constructing the improved Faster-RCNN network; the improved Faster-RCNN network includes: an input layer, a MobileNetv3 network layer, a BiFPN network layer, a ROI generation network layer, an adaptive resolution network layer, a ROI feature extraction layer, a detection network layer, and an output layer connected in sequence;
[0040] Utilizing the input layer to receive the image to be recognized;
[0041] Using the MobileNetv3 network layer to perform feature extraction on the image to be identified to obtain feature extraction data;
[0042] Using the BiFPN network layer to perform upsampling and fusion processing on the image feature extraction data to obtain an original feature map;
[0043] Using the ROI generation network layer to set the foreground area in the original feature map as the ROI area;
[0044] Mapping the ROI region to a fixed resolution size using the adaptive resolution network layer to obtain an ROI feature map;
[0045] Using the ROI feature extraction layer to perform feature extraction and pooling processing on the ROI feature map to obtain image features;
[0046] Using the detection network layer to perform probability calculation on the image features to obtain a region recognition result;
[0047] Outputting the region recognition result using the output layer;
[0048] Calculating a binary cross entropy loss based on the first training set and the region recognition result to obtain a loss value;
[0049] The improved Faster-RCNN network is iteratively trained using the loss value to obtain the target recognition model.
[0050] Preferably, the optimized foreground image and the foreground attribute data are fused to obtain an image to be identified, and the target to be tracked is identified in a target recognition model pre-trained with an improved Faster-RCNN network to obtain a target area image and an image segmentation mask corresponding to the target area image, including:
[0051] Using the target recognition model to identify the image to be identified, to obtain the target area image;
[0052] The contour of the target area image is extracted to obtain a target area mask, and the inner pixel points of the target area mask are set to 1, and the outer pixel points of the target area mask are set to 0 to obtain the image segmentation mask.
[0053] Preferably, fusing the optimized point cloud and the tracked object motion data to obtain current tracked object state data, and using a pre-trained target position prediction model to predict the current tracked object state data to obtain predicted movement position data, includes:
[0054] Pre-collecting several sets of the current tracked object state data and future tracked object position data corresponding to the current tracked object state data to obtain a second training set;
[0055] Determining the optimized point cloud as a graph node, and determining the relationship between adjacent graph nodes as a graph edge, to obtain graph structure data;
[0056] constructing node features of the graph nodes and edge features of the graph edges according to the motion data of the tracked object;
[0057] Inputting the graph structure data into a dynamic graph neural network for prediction to obtain a network prediction result;
[0058] Calculating the Euclidean distance between the network prediction result and the future tracking object position data in the second training set to obtain a distance error;
[0059] The dynamic graph neural network is iteratively trained according to the distance error to obtain the target position prediction model.
[0060] Preferably, an electronic device comprises: at least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor can execute the aforementioned surface target tracking method that integrates 4D millimeter-wave radar and UAV remote sensing images.
[0061] Preferably, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the aforementioned surface target tracking method by fusing 4D millimeter-wave radar and UAV remote sensing images.
[0062] The present invention discloses the following technical effects:
[0063] The present invention provides a surface target tracking method, electronic device and storage medium that fuses 4D millimeter-wave radar and unmanned aerial vehicle remote sensing images. Through alignment and filling processing, the problem of the inability to directly apply a Gaussian mixture model due to camera movement is solved, and foreground extraction is achieved. Through dynamic foreground extraction, noise filtering and contour optimization, and radiation feature extraction, the problem of excessive model processing data and background interference is solved, and background interference data is eliminated and foreground image optimization is achieved. Through an improved Faster-RCNN network, the problem of excessive delay in traditional model recognition is solved, and the traditional Faster-RCNN network structure is optimized. Through background point cloud elimination and tracking object motion data extraction, the problems of high computing power requirements, single data source and poor anti-interference ability are solved, and point cloud optimization and multi-dimensional data fusion are achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0065] Figure 1 A schematic diagram of a water surface target tracking process using 4D millimeter-wave radar and UAV remote sensing image fusion provided by an embodiment of the present invention;
[0066] Figure 2 A schematic diagram of the data collection process provided by an embodiment of the present invention;
[0067] Figure 3 A schematic diagram of the data preprocessing process provided by an embodiment of the present invention;
[0068] Figure 4 A schematic diagram of a foreground extraction process according to an embodiment of the present invention;
[0069] Figure 5 A schematic diagram of the attribute data extraction process provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0071] The purpose of the present invention is to provide a surface target tracking method, electronic equipment and storage medium that integrates 4D millimeter-wave radar and UAV remote sensing imagery to solve the problems of high computing power requirements, high latency, low fault tolerance, single data source and poor anti-interference ability in the prior art.
[0072] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0073] Figure 1 The schematic diagram of the water surface target tracking process of the 4D millimeter wave radar and UAV remote sensing image fusion provided by the embodiment of the present invention is as follows Figure 1 As shown, the present invention provides a surface target tracking method by fusing 4D millimeter wave radar and UAV remote sensing images, comprising:
[0074] Step 100: Collecting a point cloud to be analyzed and a remote sensing image of the target area using a 4D millimeter wave radar and a multispectral camera attached to a data acquisition drone;
[0075] Step 200: performing alignment and filling processing on the remote sensing images according to the historical movement data of the acquisition drone to obtain a virtual static acquisition image set;
[0076] Step 300: performing dynamic foreground extraction on the virtual static captured image set using the Gaussian mixture model to obtain an original foreground image;
[0077] Step 400: performing noise filtering and contour optimization on the original foreground image to obtain an optimized foreground image;
[0078] Step 500: extracting radiation characteristics of the foreground area in the optimized foreground image within the remote sensing image to obtain foreground attribute data;
[0079] Step 600: Fusing the optimized foreground image and the foreground attribute data to obtain an image to be identified, and using the target recognition model pre-trained by the improved Faster-RCNN network to identify the target to be tracked, thereby obtaining a target area image and an image segmentation mask corresponding to the target area image;
[0080] Step 700: using the image segmentation mask to remove background point clouds from the point cloud to be analyzed to obtain an optimized point cloud;
[0081] Step 800: Extracting movement data of each point cloud in the optimized point cloud to obtain tracking object motion data; the tracking object motion data includes: distance data, speed data, orientation data, and angular velocity data;
[0082] Step 900: Fusing the optimized point cloud and the tracked object motion data to obtain current tracked object state data, and using a pre-trained target position prediction model to predict the current tracked object state data to obtain predicted movement position data;
[0083] Step 1000: Calculate the target movement information of the collection drone using the position difference between the predicted mobile position data and the current tracking object state data to obtain a flight control instruction, and use the flight control instruction to control the collection drone to track the target to be tracked.
[0084] refer to Figure 2 , using the 4D millimeter wave radar and multispectral camera attached to the data acquisition drone to collect the point cloud and remote sensing images of the target area to be analyzed, including:
[0085] Step 101: Fixing the 4D millimeter-wave radar and the multispectral camera on the bottom of the data acquisition drone;
[0086] Step 102: performing spatial coordinate system calibration and time synchronization calibration on the 4D millimeter wave radar and the multispectral camera;
[0087] Step 103: setting the radar scanning range and point cloud density of the 4D millimeter wave radar according to the range of the target area;
[0088] Step 104: Preset multiple exposure times for the multispectral camera, score the extraction effects of the original foreground image at different exposure times to obtain a foreground extraction score, and update the exposure time corresponding to the maximum foreground extraction score to the multispectral camera; the foreground extraction score is calculated as follows: S = α1·P + α2·E; where S is the foreground extraction score; α1 and α2 are the first weight and the second weight, respectively; P is the foreground pixel accuracy; and E is the edge integrity under an ideal edge reference.
[0089] refer to Figure 3 , performing alignment and filling processing on the remote sensing images according to the historical movement data of the acquisition drone to obtain a virtual static acquisition image set, including:
[0090] Step 201: Determine the foreground image to be extracted;
[0091] Step 202: extracting a plurality of frames before the foreground image to be extracted and a plurality of frames after the foreground image to be extracted in sequence according to the frame setting value to obtain a frame image set to be analyzed; the calculation formula of the frame setting value includes: n f =round(β·N) and n b =Nn f ; Where, N = min (T, F + B); n f 、n b are the number of frames extracted in the forward direction and the number of frames extracted in the backward direction, respectively; round(·) indicates the rounding operation; β is the division ratio; N is the actual number of frames extracted; min(·) indicates the minimum value operation; T is the preset number threshold; F indicates the number of images with a repetition rate greater than 80% in the forward adjacent frames; B indicates the number of images with a repetition rate greater than 80% in the backward adjacent frames;
[0092] Step 203: setting a mapping matrix for each frame image in the frame image set to be analyzed according to the historical movement data, and performing mapping processing on each frame image in the frame image set to be analyzed using the mapping matrix to obtain an original mapped frame image set;
[0093] Step 204: performing similarity matching between pixel points within a critical range of the left and right boundaries of each frame image in the original mapping frame image set and the foreground image to be extracted to obtain an optimized mapping frame image set;
[0094] Step 205: Crop each frame image in the optimized mapping frame image set according to the window of the foreground image to be extracted to obtain a cropped frame image set, and use the foreground image to be extracted to fill the missing area of each frame image in the cropped frame image set to obtain the virtual static acquisition image set.
[0095] refer to Figure 4 , using the Gaussian mixture model to perform dynamic foreground extraction on the virtual static collected image set to obtain an original foreground image, including:
[0096] Step 301: Calculating the matching degree between the pixel points of each frame image in the virtual static captured image set and each Gaussian distribution in the foreground image to be extracted to obtain matching degree data;
[0097] Step 302: Filter foreground pixels of the matching degree data using a preset Gaussian threshold to obtain a foreground mask image;
[0098] Step 303: performing corrosion processing, median filtering, and dilation processing on the foreground mask image in sequence to obtain the original foreground image.
[0099] refer to Figure 5 , performing noise filtering and contour optimization on the original foreground image to obtain an optimized foreground image, including:
[0100] Step 401: determining adjacent foreground pixels in the original foreground image to be in the same area, and setting the area consisting of all foreground pixels in the same area as the foreground area;
[0101] Step 402: Calculate the distance between any two foreground areas to obtain the area pixel distance;
[0102] Step 403: Eliminate the foreground area where the pixel distances of the area are all greater than a preset pixel distance threshold to obtain a denoised foreground image;
[0103] Step 404: extracting the contour data of each foreground area in the denoised foreground image, and smoothing the contour data using a Canny operator to obtain the optimized foreground image.
[0104] Furthermore, the optimized foreground image and the foreground attribute data are fused to obtain an image to be identified, and the target to be tracked is identified in a target recognition model pre-trained with an improved Faster-RCNN network to obtain a target area image and an image segmentation mask corresponding to the target area image, including:
[0105] Pre-collecting a plurality of groups of images to be identified and recognition results corresponding to the images to be identified to obtain a first training set;
[0106] Constructing the improved Faster-RCNN network; the improved Faster-RCNN network includes: an input layer, a MobileNetv3 network layer, a BiFPN network layer, a ROI generation network layer, an adaptive resolution network layer, a ROI feature extraction layer, a detection network layer, and an output layer connected in sequence;
[0107] Utilizing the input layer to receive the image to be recognized;
[0108] Using the MobileNetv3 network layer to perform feature extraction on the image to be identified to obtain feature extraction data;
[0109] Using the BiFPN network layer to perform upsampling and fusion processing on the image feature extraction data to obtain an original feature map;
[0110] Using the ROI generation network layer to set the foreground area in the original feature map as the ROI area;
[0111] Mapping the ROI region to a fixed resolution size using the adaptive resolution network layer to obtain an ROI feature map;
[0112] Using the ROI feature extraction layer to perform feature extraction and pooling processing on the ROI feature map to obtain image features;
[0113] Using the detection network layer to perform probability calculation on the image features to obtain a region recognition result;
[0114] Outputting the region recognition result using the output layer;
[0115] Calculating a binary cross entropy loss based on the first training set and the region recognition result to obtain a loss value;
[0116] The improved Faster-RCNN network is iteratively trained using the loss value to obtain the target recognition model.
[0117] Specifically, the optimized foreground image and the foreground attribute data are fused to obtain an image to be identified, and the target to be tracked is identified in a target recognition model pre-trained by an improved Faster-RCNN network to obtain a target area image and an image segmentation mask corresponding to the target area image, including:
[0118] Using the target recognition model to identify the image to be identified, to obtain the target area image;
[0119] The contour of the target area image is extracted to obtain a target area mask, and the inner pixel points of the target area mask are set to 1, and the outer pixel points of the target area mask are set to 0 to obtain the image segmentation mask.
[0120] Furthermore, the optimized point cloud and the tracked object motion data are fused to obtain current tracked object state data, and the current tracked object state data is predicted using a pre-trained target position prediction model to obtain predicted movement position data, including:
[0121] Pre-collecting several sets of the current tracked object state data and future tracked object position data corresponding to the current tracked object state data to obtain a second training set;
[0122] Determining the optimized point cloud as a graph node, and determining the relationship between adjacent graph nodes as a graph edge, to obtain graph structure data;
[0123] constructing node features of the graph nodes and edge features of the graph edges according to the motion data of the tracked object;
[0124] Inputting the graph structure data into a dynamic graph neural network for prediction to obtain a network prediction result;
[0125] Calculating the Euclidean distance between the network prediction result and the future tracking object position data in the second training set to obtain a distance error;
[0126] The dynamic graph neural network is iteratively trained according to the distance error to obtain the target position prediction model.
[0127] Specifically, this embodiment uses a drone equipped with a 4D millimeter-wave radar and a multispectral camera to collect point cloud data and remote sensing images of the target area. Based on the drone's historical movement data, these remote sensing images are aligned and padded to create a virtual static image set. A Gaussian mixture model is then applied to extract the dynamic foreground from this virtual image set, generating an original foreground image. The original foreground image is then subjected to noise filtering and contour optimization to obtain a clearer foreground image. The radiometric characteristics of the optimized foreground image within the remote sensing image are extracted to obtain foreground attribute data. This embodiment combines the optimized foreground image and foreground attribute data to obtain an image to be identified. A pretrained target recognition model is then used to perform target recognition, generating an image of the target area and a corresponding segmentation mask. Background points are removed from the point cloud to be analyzed using the segmentation mask, resulting in an optimized point cloud. Next, motion data for each point, such as distance, speed, orientation, and angular velocity, is extracted to obtain motion data for the tracked object. Finally, the optimized point cloud is fused with the motion data of the tracked object to obtain the current state data of the tracked object, and the pretrained target position prediction model is used to predict its future position. By calculating the difference between the predicted position and the current state, the flight control instructions of the UAV are calculated, so that the desired target can be tracked effectively.
[0128] Preferably, this embodiment uses a lightweight aluminum alloy bracket to fix the 4D millimeter wave radar and multispectral camera in the middle position of the bottom of the drone. In order to calibrate the camera internal parameters, a checkerboard calibration plate is used, and the feature points of the radar point cloud and the camera image are combined for joint calibration to calculate the rotation and displacement between the radar and the camera. A unified second pulse signal is provided to the radar and camera through the GPS clock module. For asynchronously collected data, this embodiment uses a timestamp interpolation method for alignment to ensure that the time difference between the point cloud and the image is less than 5 milliseconds. According to the size of the target area, the scanning range of the 4D millimeter wave radar and the density of the point cloud are set. Then, several sets of multispectral camera exposure times are pre-set, and the foreground image effects extracted under different exposure times are evaluated to obtain a foreground extraction score. The exposure time with the highest score is then updated to the multispectral camera. The calculation formula for the foreground extraction score is as follows:
[0129] S=α1·P+α2·E
[0130]
[0131] Among them, TP is the number of correctly identified foreground pixels, and FP is the number of misidentified background pixels; E actual is the edge pixel set under the current exposure time, E reference is the ideal edge set, σ is the neighborhood size, and the edge points within the neighborhood are considered to meet the requirements.
[0132] Furthermore, the foreground image to be extracted is determined, and then several frames of images before and after the foreground image are extracted one by one according to the set frame value to obtain an image set to be analyzed. The calculation formula of the frame setting value is as follows:
[0133] n f =round(β·N)
[0134] n b =Nn f
[0135] N=min(T,F+B)
[0136] Based on historical motion data, a mapping matrix is set for each frame image. This matrix is used to process these frames to obtain a set of original mapped frames. Next, similarity matching is performed between the pixels within the left and right boundaries of each original mapped frame image and the foreground image to be extracted, thereby obtaining an optimized set of mapped frames. Finally, these optimized frames are cut according to the foreground image window to obtain a cropped frame image set. The missing parts of the cropped frame images are filled with the foreground image to form a virtual static acquisition image set.
[0137] Preferably, a Gaussian mixture model (GMM) can fit the statistical characteristics of background pixels in an image. In a set of virtual static images, the grayscale or color of each pixel can be regarded as a random value taken from a Gaussian distribution. The pixel distribution of the background is usually relatively stable, while the pixels of the foreground object are very different from the background, so the foreground and background can be distinguished by probability. In water scenes, waves and changes in light and shadow will cause the background pixels to fluctuate. The Gaussian mixture model will continuously update the mean, variance, and weight of the Gaussian distribution to better adapt to background changes and reduce the impact of dynamic noise on the extracted foreground.
[0138] Furthermore, when processing images captured by virtual static images, the Gaussian distribution parameters of the foreground are extracted from each frame of the image. The degree of match between each pixel and these Gaussian distributions is calculated. A threshold is set. If the maximum degree of match between a certain pixel and all Gaussian distributions is lower than this threshold, it is determined to be a foreground pixel. Otherwise, it is considered to be a background pixel. Finally, a foreground mask image is generated. Next, a small convolution kernel is used to process this foreground mask. If a central pixel is mostly surrounded by background, it is set as the background to remove isolated noise points at the foreground edge. The current pixel value is replaced by the median of the surrounding pixels to effectively remove salt and pepper noise while maintaining edge details. In addition, the foreground area is expanded, holes are filled, and the eroded target contour is restored to make the foreground more complete.
[0139] Optionally, water surface ripples, light reflections, and small jitters in the movement of the drone may cause some isolated pixels or small areas of noise. The edge of the target may also be affected by environmental noise, appearing jagged or discontinuous, thus affecting the subsequent recognition accuracy. This embodiment uses the 8 neighbor relationships around the pixel to classify adjacent foreground pixels into the same connected domain, and each connected domain represents an independent foreground area. For any two foreground areas, the minimum distance between them is calculated; if the pixel distance between the two areas is greater than the set threshold, it means that they may be different targets or noise. At this time, the smaller areas are removed, and the connected domain that may be the real target is retained. Finally, the Canny operator is used to remove burrs and discontinuous points on the contour, making the target edge smoother and more complete, which is convenient for subsequent segmentation and point cloud processing.
[0140] While the optimized foreground image removes noise and optimizes outlines, it only provides information about the target's geometric shape. The radiometric signature, on the other hand, reflects the target's material and surface characteristics, providing a richer input dimension for target recognition. Multispectral cameras typically collect data in the visible and near-infrared bands.
[0141] Specifically, the improved Faster-RCNN network design is as follows: Input layer: This layer primarily receives processed images, and the image size and number of channels must be pre-adjusted. MobileNetv3 network layer: Compared to the traditional ResNet, this design is more lightweight and uses depthwise separable convolution to reduce computational effort. BiFPN network layer: This layer utilizes bidirectional connections to enhance the interaction of features between different layers and weightedly integrates features of different scales to better detect small objects. ROI generation network layer: This layer directly generates regions of interest (ROIs), achieving faster detection than traditional methods. Adaptive resolution network layer: This layer dynamically adjusts the resolution of feature maps based on ROI sizes. ROI feature extraction layer and detection network layer: This layer pools the ROI feature maps, and the detection network introduces a lightweight classifier to reduce computational latency. During the recognition process, the target recognition model identifies the image to be identified, generating an image of the target region. The contours of this image are then extracted to form a mask of the target region, marking internal pixels as 1 and external pixels as 0, ultimately resulting in an image segmentation mask.
[0142] Furthermore, this image segmentation mask is used to remove background points from the point cloud to achieve point cloud optimization. Next, the movement information of each point is extracted from the optimized point cloud to obtain the motion data of the tracked object, including distance, speed, direction, and angular velocity. The training method for the target position prediction model is as follows: first, some current tracked object state data and corresponding future position data are collected to form a training set; then, the optimized point cloud is treated as a graph node, and the relationships connecting adjacent graph nodes are treated as graph edges to generate graph structure data; the features of the graph nodes and edges are constructed based on the motion data of the tracked object; then, these graph structure data are input into the dynamic graph neural network for prediction to obtain the prediction result; then, the Euclidean distance between the prediction result and the future position data in the training set is calculated to obtain the distance error; finally, the dynamic graph neural network is iteratively trained based on this distance error to iteratively obtain the target position prediction model.
[0143] Furthermore, the optimized point cloud data is combined with the motion data of the tracked object to determine the current state of the tracked object. The trained model is then used to predict this state data and determine the future movement position. Next, by comparing this predicted position with the current state, the drone's desired movement is calculated and, based on this information, the drone is controlled to track the target.
[0144] As an optional implementation, this embodiment also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor can execute the aforementioned surface target tracking method that integrates 4D millimeter-wave radar and UAV remote sensing images.
[0145] As an optional implementation, this embodiment also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the aforementioned surface target tracking method that integrates 4D millimeter-wave radar and UAV remote sensing images.
[0146] The beneficial effects of the present invention are as follows:
[0147] The present invention facilitates foreground extraction using a Gaussian mixture model through alignment and filling processing, thereby improving the foreground analysis effect. It reduces most background data irrelevant to the tracking target through dynamic foreground extraction, noise filtering, contour optimization, and radiation feature extraction, thereby improving the input data quality of the recognition model. It optimizes the structure of the recognition model and improves the recognition speed through an improved Faster-RCNN network. It reduces the point cloud volume and increases the input data dimension through background point cloud removal and tracking object motion data extraction.
[0148] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0149] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.
Claims
1. A surface target tracking method based on 4D millimeter wave radar and UAV remote sensing image fusion, characterized in that: include: The 4D millimeter-wave radar and multispectral camera attached to the data acquisition drone are used to collect the point cloud and remote sensing images of the target area. Performing alignment and filling processing on the remote sensing images according to the historical movement data of the acquisition drone to obtain a virtual static acquisition image set; Performing dynamic foreground extraction on the virtual static collected image set using the Gaussian mixture model to obtain an original foreground image; performing noise filtering and contour optimization on the original foreground image to obtain an optimized foreground image; Extracting radiation characteristics of the foreground area in the optimized foreground image within the remote sensing image to obtain foreground attribute data; The optimized foreground image and the foreground attribute data are fused to obtain an image to be identified, and the target to be tracked is identified in a target recognition model pre-trained by an improved Faster-RCNN network to obtain a target area image and an image segmentation mask corresponding to the target area image; Using the image segmentation mask to remove background point clouds from the point cloud to be analyzed to obtain an optimized point cloud; Extracting movement data of each point cloud in the optimized point cloud to obtain motion data of the tracked object; The tracking object motion data includes: distance data, speed data, orientation data and angular velocity data; fusing the optimized point cloud with the motion data of the tracked object to obtain current tracked object state data, and using a pre-trained target position prediction model to predict the current tracked object state data to obtain predicted movement position data; The target movement information of the collection drone is calculated using the position difference between the predicted movement position data and the current tracking object state data to obtain a flight control instruction, and the flight control instruction is used to control the collection drone to track the target to be tracked.
2. The method for tracking a water surface target by fusing 4D millimeter-wave radar with UAV remote sensing images according to claim 1, characterized in that: The 4D millimeter-wave radar and multispectral camera attached to the data acquisition drone are used to collect the point cloud and remote sensing images of the target area to be analyzed, including: Fixing the 4D millimeter-wave radar and the multispectral camera on the bottom of the data acquisition drone; Performing spatial coordinate system calibration and time synchronization calibration on the 4D millimeter-wave radar and the multispectral camera; Setting the radar scanning range and point cloud density of the 4D millimeter-wave radar according to the range of the target area; Preset multiple groups of exposure times for the multispectral camera, score the extraction effects of the original foreground image at different exposure times to obtain a foreground extraction score, and update the exposure time corresponding to the maximum value of the foreground extraction score to the multispectral camera; the foreground extraction score is calculated as follows: S = α1·P + α2·E; where S is the foreground extraction score; α1 and α2 are the first weight and the second weight, respectively; P is the foreground pixel accuracy; and E is the edge integrity under an ideal edge reference.
3. The method for tracking a water surface target by fusing 4D millimeter-wave radar with UAV remote sensing images according to claim 2, characterized in that: The remote sensing images are aligned and filled according to the historical movement data of the acquisition drone to obtain a virtual static acquisition image set, including: Determine the foreground image to be extracted; According to the frame setting value, a plurality of frames before the foreground image to be extracted and a plurality of frames after the foreground image to be extracted are sequentially extracted to obtain a frame image set to be analyzed; the calculation formula of the frame setting value includes: n f =round(β·N) and n b =Nn f ; Where, N = min (T, F + B); n f 、n b are the number of frames extracted in the forward direction and the number of frames extracted in the backward direction, respectively; round(·) indicates the rounding operation; β is the division ratio; N is the actual number of frames extracted; min(·) indicates the minimum value operation; T is the preset number threshold; F indicates the number of images with a repetition rate greater than 80% in the forward adjacent frames; B indicates the number of images with a repetition rate greater than 80% in the backward adjacent frames; Setting a mapping matrix for each frame image in the frame image set to be analyzed according to the historical movement data, and performing mapping processing on each frame image in the frame image set to be analyzed using the mapping matrix to obtain an original mapping frame image set; Performing similarity matching on pixel points within a critical range of the left boundary and the right boundary of each frame image in the original mapping frame image set and the foreground image to be extracted to obtain an optimized mapping frame image set; Each frame image in the optimized mapping frame image set is cropped according to the window of the foreground image to be extracted to obtain a cropped frame image set, and the missing area of each frame image in the cropped frame image set is filled with the foreground image to be extracted to obtain the virtual static acquisition image set.
4. The method for tracking a water surface target by fusing 4D millimeter-wave radar with UAV remote sensing images according to claim 3, characterized in that: Performing dynamic foreground extraction on the virtual static collected image set using the Gaussian mixture model to obtain an original foreground image includes: Calculating the matching degree between the pixel points of each frame image in the virtual static captured image set and each Gaussian distribution in the foreground image to be extracted to obtain matching degree data; Using a preset Gaussian threshold to filter foreground pixels in the matching data to obtain a foreground mask image; The foreground mask image is sequentially subjected to erosion processing, median filtering, and dilation processing to obtain the original foreground image.
5. The method for tracking a water surface target by fusing 4D millimeter-wave radar with UAV remote sensing images according to claim 4, characterized in that: Noise filtering and contour optimization are performed on the original foreground image to obtain an optimized foreground image, including: Determining adjacent foreground pixels in the original foreground image as being in the same area, and setting an area consisting of all foreground pixels in the same area as the foreground area; Calculating the distance between any two foreground areas to obtain a regional pixel distance; Eliminating the foreground area where the pixel distances of the area are all greater than a preset pixel distance threshold to obtain a denoised foreground image; Contour data of each foreground area in the denoised foreground image is extracted, and the contour data is smoothed using a Canny operator to obtain the optimized foreground image.
6. The method for tracking a water surface target by fusing 4D millimeter-wave radar with UAV remote sensing images according to claim 5, characterized in that: The optimized foreground image and the foreground attribute data are fused to obtain an image to be identified, and the target to be tracked is identified in the target recognition model pre-trained by the improved Faster-RCNN network to obtain a target area image and an image segmentation mask corresponding to the target area image, including: Pre-collecting a plurality of groups of images to be identified and recognition results corresponding to the images to be identified to obtain a first training set; Constructing the improved Faster-RCNN network; the improved Faster-RCNN network includes: an input layer, a MobileNetv3 network layer, a BiFPN network layer, a ROI generation network layer, an adaptive resolution network layer, a ROI feature extraction layer, a detection network layer, and an output layer connected in sequence; Utilizing the input layer to receive the image to be recognized; Using the MobileNetv3 network layer to perform feature extraction on the image to be identified to obtain feature extraction data; Using the BiFPN network layer to perform upsampling and fusion processing on the image feature extraction data to obtain an original feature map; Using the ROI generation network layer to set the foreground area in the original feature map as the ROI area; Mapping the ROI region to a fixed resolution size using the adaptive resolution network layer to obtain an ROI feature map; Using the ROI feature extraction layer to perform feature extraction and pooling processing on the ROI feature map to obtain image features; Using the detection network layer to perform probability calculation on the image features to obtain a region recognition result; Outputting the region recognition result using the output layer; Calculating a binary cross entropy loss based on the first training set and the region recognition result to obtain a loss value; The improved Faster-RCNN network is iteratively trained using the loss value to obtain the target recognition model.
7. The method for tracking a water surface target by fusing 4D millimeter-wave radar with UAV remote sensing images according to claim 6, characterized in that: The optimized foreground image and the foreground attribute data are fused to obtain an image to be identified, and the target to be tracked is identified in the target recognition model pre-trained by the improved Faster-RCNN network to obtain a target area image and an image segmentation mask corresponding to the target area image, including: Using the target recognition model to identify the image to be identified, to obtain the target area image; The contour of the target area image is extracted to obtain a target area mask, and the inner pixel points of the target area mask are set to 1, and the outer pixel points of the target area mask are set to 0 to obtain the image segmentation mask.
8. The method for tracking a water surface target by fusing 4D millimeter-wave radar with UAV remote sensing images according to claim 7, characterized in that: The optimized point cloud and the tracked object motion data are integrated to obtain current tracked object state data, and the current tracked object state data is predicted using a pre-trained target position prediction model to obtain predicted movement position data, including: Pre-collecting several sets of the current tracked object state data and future tracked object position data corresponding to the current tracked object state data to obtain a second training set; Determining the optimized point cloud as a graph node, and determining the relationship between adjacent graph nodes as a graph edge, to obtain graph structure data; constructing node features of the graph nodes and edge features of the graph edges according to the motion data of the tracked object; Inputting the graph structure data into a dynamic graph neural network for prediction to obtain a network prediction result; Calculating the Euclidean distance between the network prediction result and the future tracking object position data in the second training set to obtain a distance error; The dynamic graph neural network is iteratively trained according to the distance error to obtain the target position prediction model.
9. An electronic device, characterized in that: include: At least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute a surface target tracking method of fusion of 4D millimeter-wave radar and UAV remote sensing imagery according to any one of claims 1 to 8.
10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute a surface target tracking method comprising fusion of 4D millimeter-wave radar and UAV remote sensing images according to any one of claims 1 to 8.
Citation Information
Cited By
River barrier monitoring and identification method based on millimeter wave radar
CN122116277A
