Water surface target tracking method and system based on 4D millimeter wave radar and infrared image fusion, electronic equipment and storage medium

Abstract

The invention belongs to the technical field of radar tracking, and provides a water surface target tracking method and system based on 4D millimeter wave radar and infrared image fusion, electronic equipment and a storage medium. The method comprises the steps of equipment assembly, data acquisition, segmentation and classification, image set selection, infrared image cutting, feature extraction, concerned area matching, concerned point cloud extraction, tracked object information extraction, data cutting, tracked object matching and movement strategy generation. According to the invention, through image set selection, infrared image cutting and feature extraction, on-site analysis of a target tracking object is realized, and adaptability to different tracking objects is improved; through concerned area matching, concerned point cloud extraction, tracked object information extraction and tracked object matching, edge growth, point cloud extraction and information matching methods are utilized, so that the computing power required by target tracking is reduced, and the system response speed is increased; through infrared image cutting, point clouds with low influence degree are removed, and the calculation amount of edge equipment is reduced.

Description

Technical Field

[0001] The present invention relates to the field of radar tracking technology, and in particular to a surface target tracking method, system, electronic equipment and storage medium that fuses 4D millimeter-wave radar and infrared images. Background Art

[0002] 4D millimeter-wave radar is an upgraded version of traditional millimeter-wave radar, capable of detecting objects based on distance, speed, horizontal direction, and vertical height. This technology is rapidly developing, with its primary advantage being increased transmission and reception channels, which improves detection accuracy and enhances point cloud density and resolution. Currently, 4D millimeter-wave radar is widely used in target tracking technology, typically combining 4D millimeter-wave radar with neural networks.

[0003] However, existing methods often require separate training for certain types of objects in advance to achieve better tracking effects. This requires advance training when tracking different types of objects, which consumes a lot of time. It is not suitable for target tracking in emergency situations and has poor flexibility. For fixed acquisition equipment, the collected data can be uploaded to a high-computing power terminal for identification and tracking. However, for mobile devices or edge devices, they do not have high computing power capabilities. If a lightweight model deployment solution is adopted, it will lead to problems such as decreased tracking effect and slow response speed. In addition, conventional technologies often directly use the collected point cloud data when analyzing point cloud data, but there is a large amount of redundant data in the point cloud data, which has a relatively low impact on tracking, but will occupy the same proportion of computing power, resulting in excessive computational effort. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the purpose of the present invention is to provide a surface target tracking method, system, electronic device and storage medium that integrates 4D millimeter-wave radar and infrared image, so as to solve the problems of poor flexibility, slow response speed and large amount of calculation in the existing technology.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A surface target tracking method using 4D millimeter-wave radar and infrared image fusion, comprising:

[0007] The 4D millimeter-wave radar, infrared camera, and color camera are fixed on a quad-wing drone at the same angle to obtain a data collection drone.

[0008] The data acquisition drone is used to collect information about the object to be tracked in the target water area to obtain radar point cloud data, original infrared images, and original color images;

[0009] Using a preset object segmentation and classification model to perform object contour segmentation and image classification processing on the original color image, a plurality of original classified segmentation image sets and segmentation mask data sets are obtained; the images in each of the image sets have the same type label;

[0010] Selecting the original classified segmented image set corresponding to the object to be tracked to obtain a target image set, and extracting the segmentation mask data corresponding to the target image set in the segmentation mask data set to obtain a target object mask set;

[0011] Using the target object mask set to crop the collected original infrared image to obtain an infrared image of the target area;

[0012] Performing feature extraction and principal component analysis on all infrared images of the target area to obtain low-dimensional infrared features;

[0013] Performing infrared data matching on the newly acquired original infrared image using the low-dimensional infrared features to obtain position data of the area of ​​interest;

[0014] Extracting point cloud data corresponding to the position data of the area of ​​interest from the radar point cloud data to obtain a point cloud of the area of ​​interest;

[0015] Performing edge growth on the radar point cloud data with the point cloud of the area of ​​interest as a starting point to obtain target tracking object point cloud information; the target tracking object point cloud information includes: target edge data, center of gravity data, distance data, speed data, and azimuth angle data;

[0016] According to the point cloud information of the target tracking object, potential moving area prediction and redundant point cloud elimination are performed on the radar point cloud data of the next frame to obtain optimized point cloud data;

[0017] Performing edge growth on the optimized point cloud data to obtain a number of candidate edge point cloud images, and screening the candidate edge point cloud images according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data;

[0018] The target movement data of the data acquisition drone is set according to the target tracking object point cloud information of two adjacent frames, a drone movement control strategy is obtained, and the movement of the data acquisition drone is controlled using the drone movement control strategy.

[0019] Preferably, the original color image is subjected to object contour segmentation and image classification processing using a preset object segmentation and classification model to obtain a plurality of original classified segmentation image sets and segmentation mask data sets, including:

[0020] Input the original color image into the pre-trained segmentation model for detection to obtain an object detection binary mask;

[0021] performing image segmentation on the original color image using the object detection binary mask to obtain an original segmented image set;

[0022] performing standardization processing on the images in the original segmented image set to obtain a standard format input image;

[0023] Perform forward propagation of the standard format input image using a convolutional backbone network of a ResNet-50 pre-trained model based on ImageNet weights to obtain a segmented image feature matrix;

[0024] A K-means algorithm is used to perform cluster analysis on the segmented image feature matrix to obtain the original classified segmented image set, and the object detection binary mask is sorted according to the original classified segmented image set to obtain the segmentation mask data set.

[0025] Preferably, feature extraction and principal component analysis are performed on all infrared images of the target area to obtain low-dimensional infrared features, including:

[0026] The infrared image of the target area is processed by size normalization, grayscale normalization, and Gaussian filtering to remove noise, thereby obtaining a preprocessed infrared image.

[0027] Performing adaptive histogram equalization processing on the pre-processed infrared image using a contrast-limited adaptive histogram equalization method;

[0028] Calculating an image mean and an image standard deviation of the preprocessed infrared image, removing abnormal points of the preprocessed infrared image using a triple standard deviation method based on the image mean and the image standard deviation, and filling information at positions of the removed abnormal points of the preprocessed infrared image using a Navier-Stokes algorithm to obtain an optimized infrared image;

[0029] Calculating the basic features, quantile features, and hotspot ratios of the optimized infrared image to obtain global temperature statistical features; the basic features include: the image mean, the image standard deviation, the maximum value, and the minimum value;

[0030] The temperature contrast factor is integrated with the temperature gradient operator of the original SIFT algorithm to obtain the improved SIFT algorithm; the expression of the temperature contrast factor is: Wherein, C(T) is the temperature contrast factor; T(x0, y0) is the current pixel temperature value; T mean is the mean temperature of the local neighborhood centered at (x0,y0);

[0031] Using an improved SIFT algorithm to extract features from the optimized infrared image to obtain local temperature pattern features;

[0032] fusing the global temperature statistical features and the local temperature pattern features to obtain a fusion feature;

[0033] Performing principal component analysis on the fused features to obtain the low-dimensional infrared features.

[0034] Preferably, using the low-dimensional infrared features to perform infrared data matching on the newly acquired original infrared image to obtain the location data of the area of ​​interest includes:

[0035] Setting a first sliding window, a second sliding window, a matching threshold, and a maximum distance threshold; wherein the size of the second sliding window is smaller than that of the first sliding window;

[0036] Taking the upper left corner of the original infrared image as the starting point of the first sliding window, and controlling the first sliding window to slide according to an S-shaped movement manner;

[0037] Extracting features within the first sliding window to obtain a window feature vector;

[0038] Calculate the matching degree between the window feature vector and the low-dimensional infrared feature to obtain a window matching value; the calculation formula of the window matching value is:

[0039]

[0040] Among them, MatchScore is the window matching value; α is the feature weight; n G is the feature dimension; is the weight of the global feature of the i-th dimension; G i is the i-th dimension global eigenvalue of the low-dimensional infrared feature; is the i-th dimension global eigenvalue of the window feature vector; L is the local temperature pattern feature; L w is the local temperature eigenvector of the window eigenvector; is the mean value of the temperature contrast factor;

[0041] When the window matching value is greater than the matching threshold, marking the window area corresponding to the window matching value as a candidate focus area;

[0042] Marking two candidate regions of interest whose geometric center distance is less than the maximum distance threshold as being in a correlated state, and eliminating candidate regions of interest that do not have the correlated state;

[0043] Performing secondary matching screening on the candidate regions of interest using the second sliding window to obtain an optimized region of interest;

[0044] Extracting the position data of the optimized focus area to obtain the focus area position data;

[0045] When the candidate focus area is not detected in the original infrared image, the second sliding window is used to perform matching screening on the original infrared image.

[0046] Preferably, edge growing is performed on the radar point cloud data with the point cloud of the area of ​​interest as a starting point to obtain point cloud information of the target tracking object, including:

[0047] Determine a point in the point cloud of the region of interest as a seed point;

[0048] Constructing a spatial index tree based on the radar point cloud data to obtain a point cloud space;

[0049] Searching for neighboring points of the seed point through the point cloud space;

[0050] Performing local plane fitting on the neighborhood points using the Open3D algorithm to obtain point normal vector data;

[0051] When the point normal vector data meets the normal vector threshold, the point corresponding to the point normal vector data is determined as the new seed point; when the point normal vector data does not meet the normal vector threshold, the point corresponding to the point normal vector data is determined as an edge point;

[0052] The seed points are grown using the normal vector threshold, and growth is stopped when all the seed points are inside the edge points. The point cloud information of all the seed points and the edge points is integrated to obtain the point cloud information of the target tracking object; the point cloud information includes: the distance data, the speed data, and the azimuth data.

[0053] Preferably, the potential moving area prediction and redundant point cloud elimination are performed on the radar point cloud data of the next frame according to the point cloud information of the target tracking object to obtain optimized point cloud data, including:

[0054] Extracting the time difference between two point cloud data acquisitions of the 4D millimeter-wave radar to obtain the time difference between adjacent frames;

[0055] Calculate the movement data of the data acquisition drone within the past time difference of the adjacent frames according to the adjacent frame time difference to obtain the device movement information;

[0056] According to the time difference between adjacent frames, linearly predict the target edge data and the center of gravity data using the distance data, the speed data, and the azimuth data to obtain the potential movement area;

[0057] The point clouds in the newly collected radar point cloud data that are more than a distance safety margin away from the potential movement area are eliminated to obtain the optimized point cloud data.

[0058] Preferably, edge growing is performed on the optimized point cloud data to obtain a plurality of candidate edge point cloud images, and the candidate edge point cloud images are screened according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data, including:

[0059] Performing edge point detection and data segmentation on the optimized point cloud data using a KD tree algorithm to obtain a plurality of candidate edge point cloud images;

[0060] Similarity analysis and similar image screening are performed on the candidate edge point cloud image and the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data.

[0061] Preferably, a surface target tracking system integrating 4D millimeter-wave radar and infrared imagery comprises:

[0062] A region of interest matching module is used to perform infrared data matching on the newly acquired original infrared image using the low-dimensional infrared features to obtain the region of interest position data;

[0063] A point cloud extraction module for extracting point cloud data corresponding to the position data of the area of ​​interest from the radar point cloud data to obtain a point cloud of the area of ​​interest;

[0064] A tracking object information extraction module is configured to perform edge growth on the radar point cloud data starting from the point cloud of the area of ​​interest to obtain target tracking object point cloud information; the target tracking object point cloud information includes: target edge data, center of gravity data, distance data, speed data, and azimuth angle data;

[0065] A data clipping module is used to predict the potential moving area and eliminate redundant point clouds for the next frame of radar point cloud data based on the point cloud information of the target tracking object to obtain optimized point cloud data;

[0066] a tracking object matching module, configured to perform edge growth on the optimized point cloud data to obtain a plurality of candidate edge point cloud images, and filter the candidate edge point cloud images according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data;

[0067] The strategy generation module is used to set the target movement data of the data acquisition drone according to the target tracking object point cloud information of two adjacent frames, obtain the drone movement control strategy, and use the drone movement control strategy to control the movement of the data acquisition drone.

[0068] Preferably, an electronic device comprises: at least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor can execute the aforementioned surface target tracking method that integrates 4D millimeter-wave radar and infrared images.

[0069] Preferably, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the aforementioned surface target tracking method by fusing 4D millimeter-wave radar and infrared images.

[0070] The present invention discloses the following technical effects:

[0071] The present invention provides a surface target tracking method, system, electronic device and storage medium that integrates 4D millimeter-wave radar and infrared images. Through image set selection, infrared image cropping and feature extraction, the method solves the defect that conventional recognition models require a lot of time to train in advance, and realizes on-site analysis and feature extraction of target tracking objects. Through focus area matching, focus point cloud extraction, tracking object information extraction and tracking object matching, the method solves the problem that existing models require a lot of computing power and slow response speed, and realizes target tracking by edge feature analysis of infrared images and point cloud images. Through infrared image cropping, the method solves the defect that conventional technologies often directly use collected point cloud data when analyzing point cloud data, and realizes the elimination of potential point clouds with low tracking impact. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0073] Figure 1 A schematic diagram of a surface target tracking process using 4D millimeter-wave radar and infrared image fusion provided by an embodiment of the present invention;

[0074] Figure 2 A schematic diagram of the object segmentation and classification process provided by an embodiment of the present invention;

[0075] Figure 3 A schematic diagram of a low-dimensional infrared feature extraction process according to an embodiment of the present invention;

[0076] Figure 4 A schematic diagram of the flow of location data of a region of interest provided by an embodiment of the present invention;

[0077] Figure 5 A schematic diagram of the point cloud information extraction process for tracking objects provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0079] The purpose of the present invention is to provide a surface target tracking method, system, electronic device and storage medium that integrates 4D millimeter-wave radar and infrared imagery to solve the problems of poor flexibility, slow response speed and large computational complexity in the prior art.

[0080] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0081] Figure 1 The schematic diagram of the surface target tracking process of 4D millimeter wave radar and infrared image fusion provided by the embodiment of the present invention is as follows Figure 1 As shown, the present invention provides a surface target tracking method by fusing 4D millimeter-wave radar and infrared image, comprising:

[0082] Step 100: Fix the 4D millimeter-wave radar, infrared camera, and color camera at the same angle on a quadcopter drone to obtain a data collection drone;

[0083] Step 200: using the data acquisition drone to collect information about the object to be tracked in the target water area to obtain radar point cloud data, original infrared images, and original color images;

[0084] Step 300: Using a preset object segmentation and classification model, perform object contour segmentation and image classification processing on the original color image to obtain a plurality of original classified segmented image sets and segmentation mask data sets; each image in the image set has the same type label;

[0085] Step 400: Selecting the original classified segmented image set corresponding to the object to be tracked to obtain a target image set, and extracting the segmentation mask data corresponding to the target image set in the segmentation mask data set to obtain a target object mask set;

[0086] Step 500: cropping the collected original infrared image using the target object mask set to obtain an infrared image of the target area;

[0087] Step 600: performing feature extraction and principal component analysis on all infrared images of the target area to obtain low-dimensional infrared features;

[0088] Step 700: performing infrared data matching on the newly acquired original infrared image using the low-dimensional infrared features to obtain location data of the area of ​​interest;

[0089] Step 80: extracting point cloud data corresponding to the position data of the area of ​​interest from the radar point cloud data to obtain a point cloud of the area of ​​interest;

[0090] Step 900: performing edge growth on the radar point cloud data starting from the point cloud of the region of interest to obtain target tracking object point cloud information; the target tracking object point cloud information includes: target edge data, center of gravity data, distance data, speed data, and azimuth angle data;

[0091] Step 1000: Predicting potential moving areas and eliminating redundant point clouds for the next frame of radar point cloud data based on the target tracking object point cloud information to obtain optimized point cloud data;

[0092] Step 1100: performing edge growing on the optimized point cloud data to obtain a plurality of candidate edge point cloud images, screening the candidate edge point cloud images according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data;

[0093] Step 1200: Setting target movement data of the data acquisition drone based on the target tracking object point cloud information of two adjacent frames, obtaining a drone movement control strategy, and controlling the movement of the data acquisition drone using the drone movement control strategy.

[0094] refer to Figure 2 , using a preset object segmentation and classification model to perform object contour segmentation and image classification processing on the original color image, to obtain several original classified segmentation image sets and segmentation mask data sets, including:

[0095] Step 301: Input the original color image into a pre-trained segmentation model for detection to obtain an object detection binary mask;

[0096] Step 302: performing image segmentation on the original color image using the object detection binary mask to obtain an original segmented image set;

[0097] Step 303: performing standardization processing on the images in the original segmented image set to obtain a standard format input image;

[0098] Step 304: forward propagating the standard format input image using a convolutional backbone network of a ResNet-50 pre-trained model based on ImageNet weights to obtain a segmentation image feature matrix;

[0099] Step 305: performing cluster analysis on the segmented image feature matrix using a K-means algorithm to obtain the original classified segmented image set, and organizing the object detection binary mask according to the original classified segmented image set to obtain the segmentation mask data set.

[0100] refer to Figure 3 , feature extraction and principal component analysis are performed on all infrared images of the target area to obtain low-dimensional infrared features, including:

[0101] Step 601: performing size normalization, grayscale normalization, and Gaussian filtering denoising processing on the infrared image of the target area in sequence to obtain a preprocessed infrared image;

[0102] Step 602: performing adaptive histogram equalization processing on the pre-processed infrared image using a contrast-limited adaptive histogram equalization method;

[0103] Step 603: Calculating the image mean and image standard deviation of the preprocessed infrared image, removing abnormal points from the preprocessed infrared image using a triple standard deviation method based on the image mean and the image standard deviation, and filling in information at the positions of the removed abnormal points in the preprocessed infrared image using a Navier-Stokes algorithm to obtain an optimized infrared image;

[0104] Step 604: Calculate the basic features, quantile features, and hotspot ratio of the optimized infrared image to obtain global temperature statistical features; the basic features include: the image mean, the image standard deviation, the maximum value, and the minimum value;

[0105] Step 605: Fusing the temperature contrast factor with the temperature gradient operator of the original SIFT algorithm to obtain an improved SIFT algorithm; the expression of the temperature contrast factor is: Wherein, C(T) is the temperature contrast factor; T(x0, y0) is the current pixel temperature value; T mean is the mean temperature of the local neighborhood centered at (x0,y0);

[0106] Step 606: Using an improved SIFT algorithm to perform feature extraction on the optimized infrared image to obtain local temperature pattern features;

[0107] Step 607: Fusing the global temperature statistical features and the local temperature pattern features to obtain fused features;

[0108] Step 608: Perform principal component analysis on the fused features to obtain the low-dimensional infrared features.

[0109] refer to Figure 4 , using the low-dimensional infrared features to perform infrared data matching on the newly acquired original infrared image to obtain the location data of the area of ​​interest, including:

[0110] Step 701: Set a first sliding window, a second sliding window, a matching threshold, and a maximum distance threshold; the size of the second sliding window is smaller than the first sliding window;

[0111] Step 702: Using the upper left corner of the original infrared image as the starting point of the first sliding window, and controlling the first sliding window to slide according to an S-shaped movement;

[0112] Step 703: extracting features within the first sliding window to obtain a window feature vector;

[0113] Step 704: Calculate the matching degree between the window feature vector and the low-dimensional infrared feature to obtain a window matching value. The calculation formula for the window matching value is:

[0114]

[0115] Among them, MatchScore is the window matching value; α is the feature weight; n G is the feature dimension; is the weight of the global feature of the i-th dimension; G i is the i-th dimension global eigenvalue of the low-dimensional infrared feature; is the i-th dimension global eigenvalue of the window feature vector; L is the local temperature pattern feature; L w is the local temperature eigenvector of the window eigenvector; is the mean value of the temperature contrast factor;

[0116] Step 705: When the window matching value is greater than the matching threshold, mark the window area corresponding to the window matching value as a candidate focus area;

[0117] Step 706: Mark the two candidate regions of interest whose geometric center distance is less than the maximum distance threshold as being in a correlated state, and eliminate the candidate regions of interest that do not have the correlated state;

[0118] Step 707: using the second sliding window to perform secondary matching screening on the candidate regions of interest to obtain an optimized region of interest;

[0119] Step 708: extracting the location data of the optimized focus area to obtain the location data of the focus area;

[0120] Step 709: When the candidate region of interest is not detected in the original infrared image, the second sliding window is used to perform matching screening on the original infrared image.

[0121] refer to Figure 5 , performing edge growth on the radar point cloud data with the point cloud of the area of ​​interest as the starting point to obtain the target tracking object point cloud information, including:

[0122] Step 901: determining a point in the point cloud of the area of ​​interest as a seed point;

[0123] Step 902: constructing a spatial index tree based on the radar point cloud data to obtain a point cloud space;

[0124] Step 903: searching for neighboring points of the seed point in the point cloud space;

[0125] Step 904: performing local plane fitting on the neighborhood points using the Open3D algorithm to obtain point normal vector data;

[0126] Step 905: When the point normal vector data meets the normal vector threshold, the point corresponding to the point normal vector data is determined as the new seed point; when the point normal vector data does not meet the normal vector threshold, the point corresponding to the point normal vector data is determined as an edge point;

[0127] Step 906: Grow the seed points using the normal vector threshold, stop growing when all the seed points are inside the edge points, and integrate the point cloud information of all the seed points and the edge points to obtain the point cloud information of the target tracking object; the point cloud information includes: the distance data, the speed data, and the azimuth data.

[0128] Preferably, the potential moving area prediction and redundant point cloud elimination are performed on the radar point cloud data of the next frame according to the point cloud information of the target tracking object to obtain optimized point cloud data, including:

[0129] Extracting the time difference between two point cloud data acquisitions of the 4D millimeter-wave radar to obtain the time difference between adjacent frames;

[0130] Calculate the movement data of the data acquisition drone within the past time difference of the adjacent frames according to the adjacent frame time difference to obtain the device movement information;

[0131] According to the time difference between adjacent frames, linearly predict the target edge data and the center of gravity data using the distance data, the speed data, and the azimuth data to obtain the potential movement area;

[0132] The point clouds in the newly collected radar point cloud data that are more than a distance safety margin away from the potential movement area are eliminated to obtain the optimized point cloud data.

[0133] Preferably, edge growing is performed on the optimized point cloud data to obtain a plurality of candidate edge point cloud images, and the candidate edge point cloud images are screened according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data, including:

[0134] Performing edge point detection and data segmentation on the optimized point cloud data using a KD tree algorithm to obtain a plurality of candidate edge point cloud images;

[0135] Similarity analysis and similar image screening are performed on the candidate edge point cloud image and the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data.

[0136] Preferably, a surface target tracking system integrating 4D millimeter-wave radar and infrared imagery comprises:

[0137] A region of interest matching module is used to perform infrared data matching on the newly acquired original infrared image using the low-dimensional infrared features to obtain the region of interest position data;

[0138] A point cloud extraction module for extracting point cloud data corresponding to the position data of the area of ​​interest from the radar point cloud data to obtain a point cloud of the area of ​​interest;

[0139] A tracking object information extraction module is configured to perform edge growth on the radar point cloud data starting from the point cloud of the area of ​​interest to obtain target tracking object point cloud information; the target tracking object point cloud information includes: target edge data, center of gravity data, distance data, speed data, and azimuth angle data;

[0140] A data clipping module is used to predict the potential moving area and eliminate redundant point clouds for the next frame of radar point cloud data based on the point cloud information of the target tracking object to obtain optimized point cloud data;

[0141] a tracking object matching module, configured to perform edge growth on the optimized point cloud data to obtain a plurality of candidate edge point cloud images, and filter the candidate edge point cloud images according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data;

[0142] The strategy generation module is used to set the target movement data of the data acquisition drone according to the target tracking object point cloud information of two adjacent frames, obtain the drone movement control strategy, and use the drone movement control strategy to control the movement of the data acquisition drone.

[0143] Specifically, the four-wing drone can fly stably and can be equipped with various equipment. The sampling drone of this embodiment has a flat plate at the bottom, which is directly fixed in the middle of the drone and connected through the original screw holes. The middle and top flat plates are supported on the bottom plate by four columns to form a multi-layer mounting surface. The radar is fixed to the middle flat plate with screws, with the antenna facing downward, and the radar cable is led out through the pipe on the side of the bracket and connected to the data interface of the drone. The infrared camera and the color camera are installed side by side in front of the top flat plate, and the bottom of the camera is close to the flat plate with an adapter. In order to reduce vibration during flight, this embodiment adds silicone pads to the contact points of the equipment, and the cable part is fixed to the side of the bracket with Velcro to prevent loosening. Finally, place the bracket on a level workbench and use a high-precision level to adjust the angles of the radar, infrared camera and color camera to ensure that the angles between them are consistent.

[0144] Furthermore, multiple color pictures are collected, trying to cover similar objects at different angles, light and backgrounds. Then, the pre-trained weights of the SegmentAnything Model are loaded, and the complete image is input. The model will automatically identify all objects and generate binary masks, each mask corresponding to an object. Finally, the image is segmented according to these binary masks. In this embodiment, the size of each sub-image is first adjusted to the target size, and then the pixel values ​​are normalized. Next, the classification layer of the pre-trained ResNet-50 model is removed, and only the convolutional network is used to extract the feature vector. Afterwards, the K-means algorithm is used for clustering, and a cluster label is assigned to each sub-image. Finally, folders are created according to these labels, and sub-images with the same label are placed in the corresponding folders to organize them into an image set.

[0145] Specifically, a set of original classified segmented images corresponding to the object to be tracked is selected. Segmentation mask data corresponding to the target image set is then extracted from the segmentation mask dataset. The infrared image is cropped based on the extracted mask data. All images are resized to the same size and saved as single-channel grayscale images, while preserving the original temperature information and eliminating brightness differences. A Gaussian filter is then used to remove noise and smooth the image. Next, CLAHE is used to adjust the image contrast. The mean and standard deviation of the image are calculated, outliers are identified and removed, and the Navier-Stokes algorithm is then applied to repair these outliers.

[0146] Furthermore, the global temperature statistics feature shows the overall distribution of heat, including some basic data: mean, standard deviation, maximum and minimum values; the 90th percentile, which indicates the lower limit of the high-temperature area; and the hotspot ratio, which is used to measure the concentration of heat. The local temperature pattern feature captures the details of the heat distribution. By adding a temperature-weighted descriptor to the SIFT algorithm, the descriptor of each SIFT key point is multiplied by the normalized temperature value of the corresponding point, making the feature more sensitive to temperature differences. The temperature-weighted descriptor is calculated as follows:

[0147]

[0148] The optimized infrared image is extracted using an improved SIFT algorithm to obtain local temperature pattern features. This is then combined with global temperature statistics and local temperature pattern features to generate fused features. Finally, principal component analysis is performed on these fused features to obtain low-dimensional infrared features. The above process is performed on the cloud server, while the following process is performed within the control module of the edge device, the drone.

[0149] Preferably, this embodiment does not directly use the mask after color image segmentation to track the point cloud image, because this cannot be segmented in real time during tracking and the speed will be very slow. This embodiment converts the information into an infrared image. In this way, if there is a problem in tracking, it can be quickly corrected through the infrared image, avoiding the delay caused by color image recognition and reducing the risk of tracking failure.

[0150] Specifically, the sliding window setting and movement strategy. This embodiment adopts a dual-window design. The first window is relatively large, mainly used to quickly screen the original infrared image and effectively remove the obviously mismatched parts; the second window is smaller, focusing on fine matching of the candidate area, thereby improving the accuracy of positioning. This embodiment uses an "S"-shaped sliding method, starting from the upper left corner of the image, and moving the window line by line along the "S"-shaped path. Like a snake scan, this method can ensure that the entire image is covered while reducing the overlap of adjacent windows, thereby improving scanning efficiency. For each infrared image area within the sliding window, global features and local temperature pattern features of the same dimension as the low-dimensional infrared features are extracted, and calculated using the matching formula. The calculation formula is as follows:

[0151]

[0152] Candidate area screening and optimization. Preliminary screening: When the window matching degree exceeds the set value, it is marked as the focus area, and the obviously mismatched background will be excluded. Geometric correlation filtering: If the geometric centers of two candidate areas are less than the maximum threshold, it means that they may be different parts of the same target and are marked as "related state"; otherwise, isolated areas are removed to reduce misjudgment, such as avoiding false positives caused by noise points. Secondary screening: Use the second sliding window to match the candidate area more carefully, capture the details of the target through a smaller window, further filter out false detections, and obtain more accurate focus areas. Exception handling: If the candidate area is not found in the original image, it may be that the target is small or the features are not obvious. Use the second sliding window directly for comprehensive and detailed screening to ensure that the target is not missed.

[0153] Furthermore, point cloud data related to the location of the area of ​​interest is extracted from the radar point cloud data to obtain a point cloud for that area. Points in the point cloud of the area of ​​interest are selected as seed points, which are the starting points for growth. These point clouds of the area of ​​interest are matched using infrared images and generally correspond to high-probability areas of the target object. This embodiment uses a KD tree to establish a spatial index for the radar point cloud data, forming a point cloud space. This spatial index is used to search for points surrounding the seed point, and the Open3D algorithm is used to perform local plane fitting on these neighboring points. Specifically, a plane equation is fitted using the least squares method, and the plane's normal vector represents the direction of the local surface of the point cloud. A normal vector angle threshold is set to determine whether the normal vectors of the neighboring point and the seed point are consistent. If the angle between the normal vector of the neighboring point and the seed point is less than the threshold, it indicates that the local surface of the point cloud is continuous, and this point is determined as a new seed point and growth continues. If the angle is greater than the threshold, it may be the edge of an object or an area with a sudden surface change. This point is marked as an edge point and growth stops. In the case of surface targets, the normal vectors of the main point cloud are relatively consistent, while the normal vectors of edge points vary significantly. Using a normal vector threshold, we can clearly distinguish the boundary between the main object and the background. When all neighboring points of a seed point are edge points, the entire boundary of the target object has been covered and growth stops. By integrating all seed points and edge points, we extract point cloud information for target tracking, including distance data, velocity data, azimuth data, target edge data, and center of gravity data.

[0154] Preferably, the time difference between two point cloud data acquisitions by the 4D millimeter-wave radar is extracted to obtain the time difference between adjacent frames, and the movement information of the drone during this period is calculated based on this time difference. Then, using this time difference between adjacent frames, combined with distance, speed and azimuth data, a linear prediction is performed on the edge data and center of gravity data of the target to find the potential movement area. Finally, the point clouds in the newly collected radar point cloud data that are beyond the safety boundary of this potential movement area are eliminated, and finally the optimized point cloud data is obtained.

[0155] Furthermore, drone motion data includes displacement and rotation angles. Assuming the target moves in a uniform linear motion over a short period of time, the next frame's position is predicted based on the object point cloud information in the current frame. In edge regions, the predicted center of gravity is used as the center, and the possible range of motion is generated based on the target's size and direction of motion. Since the drone itself is also moving, when considering the target's relative motion, the drone's displacement is inversely compensated for within the target's motion model. This avoids target position deviations caused by the drone's flight.

[0156] Specifically, for each point in the optimized point cloud, this embodiment uses the KD tree to find the points around it and calculate the distance or normal vector difference between them. If this difference exceeds the set threshold, the point is treated as an edge point. Based on the edge points, the point cloud is divided into several connected areas, and each area becomes a candidate edge point cloud image. Then, features related to past target tracking are extracted from the candidate image, and these candidate features are compared with historical target features using a similarity formula, and finally, candidate point cloud images that meet the requirements are screened out. The target center of gravity coordinates, distance data, edge contours, speed data, and azimuth data are obtained from the target tracking object point cloud information of two adjacent frames, and the UAV is controlled using a PID control strategy to achieve the purpose of accident tracking.

[0157] As a feasible implementation method, this embodiment also provides an electronic device, including: at least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor so that the processor can execute the aforementioned surface target tracking method that integrates 4D millimeter-wave radar and infrared images.

[0158] As a feasible implementation method, this embodiment also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the aforementioned surface target tracking method that integrates 4D millimeter-wave radar and infrared images.

[0159] The beneficial effects of the present invention are as follows:

[0160] The present invention realizes on-site analysis and feature extraction of target tracking objects through image set selection, infrared image cropping, and feature extraction, thereby improving the adaptability and flexibility of different tracking objects. By focusing on region matching, focusing on point cloud extraction, tracking object information extraction, and tracking object matching, the present invention utilizes the focus on region edge growth, point cloud information extraction, and information matching methods to reduce the computing power required for target tracking and speed up the system response. By cropping infrared images, potential point clouds with low impact on tracking are eliminated, which directly reduces the computing power of edge devices.

[0161] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0162] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A surface target tracking method based on 4D millimeter wave radar and infrared image fusion, characterized in that: include: The 4D millimeter-wave radar, infrared camera, and color camera are fixed on a quad-wing drone at the same angle to obtain a data collection drone. The data acquisition drone is used to collect information about the object to be tracked in the target water area to obtain radar point cloud data, original infrared images, and original color images; Using a preset object segmentation and classification model to perform object contour segmentation and image classification processing on the original color image, a plurality of original classified segmentation image sets and segmentation mask data sets are obtained; the images in each of the image sets have the same type label; Selecting the original classified segmented image set corresponding to the object to be tracked to obtain a target image set, and extracting the segmentation mask data corresponding to the target image set in the segmentation mask data set to obtain a target object mask set; Using the target object mask set to crop the collected original infrared image to obtain an infrared image of the target area; Performing feature extraction and principal component analysis on all infrared images of the target area to obtain low-dimensional infrared features; Performing infrared data matching on the newly acquired original infrared image using the low-dimensional infrared features to obtain position data of the area of ​​interest; Extracting point cloud data corresponding to the position data of the area of ​​interest from the radar point cloud data to obtain a point cloud of the area of ​​interest; Performing edge growth on the radar point cloud data with the point cloud of the area of ​​interest as a starting point to obtain point cloud information of the target tracking object; The target tracking object point cloud information includes: target edge data, center of gravity data, distance data, speed data, and azimuth angle data; According to the point cloud information of the target tracking object, potential moving area prediction and redundant point cloud elimination are performed on the radar point cloud data of the next frame to obtain optimized point cloud data; Performing edge growth on the optimized point cloud data to obtain a number of candidate edge point cloud images, and screening the candidate edge point cloud images according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data; The target movement data of the data acquisition drone is set according to the target tracking object point cloud information of two adjacent frames, a drone movement control strategy is obtained, and the movement of the data acquisition drone is controlled using the drone movement control strategy.

2. The surface target tracking method of 4D millimeter wave radar and infrared image fusion according to claim 1 is characterized in that: The original color image is processed with object contour segmentation and image classification using a preset object segmentation and classification model to obtain a number of original classified segmentation image sets and segmentation mask data sets, including: Input the original color image into the pre-trained segmentation model for detection to obtain an object detection binary mask; performing image segmentation on the original color image using the object detection binary mask to obtain an original segmented image set; performing standardization processing on the images in the original segmented image set to obtain a standard format input image; Perform forward propagation of the standard format input image using a convolutional backbone network of a ResNet-50 pre-trained model based on ImageNet weights to obtain a segmented image feature matrix; A K-means algorithm is used to perform cluster analysis on the segmented image feature matrix to obtain the original classified segmented image set, and the object detection binary mask is sorted according to the original classified segmented image set to obtain the segmentation mask data set.

3. The surface target tracking method of 4D millimeter wave radar and infrared image fusion according to claim 2 is characterized in that: Perform feature extraction and principal component analysis on all infrared images of the target area to obtain low-dimensional infrared features, including: The infrared image of the target area is processed by size normalization, grayscale normalization, and Gaussian filtering to remove noise, thereby obtaining a preprocessed infrared image. Performing adaptive histogram equalization processing on the pre-processed infrared image using a contrast-limited adaptive histogram equalization method; Calculating an image mean and an image standard deviation of the preprocessed infrared image, removing abnormal points of the preprocessed infrared image using a triple standard deviation method based on the image mean and the image standard deviation, and filling information at positions of the removed abnormal points of the preprocessed infrared image using a Navier-Stokes algorithm to obtain an optimized infrared image; Calculating the basic features, quantile features, and hotspot ratios of the optimized infrared image to obtain global temperature statistical features; the basic features include: the image mean, the image standard deviation, the maximum value, and the minimum value; The temperature contrast factor is integrated with the temperature gradient operator of the original SIFT algorithm to obtain the improved SIFT algorithm; the expression of the temperature contrast factor is: Wherein, C(T) is the temperature contrast factor; T(x0, y0) is the current pixel temperature value; T mean is the mean temperature of the local neighborhood centered at (x0,y0); Using an improved SIFT algorithm to extract features from the optimized infrared image to obtain local temperature pattern features; fusing the global temperature statistical features and the local temperature pattern features to obtain a fusion feature; Performing principal component analysis on the fused features to obtain the low-dimensional infrared features.

4. The surface target tracking method of 4D millimeter wave radar and infrared image fusion according to claim 3 is characterized in that: Performing infrared data matching on the newly acquired original infrared image using the low-dimensional infrared features to obtain location data of the area of ​​interest, including: Setting a first sliding window, a second sliding window, a matching threshold, and a maximum distance threshold; wherein the size of the second sliding window is smaller than that of the first sliding window; Taking the upper left corner of the original infrared image as the starting point of the first sliding window, and controlling the first sliding window to slide according to an S-shaped movement manner; Extracting features within the first sliding window to obtain a window feature vector; Calculate the matching degree between the window feature vector and the low-dimensional infrared feature to obtain a window matching value; the calculation formula of the window matching value is: Among them, MatchScore is the window matching value; α is the feature weight; n G is the feature dimension; is the weight of the global feature of the i-th dimension; G i is the i-th dimension global eigenvalue of the low-dimensional infrared feature; is the i-th dimension global eigenvalue of the window feature vector; L is the local temperature pattern feature; L w is the local temperature eigenvector of the window eigenvector; is the mean value of the temperature contrast factor; When the window matching value is greater than the matching threshold, marking the window area corresponding to the window matching value as a candidate focus area; Marking two candidate regions of interest whose geometric center distance is less than the maximum distance threshold as being in a correlated state, and eliminating candidate regions of interest that do not have the correlated state; Performing secondary matching screening on the candidate regions of interest using the second sliding window to obtain an optimized region of interest; Extracting the position data of the optimized focus area to obtain the focus area position data; When the candidate focus area is not detected in the original infrared image, the second sliding window is used to perform matching screening on the original infrared image.

5. The surface target tracking method of 4D millimeter wave radar and infrared image fusion according to claim 4 is characterized in that: The radar point cloud data is edge-grown with the point cloud of the area of ​​interest as a starting point to obtain the point cloud information of the target tracking object, including: Determine a point in the point cloud of the region of interest as a seed point; Constructing a spatial index tree based on the radar point cloud data to obtain a point cloud space; Searching for neighboring points of the seed point through the point cloud space; Performing local plane fitting on the neighborhood points using the Open3D algorithm to obtain point normal vector data; When the point normal vector data meets the normal vector threshold, the point corresponding to the point normal vector data is determined as the new seed point; when the point normal vector data does not meet the normal vector threshold, the point corresponding to the point normal vector data is determined as an edge point; The seed points are grown using the normal vector threshold, and growth is stopped when all the seed points are inside the edge points. The point cloud information of all the seed points and the edge points is integrated to obtain the point cloud information of the target tracking object; the point cloud information includes: the distance data, the speed data, and the azimuth data.

6. The method for tracking a surface target by fusing 4D millimeter-wave radar and infrared images according to claim 5, characterized in that: According to the target tracking object point cloud information, potential moving area prediction and redundant point cloud elimination are performed on the radar point cloud data of the next frame to obtain optimized point cloud data, including: Extracting the time difference between two point cloud data acquisitions of the 4D millimeter-wave radar to obtain the time difference between adjacent frames; Calculate the movement data of the data acquisition drone within the past time difference of the adjacent frames according to the adjacent frame time difference to obtain the device movement information; According to the time difference between adjacent frames, linearly predict the target edge data and the center of gravity data using the distance data, the speed data, and the azimuth data to obtain the potential movement area; The point clouds in the newly collected radar point cloud data that are more than a distance safety margin away from the potential movement area are eliminated to obtain the optimized point cloud data.

7. The surface target tracking method of 4D millimeter wave radar and infrared image fusion according to claim 6 is characterized in that: Performing edge growing on the optimized point cloud data to obtain a number of candidate edge point cloud images, screening the candidate edge point cloud images according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data, including: Performing edge point detection and data segmentation on the optimized point cloud data using a KD tree algorithm to obtain a plurality of candidate edge point cloud images; Similarity analysis and similar image screening are performed on the candidate edge point cloud image and the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data.

8. A surface target tracking system that integrates 4D millimeter-wave radar and infrared imagery, characterized in that: The surface target tracking method according to claim 1, which is a 4D millimeter-wave radar and infrared image fusion method, comprises: A region of interest matching module is used to perform infrared data matching on the newly acquired original infrared image using the low-dimensional infrared features to obtain the region of interest position data; A point cloud extraction module for extracting point cloud data corresponding to the position data of the area of ​​interest from the radar point cloud data to obtain a point cloud of the area of ​​interest; A tracking object information extraction module is configured to perform edge growth on the radar point cloud data starting from the point cloud of the area of ​​interest to obtain target tracking object point cloud information; the target tracking object point cloud information includes: target edge data, center of gravity data, distance data, speed data, and azimuth angle data; A data clipping module is used to predict the potential moving area and eliminate redundant point clouds for the next frame of radar point cloud data based on the point cloud information of the target tracking object to obtain optimized point cloud data; a tracking object matching module, configured to perform edge growth on the optimized point cloud data to obtain a plurality of candidate edge point cloud images, and filter the candidate edge point cloud images according to the target tracking object point cloud information to obtain the target tracking object point cloud information corresponding to the radar point cloud data; The strategy generation module is used to set the target movement data of the data acquisition drone according to the target tracking object point cloud information of two adjacent frames, obtain the drone movement control strategy, and use the drone movement control strategy to control the movement of the data acquisition drone.

9. An electronic device, characterized in that: include: At least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute a surface target tracking method of 4D millimeter-wave radar and infrared image fusion according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute a surface target tracking method comprising 4D millimeter-wave radar and infrared image fusion according to any one of claims 1 to 7.

Images