Multi-Target Tracking Method for UAVs Based on Visual / Millimeter-Wave Radar Information Fusion

Abstract

Multi-object tracking method for unmanned aerial vehicle based on visual / mmWave radar information fusion. 1. Read the point cloud information of the mmWave radar at the current moment, and calculate the centroid information of each type of point cloud to represent the information of the object corresponding to this type of point cloud; 2. Read the monocular vision RGB image, use the deep learning object detection algorithm to generate the object detection frame, and calculate the depth information of the object through the monocular depth estimation method; 3. Perform sensor joint calibration on the mmWave radar and monocular vision; 4. Match the mmWave radar and monocular vision detection data that have completed spatio-temporal synchronization one by one; 5. According to the data matching result, fuse the mmWave radar and vision data, and design a Kalman filter for the fused data; 6. Based on the global nearest neighbor matching algorithm and the aforementioned Kalman filter, design a multi-object tracker to obtain the online multi-object tracking result. This application can improve the environmental perception ability of the unmanned aerial vehicle, and thus improve the intelligence level of the unmanned aerial vehicle.

Description

Technical Field

[0001] The present invention belongs to the field of UAV environmental perception, and particularly relates to a multi-target tracking method for UAVs based on visual / mmWave radar information fusion. Background Technique

[0002] As a flying robot, UAVs have been widely used in many fields such as industry, agriculture, commerce, and civilian consumer markets due to their advantages of flexibility, portability, and the ability to work in various complex environments, and have broad development prospects. In UAV applications, online tracking of multiple targets of interest in the environment is an important basis for realizing functions such as UAV monitoring, following, and target guidance, and has received increasing attention from researchers in recent years.

[0003] To achieve online multi-target tracking, commonly used sensing sensors include monocular cameras, depth cameras, lidars, and mmWave radars. Monocular cameras have the advantages of low cost, easy installation, and rich semantic information in the collected images. Commonly used real-time monocular camera multi-target tracking algorithms include DeepSort, ByteTrack, and OC_SORT. However, due to camera jitter during UAV flight, it is easy to cause target tracking loss, and the monocular ranging accuracy is limited. Therefore, using only a monocular camera to achieve UAV multi-target tracking has poor effects. The ranging range of depth cameras is relatively small, and they are easily affected by environmental factors such as light. Therefore, when used alone, their limitations are relatively large. Lidars can provide both rich semantic information and accurate target distance information at the same time, but high-performance lidars are usually expensive, bulky, and have high power consumption, making it difficult to be widely deployed on UAV platforms. MmWave radars have the advantages of small size, low power consumption, little influence by environmental factors, and the ability to accurately measure the distances of multiple targets. However, mmWave radars have the disadvantages of lack of semantic information in data and low angle measurement accuracy, making it difficult to complete accurate multi-target tracking tasks alone.

[0004] The currently applied solutions are as follows:

[0005] Application No.: CN201811444277.8, Application Title: A Method for Information Fusion of Millimeter-Wave Radar and Monocular Camera for Intelligent Vehicles. It claims to protect a method for information fusion of millimeter-wave radar and monocular camera for intelligent vehicles, mainly including three stages: In the first stage, first install and adjust the positions of the camera and millimeter-wave radar on the intelligent vehicle, then obtain the internal parameters of the camera using the Zhang Zhengyou calibration method, and finally project the camera imaging plane onto the top view to ensure that the target information recognized by the two sensors is in the same vehicle coordinate system; In the second stage, offline, use the deep learning method to train the multi-target recognition model based on images. Online, during the driving of the intelligent vehicle, the radar and camera collect data in real time, convert the recognized target information into the unified vehicle coordinate system and exclude invalid targets; In the third stage, use the global nearest neighbor matching algorithm to judge whether the targets are consistent, and track the matched and unmatched targets through the extended Kalman filter to achieve multi-target recognition. The invention has a high target recognition rate and reliable safety, and can effectively improve the practicality of intelligent vehicles' environmental perception.

[0006] First, it projects the camera imaging plane onto the top view to ensure that the target information recognized by the two sensors is in the same vehicle coordinate system. Then, it uses the deep learning method for multi-target recognition, converts the obtained radar and camera target information into the unified vehicle coordinate system and excludes invalid targets. Finally, it uses the global nearest neighbor matching algorithm to judge the target consistency, and tracks the matched and unmatched targets through the extended Kalman filter to achieve multi-target recognition.

[0007] When the method of this application processes data, compared with the compared patent, it performs additional clustering and centroid calculation processing on the millimeter-wave radar data, improving the quality of the radar data. When processing the camera data, after obtaining the detection box of the target, it further proposes a monocular depth estimation method to complete the visual distance information complementation and obtain richer visual detection results. At the same time, when estimating the target state and generating the target trajectory, the compared patent simply judges the target validity according to whether the current detection matches the trajectory for the purpose of target recognition, while this patent further designs a cascade multi-target tracker based on the global nearest neighbor matching algorithm and Kalman filter for the purpose of accurate multi-target tracking in the case of long time and detection loss. Therefore, these two patents are different both in the specific data processing methods and the final achieved purposes.

[0008] Application No.: CN202210562730.5, Application Name: Vehicle Tracking Method, Device and System Based on Multi-Information Fusion. This application relates to the field of intelligent vehicle technologies. Specifically, a vehicle tracking method based on multi-information fusion is disclosed, which includes: respectively obtaining the initial point cloud data and visual image information of a target vehicle, where the initial point cloud data is the point cloud data of the target vehicle collected by a calibrated millimeter-wave radar, and the visual image information is the image information of the target vehicle collected by a calibrated visual acquisition device; performing clustering processing on the initial point cloud data to obtain a radar detection box, and performing target detection processing on the visual image information to obtain a visual detection box; fusing the radar detection box and the visual detection box, and determining a main fusion target according to the fusion result; performing optimization processing on the main fusion target to obtain the tracking result of the target vehicle. The present invention also discloses a vehicle tracking device and system based on multi-information fusion. The vehicle tracking method based on multi-information fusion provided by the present invention can improve the vehicle tracking effect.

[0009] After this application obtains the initial point cloud data and visual image information of the target vehicle, it performs DBSCAN clustering processing on the initial point cloud data to obtain a radar detection box, performs target detection processing on the visual image information to obtain a visual detection box, then fuses the radar detection box and the visual detection box, determines a main fusion target according to the fusion result, and finally performs optimization processing on the main fusion target to obtain the tracking result of the target vehicle.

[0010] When processing millimeter-wave radar data in this application, the clustering method used is Euclidean clustering. After clustering, the centroid information of each object class is used to replace the information of that object class through the centroid calculation method, achieving data dimensionality reduction. When processing visual data, after generating the target detection box, a monocular depth estimation method is proposed to complete the complement of the distance information of vision, further enriching the visual detection results. When performing spatial alignment and data matching of the two sensors, the visual depth estimation results are used to achieve the alignment and matching of visual data and radar data in the vehicle coordinate system. Compared with the method in the comparative patent that directly projects radar data into the image coordinate system for matching, this method further considers the problems of large deviations usually existing when projecting radar data onto the image and the loss of the depth scale of the image itself, and can obtain more accurate and robust data matching results. When performing Kalman update, the global nearest neighbor matching algorithm used comprehensively considers the global cost during data matching compared with the nearest neighbor matching algorithm in the comparative patent, so it can effectively reduce the probability of false matching. When designing the target tracker, a cascade multi-object tracker based on the global nearest neighbor matching algorithm and the Kalman filter is designed. Compared with the method in the comparative patent that only outputs the tracking result based on the matching result of one detection and the trajectory, the proposed tracker has a more complete tracking strategy and can better handle the target tracking problem under long-term and detection loss conditions, achieving better multi-object tracking results. There are significant differences between the two patents in sensor data processing, data fusion matching, and target tracker design. Moreover, the method proposed in this patent is more comprehensive than the comparative patent and can achieve better multi-object tracking results.

[0011] To make up for the deficiencies of single sensors, multi-source sensor fusion perception is a current research hotspot. Considering the characteristics of the UAV platform itself and the characteristics of each sensor, the fusion of millimeter-wave radar and vision is an ideal solution to the UAV online multi-object tracking problem. In existing multi-object tracking algorithms based on the fusion of millimeter-wave radar and vision, sensor data matching is generally performed in the pixel coordinate system. However, the angle measurement accuracy of millimeter-wave radar is lower than that of vision, and there are usually deviations between the corresponding radar pixel projection points and visual measurements. In addition, the depth dimension information of the pixel coordinate system itself is lost. Therefore, the overall accuracy of this matching strategy is limited. In addition, after existing methods complete data matching fusion and target state estimation, only a simple trajectory management strategy is used to achieve multi-object tracking. Therefore, existing fusion multi-object tracking algorithms usually have disadvantages such as discontinuous trajectory generation and low tracking accuracy under detection loss conditions. Summary of the Invention

[0012] To solve the above problems, a multi-target tracking method for drones based on visual / millimeter-wave radar information fusion is proposed. After separately processing the radar data and monocular visual data, this method uses the depth estimation information of monocular vision to perform data matching in the carrier coordinate system, improving the accuracy of matching. After completing the sensor data matching fusion and target state estimation, a complete multi-target tracker is further designed to improve the accuracy of online multi-target tracking for drones, solving the problem of poor tracking effects of single sensors and existing fusion tracking methods.

[0013] To achieve the above object, the technical solution adopted by the present invention is:

[0014] A multi-target tracking method for drones based on visual / millimeter-wave radar information fusion, the specific steps are as follows, and it is characterized in that:

[0015] (1) Read the point cloud information of the millimeter-wave radar at the current moment, where the i-th point cloud information is denoted as p i = [x, y, z, v x , v y , v z , where x, y, z represent the position components of the target in the directions of each coordinate axis in space, and v x , v y , v z represent the velocity components of the target in the directions of each coordinate axis in space. After obtaining the radar point cloud information, first use the Euclidean clustering algorithm to filter out the outlier noise points, and then use the centroid of each class of point cloud to represent the object corresponding to the class of point cloud, realizing an effective reduction in the amount of radar data. The expression for calculating the centroid is:

[0016]

[0017] (2) Read the RGB image collected by the monocular vision, and use the trained YOLOv5n target detection model to realize the online detection and extraction of the target of interest. The detection box information of the target of interest obtained by the target detection algorithm is:

[0018] bbox = [u, v, w b , h b

[0019] where u, v are the pixel coordinates of the center point of the detection box, and w b , h b are the width and height of the detection box;

[0020] Combined with the width or height dimension information of the target of interest, using the image detection box information and the camera projection model, the calculation expression for monocular depth estimation is proposed as follows:

[0021] ​

[0022] Wherein, d is the estimated depth of the target, f is the focal length of the camera, w and h are the actual sizes of the target, and u and v are the sizes of the target in the pixel coordinate system;

[0023] (3) Jointly calibrate the millimeter-wave radar and monocular vision to achieve spatio-temporal alignment of heterogeneous sensor data;

[0024] (4) Match the millimeter-wave radar and visual detection data that have completed spatio-temporal synchronization one by one, screen out the multi-source sensor information of the target of interest, and use the global nearest neighbor matching GNN algorithm to achieve data matching. The distance calculation criterion between the two sensor data selects the Mahalanobis distance, and the calculation expression is:

[0025]

[0026] Wherein, e ij represents the residual vector measured by the two sensors, and S is the corresponding covariance. In order to measure the risk cost of data matching, a threshold G is set to represent the maximum matching distance between the two sensor data. The risk cost calculation formula for data matching under the threshold constraint is:

[0027]

[0028] Wherein, c ij is the risk cost of matching a certain data pair, D max is a constant much larger than d ij and G. When the total risk of allocating each measurement element of the radar to each measurement element of the vision is minimized, the obtained allocation result is the required global nearest neighbor data matching result. The optimization objective for realizing this process is:

[0029]

[0030]

[0031] (5) For the successfully matched radar data and visual data, fuse the two sensor data, select the center point of the visual detection frame, and use the radar depth information matched with it to correct its projection in the vehicle coordinate system. The calculation formula is:

[0032]

[0033] Use the velocity information of the radar measurement value to supplement the corrected visual measurement value projection, and obtain the sensor fusion measurement value of each target center point as Z k =[x b ,y b ,v xb ,v yb, for the fused measurement values, a Kalman filter is designed to improve the detection robustness. The prediction part of the filter is as follows:

[0034]

[0035] In the formula, the state transition matrix F k,k-1 takes the value of indicating a uniform motion model; the initial value P0 of the system covariance matrix takes the identity matrix, and the system process noise matrix Q k represents the perturbation amount of the uniform motion model. The data update part of the filter is as follows:

[0036]

[0037] In the formula, Z k is the current measurement value, H k is the observation matrix, which takes the identity matrix here, and R k is the observation noise covariance matrix.

[0038] (6) Design a cascaded online multi-object tracker based on the global nearest neighbor matching GNN algorithm and the aforementioned Kalman filter to output accurate online multi-object tracking results.

[0039] As a further improvement of the present invention, in step (2), TensorRT is required to perform inference acceleration on the YOLOv5n model to improve the operating efficiency of the object detection algorithm on the airborne embedded platform and meet the requirements of online real-time detection.

[0040] As a further improvement of the present invention, the specific sensor spatio-temporal alignment method proposed in step (3) is as follows:

[0041] (3-1) When performing time alignment, it is proposed to adopt the nearest neighbor time matching algorithm. The principle of this algorithm is: calculate the difference between the radar data timestamp and the adjacent visual detection data timestamp, and match the data pair with the smallest difference and less than the set threshold, and consider it corresponding to the same moment to achieve time alignment;

[0042] (3-2) After completing time alignment, it is proposed to convert the data of both sensors to the body frame to achieve spatial alignment of the data. In the designed system, the coordinates of points in the radar coordinate system, camera coordinate system, and body coordinate system are (x r , y r , z r ), (x c , y c , z c ) and (x b , y b , z b), the coordinates of a point in the pixel coordinate system are (u, v). According to the camera projection model, the formula for converting a point in the pixel coordinate system to the camera coordinate system is:

[0043]

[0044] In the formula, Z c is the target distance obtained by the monocular depth estimation algorithm, and f x , f y , c x and c y are the camera internal parameters, which are obtained by the checkerboard calibration method. According to the installation position of the camera on the UAV, the conversion relationship between the camera coordinate system and the carrier coordinate system is:

[0045]

[0046] The conversion relationship between the millimeter-wave radar coordinate system and the camera coordinate system is solved by the direct linear transformation method (DLT). Combining the camera projection model, there is a relational expression:

[0047]

[0048] In the formula, (u, v, 1) T is the projection of the radar point cloud in the pixel coordinate system, which is determined by the manual observation method. Expanding this formula further gives

[0049]

[0050] In the formula, Define P=(x r , y r , z r , 1), a1=(a1, a2, a3, a4) T , a2=(a5, a6, a7, a8) T , a3=(a9, a 10 , a 11 , a 12 ) T , eliminating Z c results in the following constraints:

[0051]

[0052] From this formula, two equations are constructed for each pair of points. When there are a total of N pairs of points, the following matrix equation is constructed:

[0053]

[0054] When N≥6, the least-squares solution of this equation is solved by SVD decomposition. Performing SVD decomposition on A gives:

[0055] (U1 Σ1 V1) = SVD(A)

[0056] Further considering the scale factor, the actual solution of the matrix equation is as follows:

[0057]

[0058] where β is the proportionality coefficient, The obtained rotation matrix is

[0059]

[0060] To calculate the optimal rotation matrix with scale and orthogonality properties, perform SVD decomposition on to obtain:

[0061]

[0062] The obtained optimal rotation matrix is:

[0063] R cr = ±U2V2 T

[0064] The solved proportionality coefficient is:

[0065] β = ±1 / (tr(Σ2) / 3)

[0066] Since the detection points should be in front of the radar and the camera, there is:

[0067]

[0068] Determine the positive and negative of β and R from this formula, and further obtain the translation vector as cr After calculating the rotation matrix R

[0069]

[0070] After calculating the rotation matrix R cr and the translation vector t cr the conversion relationship between the radar coordinate system and the camera coordinate system is:

[0071]

[0072] Then, based on the aforementioned conversion relationship between the camera coordinate system and the vehicle body coordinate system, further convert the points in the radar coordinate system to the vehicle body coordinate system. Thus, the visual detection data and the millimeter-wave radar data can both be converted to the unified vehicle body coordinate system, realizing the data space alignment of the two sensors.

[0073] As a further improvement of the present invention, in step (4), when solving the global nearest neighbor data matching result, the Hungarian algorithm is adopted to improve the solving efficiency.

[0074] As a further improvement of the present invention, the principle of the cascade online multi-object tracker designed based on the global nearest neighbor matching GNN algorithm and the aforementioned Kalman filter in step (6) is as follows: According to the credibility of the trajectory, the trajectory state is divided into a confirmed state and an unconfirmed state. The process of the tracking algorithm is as follows:

[0075] (1) Use the GNN algorithm to match the fused measurement data and the predicted values of the confirmed state trajectories, obtaining three groups of data: unmatched trajectories, unmatched measurements, and matched trajectories. For the matched trajectories, update the trajectory state according to the associated current measurement values; for the unmatched trajectories, if they have failed to match for a continuously set threshold number of cycles, delete this trajectory, otherwise temporarily retain this trajectory but do not update the trajectory; for the unmatched measurements, use them as the input for the next-level data matching.

[0076] (2) Use the GNN algorithm again to match the unmatched measurements from the previous level and the predicted values of the unconfirmed state trajectories, obtaining new unmatched trajectories, unmatched measurements, and matched trajectories. For the matched trajectories, update the trajectory state using the associated measurements. At this time, if the matched trajectories have been successfully matched for a continuously set threshold number of cycles, their state will be converted to the confirmed state; for the unmatched trajectories, delete this trajectory; for the unmatched measurements, generate new unconfirmed state trajectories.

[0077] (3) Output the confirmed state trajectories generated in steps (1) and (2), and these trajectories are the results of online multi-object tracking.

[0078] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0079] (1) The present invention processes millimeter-wave radar data using Euclidean clustering and centroid calculation, and processes visual data using deep learning object detection and monocular depth estimation methods, realizing the extraction of effective information of the targets to be tracked, reducing the amount of sensor data, and improving the running efficiency of the algorithm.

[0080] (2) By combining the depth estimated by monocular vision, the camera projection model, and the coordinate conversion relationship between the calibrated sensors, the present invention proposes to convert the visual detection data and the radar detection data into the vehicle coordinate system for data matching, making up for the defect that other existing methods are prone to matching failure when performing data matching in the pixel coordinate system, and improving the accuracy of data matching.

[0081] (3) After completing the sensor data matching fusion and target state estimation, the present invention further designs a complete multi-object tracker based on the global nearest neighbor matching algorithm and the Kalman filter, improving the accuracy of UAV online multi-object tracking and solving the problem of poor tracking effects of single sensors and existing fusion tracking methods. Brief Description of the Drawings

[0082] Figure 1 is a flowchart of the disclosed method of the present invention;

[0083] Figure 2 is a structural diagram of the online multi-target tracker in the disclosed method of the present invention. Detailed Embodiments

[0084] To make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the following further elaborates on the present invention with reference to the drawings and by way of examples.

[0085] The present invention discloses a multi-target tracking method for unmanned aerial vehicles based on vision / millimeter-wave radar information fusion with a flowchart as Figure 1 shown, including the following steps:

[0086] Step 1: Read the point cloud information of the millimeter-wave radar at the current moment, where the i-th point cloud information is denoted as p i =[x, y, z, v x , v y , v z , where x, y, and z represent the position components of the target in the directions of each coordinate axis in space, and v x , v y , v z represent the velocity components of the target in the directions of each coordinate axis in space. After obtaining the radar point cloud information, first use the Euclidean clustering algorithm to filter out outlier noise points, and then use the centroid of each class of point cloud to represent the object corresponding to that class of point cloud to effectively reduce the amount of radar data. The expression for centroid calculation is:

[0087]

[0088] Step 2: Read the RGB image collected by the monocular vision, and use the trained YOLOv5n target detection algorithm to detect and extract the target of interest. To improve the running efficiency of the target detection algorithm on the embedded platform and meet the requirements of online real-time detection, it is proposed to use TensorRT to accelerate the inference of the YOLOv5n model. The detection box information of the target of interest is obtained through the target detection algorithm, and the expression is:

[0089] bbox = [u, v, w b , h b

[0090] where u and v are the pixel coordinates of the center point of the detection box, and w b , h b are the width and height of the detection box.

[0091] ​Combined with the size information such as the width or height of the target of interest, using the image detection box information and the camera projection model, the calculation expression for monocular depth estimation is proposed as follows:

[0092]

[0093] In the formula, d is the estimated depth of the target, f is the camera focal length, w and h are the actual sizes of the target, and u and v are the sizes of the target in the pixel coordinate system.

[0094] Step 3: Jointly calibrate the millimeter-wave radar and monocular vision to achieve spatio-temporal alignment of heterogeneous sensor data. The specific steps are as follows:

[0095] (3-1) When performing time alignment, the nearest-neighbor time matching algorithm is proposed. The principle of this algorithm is: calculate the difference between the radar data timestamp and the adjacent visual detection data timestamp, and match the data pair with the smallest difference and less than the set threshold, considering that they correspond to the same moment to achieve time alignment.

[0096] (3-2) After completing time alignment, the present invention proposes to convert the data of both sensors to the vehicle coordinate system to achieve spatial alignment of the data. In the designed system, the coordinates of points in the radar coordinate system, camera coordinate system, and vehicle coordinate system are (x r , y r , z r ), (x c , y c , z c ), and (x b , y b , z b ), respectively, and the coordinates of points in the pixel coordinate system are (u, v). According to the camera projection model, the formula for converting points in the pixel coordinate system to the camera coordinate system is:

[0097]

[0098] In the formula, Z c is the target distance obtained through the monocular depth estimation algorithm, f x , f y , c x and c y are the camera internal parameters, which can be obtained through the checkerboard calibration method. According to the installation position of the camera on the vehicle, the conversion relationship between the camera coordinate system and the vehicle coordinate system is:

[0099] <gro

[0100] The conversion relationship between the millimeter-wave radar coordinate system and the camera coordinate system is solved by the direct linear transformation method (DLT). Combining the camera projection model, there is a relational expression:

[0101]

[0102] where \((u, v, 1)\) T is the projection of the radar point cloud in the pixel coordinate system, which can be determined by manual observation method. Further expanding this formula gives

[0103]

[0104] where define \(P=(x\) r , \(y\) r , \(z\) r , \(1)\), \(a1=(a1, a2, a3, a4)\) T , \(a2=(a5, a6, a7, a8)\) T , \(a3=(a9, a\) 10 , \(a\) 11 , \(a\) 12 ) T , eliminating \(Z\) c results in the following constraints:

[0105]

[0106] From this formula, it can be seen that two equations can be constructed for each pair of points. When there are a total of \(N\) pairs of points, the following matrix equation can be constructed:

[0107]

[0108] When \(N\geq6\), the least squares solution of this equation is solved by SVD decomposition. Performing SVD decomposition on \(A\) gives:

[0109] \((U1\ \Sigma1\ V1) = SVD(A)\)

[0110] Further considering the scale factor, the actual solution of the matrix equation is:

[0111]

[0112] where \(\beta\) is the proportionality coefficient, the obtained rotation matrix is

[0113]

[0114] To calculate the optimal rotation matrix with scale and orthogonality properties, performing SVD decomposition on gives:

[0115]

[0116] The obtained optimal rotation matrix is:

[0117] Rcr = ±U2V2 T

[0118] The obtained proportionality coefficient is:

[0119] β = ±1 / (tr(Σ2) / 3)

[0120] Since the detection point should be in front of the radar and the camera, there is:

[0121]

[0122] From this equation, the positive and negative of β and R can be determined cr and the translation vector is further obtained as

[0123]

[0124] After calculating the rotation matrix R cr and the translation vector t cr the conversion relationship between the radar coordinate system and the camera coordinate system can be obtained as:

[0125]

[0126] Furthermore, based on the aforementioned conversion relationship between the camera coordinate system and the vehicle body coordinate system, the points in the radar coordinate system can be further converted into the vehicle body coordinate system. Thus, the visual detection data and the millimeter-wave radar data can both be converted into the unified vehicle body coordinate system, realizing the spatial alignment of the data of the two sensors.

[0127] Step 4: Match the millimeter-wave radar and visual detection data that have completed spatio-temporal synchronization one by one, and filter out the multi-source sensor information of the target of interest. The present invention uses the global nearest neighbor matching (GNN) algorithm to achieve data matching, and the Mahalanobis distance is selected as the distance calculation criterion between the data of the two sensors. The calculation expression is:

[0128]

[0129] In the formula, e ij represents the residual vector measured by the two sensors, and S is the corresponding covariance. In order to measure the risk cost of data matching, a threshold value G is set to represent the maximum matching distance between the data of the two sensors. The risk cost calculation formula for data matching under the threshold constraint is:

[0130]

[0131] Among them, c ij is the risk cost of a certain data pair matching, and D max is a value much larger than d ijConstants of G. When the total risk of allocating each measurement element of the radar to each measurement element of the vision is minimized, the obtained allocation result is the required global nearest neighbor data matching result. The optimization objective for realizing this process is:

[0132]

[0133]

[0134] This allocation problem is quickly solved by the Hungarian algorithm in the present invention.

[0135] Step 5: For the successfully matched radar data and vision data, fuse the data of the two sensors. Select the center point of the vision detection frame, and use the radar depth information matched with it to correct its projection in the vehicle coordinate system. The calculation formula is:

[0136]

[0137] Use the velocity information of the radar measurement value to supplement the corrected vision measurement value projection, and obtain the sensor fusion measurement value of each target center point as Z k =[x b ,y b ,v xb ,v yb . For the fusion measurement value, the present invention designs a Kalman filter to improve the detection robustness. The prediction part of the filter is:

[0138]

[0139] In the formula, the state transition matrix F k,k-1 takes a value of indicating a constant velocity motion model; the initial value P0 of the system covariance matrix takes the identity matrix, and the system process noise matrix Q k indicates the disturbance amount of the constant velocity motion model. The data update part of the filter is:

[0140]

[0141] In the formula, Z k is the current measurement value, H k is the observation matrix, which takes the identity matrix here, and R k is the observation noise covariance matrix.

[0142] Step 6: Design a cascade online multi-target tracker based on the global nearest neighbor matching (GNN) algorithm and the aforementioned Kalman filter. The framework of the tracker is as Figure 2 shown.

[0143] In this multi-object tracking algorithm, according to the credibility of the trajectory, the trajectory state is divided into a confirmed state and an unconfirmed state. The process of the tracking algorithm is as follows:

[0144] (1) Use the GNN algorithm to match the fused measurement data with the predicted values of the confirmed state trajectories, obtaining three groups of data: unmatched trajectories, unmatched measurements, and matched trajectories. For the matched trajectories, update the trajectory state according to the associated current measurement values; for the unmatched trajectories, if they have failed to match for a continuously set number of cycles, delete the trajectory, otherwise temporarily retain the trajectory without updating it; for the unmatched measurements, use them as the input for the next-level data matching.

[0145] (2) Use the GNN algorithm again to match the unmatched measurements from the previous level with the predicted values of the unconfirmed state trajectories, obtaining new unmatched trajectories, unmatched measurements, and matched trajectories. For the matched trajectories, update the trajectory state using the associated measurements. At this time, if the matched trajectory has been successfully matched for a continuously set number of cycles, its state will be converted to the confirmed state; for the unmatched trajectories, delete the trajectory; for the unmatched measurements, generate new unconfirmed state trajectories.

[0146] (3) Output the confirmed state trajectories generated in steps (1) and (2), which are the results of online multi-object tracking.

[0147] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any other form. Any modification or equivalent change made based on the technical essence of the present invention still falls within the scope of protection required by the present invention.

Claims

1. A multi-target tracking method for drones based on vision / mmWave radar information fusion, the specific steps are as follows, and its characteristics are: (1) Read the point cloud information of the millimeter-wave radar at the current moment, where the i-th point cloud information is denoted as p i = [x, y, z, v x , v y , v z , where x, y, and z represent the position components of the target in the directions of each coordinate axis in space, and v x , v y , v z represent the velocity components of the target in the directions of each coordinate axis in space. After obtaining the radar point cloud information, first use the Euclidean clustering algorithm to filter out the outlier noise points, and then use the centroid of each class of point cloud to represent the object corresponding to that class of point cloud. The expression for calculating the centroid is: (2) Read the RGB images collected by the monocular vision, use the trained YOLOv5n object detection model to achieve online detection and extraction of the target of interest, and obtain the detection box information of the target of interest through the object detection algorithm as: bbox = [u, v, w b , h b ​ where u and v are the pixel coordinates of the center point of the detection box, and w b , h b are the width and height of the detection box; Combined with the width or height dimension information of the target of interest, using the image detection box information and the camera projection model, the calculation expression for monocular depth estimation is proposed as follows: In the formula, d is the target estimated depth, f is the camera focal length, w and h are the actual sizes of the target, and u and v are the sizes of the target in the pixel coordinate system; (3) Calibrate the millimeter-wave radar and the monocular vision jointly to achieve spatio-temporal alignment of heterogeneous sensor data; (4) Match the millimeter-wave radar and visual detection data that have completed spatio-temporal synchronization one by one, screen out the multi-source sensor information of the target of interest, and use the global nearest neighbor matching GNN algorithm to achieve data matching. The distance calculation criterion between the two sensor data selects the Mahalanobis distance, and the calculation expression is: where, e ij represents the residual vector measured by the two sensors, S is the corresponding covariance. To measure the risk cost of data matching, a threshold G is set to represent the maximum matching distance between the data of the two sensors. The calculation formula for the risk cost of data matching under the threshold constraint is: Among them, c ij is the risk cost for a certain data pair match, D max is a constant much larger than d ij and G. When the total risk of allocating each measurement element of the radar to each measurement element of the vision is minimized, the obtained allocation result is the required global nearest neighbor data matching result. The optimization objective for realizing this process is as follows: (5) For the successfully matched radar data and visual data, fuse the two sensor data, select the center point of the visual detection box, and use the matched radar depth information to correct its projection in the vehicle coordinate system. The calculation formula is: Using the speed information of radar measurements to supplement the corrected visual measurement projection, the sensor fusion measurement value of each target center point is obtained as Z k = [x b , y b , v xb , v yb . For the fusion measurement value, a Kalman filter is designed to improve the detection robustness. The prediction part of the filter is as follows: In the formula, the state transition matrix F k,k-1 takes the value of indicating a uniform motion model; the initial value P0 of the system covariance matrix takes the identity matrix, and the system process noise matrix Q k represents the disturbance amount of the uniform motion model, and the data update part of the filter is as follows: where Z k is the current measurement value, H k is the observation matrix, which takes the value of the identity matrix here, and R k is the observation noise covariance matrix; (6) Design a cascaded online multi-target tracker based on the global nearest neighbor matching GNN algorithm and the aforementioned Kalman filter to output accurate online multi-target tracking results.

2. The multi-target tracking method for unmanned aerial vehicles based on visual / mmWave radar information fusion according to claim 1, wherein In step (2), TensorRT is required to accelerate the inference of the YOLOv5n model.

3. The multi-target tracking method for an unmanned aerial vehicle based on visual / mmWave radar information fusion according to claim 1, wherein The specific sensor spatio-temporal alignment method proposed in step (3) is: (3-1) When performing time alignment, a nearest neighbor time matching algorithm is proposed. The principle of this algorithm is: calculate the difference between the radar data timestamp and the adjacent visual detection data timestamp, and match the data pair with the smallest difference and less than the set threshold, and consider it corresponding to the same moment to achieve time alignment; (3-2) After the completion of time alignment, it is proposed to convert the data of both sensors to the vehicle coordinate system to achieve spatial alignment of the data. In the designed system, the coordinates of points in the radar coordinate system, camera coordinate system, and vehicle coordinate system are (x r , y r , z r ), (x c , y c , z c ), and (x b , y b , z b ), respectively. The coordinates of points in the pixel coordinate system are (u, v). According to the camera projection model, the formula for converting points in the pixel coordinate system to the camera coordinate system is: Where Z c is the target distance obtained by the monocular depth estimation algorithm, and f x , f y , c x and c y are the camera internal parameters, which are obtained by the checkerboard calibration method. According to the installation position of the camera on the UAV, the conversion relationship between the camera coordinate system and the carrier coordinate system is as follows: The conversion relationship between the millimeter-wave radar coordinate system and the camera coordinate system is solved by the direct linear transformation method (DLT). Combining the camera projection model, there is a relational expression: where \((u, v, 1)\) T is the projection of the radar point cloud in the pixel coordinate system, which is determined by the manual observation method. Further expanding this equation gives Wherein, Define \(P=(x r ,y r ,z r ,1)\), \(a1=(a1,a2,a3,a4) T , \(a2=(a5,a6,a7,a8) T , \(a3=(a9,a 10 ,a 11 ,a 12 ) T , eliminate \(Z c to obtain the following constraints: From this formula, two equations are constructed for each pair of points. When there are a total of N pairs of points, the following matrix equation is constructed: When N≥6, the least squares solution of this equation is solved by SVD decomposition. Perform SVD decomposition on A to get: (U1 Σ1 V1)=SVD(A) Further considering the scale factor, the actual solution of the matrix equation is: where β is a proportionality coefficient, The rotation matrix is obtained as To calculate the optimal rotation matrix with scale and orthogonality properties, for Performing SVD decomposition gives: The obtained optimal rotation matrix is: R cr = ±U2V2 T The solved scale factor is: β=±1 / (tr(Σ2) / 3) Since the detection points should be in front of the radar and the camera, so there is: Determine β and R from this formula cr The positive or negative sign of, and further obtain the translation vector as After calculating the rotation matrix R cr and the translation vector t cr the transformation relationship between the radar coordinate system and the camera coordinate system is obtained as follows: Then, based on the aforementioned conversion relationship between the camera coordinate system and the vehicle coordinate system, the points in the radar coordinate system are further converted into the vehicle coordinate system.

4. The multi-target tracking method for unmanned aerial vehicles based on visual / mmWave radar information fusion according to claim 1, wherein, In step (4), when solving the global nearest neighbor data matching result, the Hungarian algorithm is used.

5. The multi-target tracking method for an unmanned aerial vehicle based on visual / millimeter-wave radar information fusion according to claim 1, wherein The principle of the cascade online multi-object tracker designed in step (6) based on the global nearest neighbor matching GNN algorithm and the aforementioned Kalman filter is as follows: According to the credibility of the trajectory, the trajectory state is divided into a confirmed state and an unconfirmed state. The flow of the tracking algorithm is as follows: (1) Use the GNN algorithm to match the fused measurement data and the predicted values of the confirmed state trajectories, obtaining three groups of data: unmatched trajectories, unmatched measurements, and matched trajectories. For the matched trajectories, update the trajectory state according to the associated current measurement values. For the unmatched trajectories, if they have failed to match for a continuously set number of cycles, delete the trajectory; otherwise, temporarily retain the trajectory but do not update the trajectory. The unmatched measurements will be used as the input for the next-level data matching; (2) Use the GNN algorithm again to match the unmatched measurements from the previous level and the predicted values of the unconfirmed state trajectories, obtaining new unmatched trajectories, unmatched measurements, and matched trajectories. For the matched trajectories, update the trajectory state using the associated measurements. At this time, if the matched trajectory has been successfully matched for a continuously set number of cycles, its state will be converted to the confirmed state. For the unmatched trajectories, delete the trajectory. For the unmatched measurements, generate new unconfirmed state trajectories; (3) Output the confirmed state trajectories generated in steps (1) and (2), which are the results of online multi-object tracking.

Images