Multi-target detection method based on multi-source high-frequency track and high-precision map

Abstract

The invention discloses a multi-target detection method based on a multi-source high-frequency track and a high-precision map, which realizes conversion of a video coordinate system, a millimeter wave radar coordinate system and a world coordinate system and fuses a radar target and a video target. The method comprises the following steps: automatically calibrating video and thunder vision; performing automatic deviation correction on a north corner and an equipment position based on multi-straight-line regression; transforming the video sensor into a world coordinate system; and carrying out decision association of the Leiyu tracking target. According to the method, the target trajectory and the high-precision map data are tracked by utilizing the thunder vision high frequency, the complexity of calibration between the sensors and between the sensors and a world coordinate system is reduced, meanwhile, the precision errors of coordinate transformation between the thunder vision sensors and thunder vision data world coordinate transformation are reduced, and compared with the prior art, the method has the advantages that in the application of complex traffic roadside vehicle detection scenes, the detection efficiency is improved. The precision is higher; and the data association has robustness.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and in particular to a multi-target detection method based on multi-source high-frequency trajectories and high-precision maps. Background Art

[0002] With the development of science and technology and the progress of society, my country's rail transit has gradually become intelligent and has achieved excellent results. In recent years, the field of computer vision has developed rapidly, but due to video sensor acquisition failures or visual algorithm errors, it often brings some hidden dangers to intelligent transportation. Therefore, the introduction of radar data is very necessary. Video information combined with radar captured data can greatly alleviate problems such as data errors, algorithm errors, and data loss. How to more efficiently fuse radar data and video data has become the focus of research in the industry. Radar-vision fusion can be divided into calibration between radar-vision sensors, calibration between radar-vision sensors and the world coordinate system, and radar-vision data association and decision fusion.

[0003] In the past, the calibration method between radar and vision sensors mainly adopted the method of manually selecting point pairs, and the calibration between sensors and the world coordinate system mainly adopted the method of on-site point marking and repeated debugging. Both operations are too cumbersome and easy to introduce operational errors. At the same time, there are also automatic calibration methods, which divide lane areas by statistically counting target trajectories, and then find matching points in each area according to spatial positions. However, this type of method ignores the reliability and uncertainty of trajectories and does not extract valid trajectories. Through frequency statistics, trajectories can be divided into high-frequency and low-frequency. High-frequency trajectories have long-term target superposition and are reliable and certain. At present, the multi-target association of radar and vision fusion decision-making mostly adopts two-level matching or introduces ReID information, but ignores the data characteristics and effectiveness of targets under occlusion.

[0004] The previous calibration method between radar and visual sensors is roughly as follows: First, collect radar data and image acquisition videos over a period of time. Then, rely on manual observation and manual selection of target points, and select target point pairs based on this. Next, calculate the perspective transformation matrix. However, this method has several significant defects: when the number of targets is small, manual point selection is relatively accurate, but once multiple targets are involved, due to the difference in detection angle of view and the inaccuracy of radar detection of stationary targets, the number of target points between the radar and the image often cannot match, resulting in the difficulty in accurately establishing the point pair correspondence. In addition, in the process of manually clicking to select points, point pairs are prone to offset, which in turn causes errors in solving the coordinate plane. The solution of the perspective transformation matrix relies on the fitting of multiple sets of non-collinear point pairs, and manual operation cannot ensure full coverage of all necessary point pairs, which leads to poor calibration results and often requires repeated calibration, which wastes time and cannot guarantee the accuracy of the results.

[0005] Previous sensor and world coordinate system calibration methods mainly involved using standard equipment to generate known "standard" input quantities, inputting them into the sensor to be calibrated, and obtaining calibration data and input-output curves by comparing the sensor output with the input standard quantity, and then analyzing and calculating the technical performance parameters of the calibrated sensor. For the calibration of video and millimeter-wave radar, the basic idea is to establish an accurate coordinate transformation relationship, such as projecting the point cloud generated by the laser into the image through the projection method, looking for markers and checking the alignment of their edge contours. During the calibration process, the sensor's measurement range needs to be divided into several equally spaced points, and repeated measurements of forward and reverse strokes are performed multiple times, and data processing is performed to determine the static characteristic indicators of the sensor, which consumes a lot of time, and manual intervention introduces inestimable errors.

[0006] Previous decision-level fusion methods for radar data directly used motion features to associate the tracking targets of video sensors and radar sensors, or adopted a two-level association between sensor tracking ID and motion features. This ignored the unreliability of occluded targets and introduced erroneous data associations or false fusion targets.

[0007] The effect of radar-vision fusion depends on the performance of the above three parts. Therefore, efficient, accurate and convenient calibration between radar-vision sensors and calibration between sensors and world coordinate system are very necessary, and real-time, accurate and robust data association is essential. Summary of the invention

[0008] The purpose of the present invention is to provide a radar fusion method with high accuracy, high robustness and convenient operation in order to solve the problems existing in the above-mentioned prior art.

[0009] The technical solution to achieve the purpose of the present invention is: a multi-target detection method based on multi-source high-frequency trajectories and high-precision maps, the method comprising:

[0010] Step 1: Automatic coordinate conversion between radar sensors: synchronize video data and radar data, extract features of video targets and radar targets, track video targets and radar targets, count radar and video tracking target frequency graphs, and then use seed filling method for analysis to obtain the effective detection area of ​​the sensor. Based on the statistical analysis of high-frequency trajectories of radar data, automatically select high-frequency point pairs to perform automatic calibration between radar and video sensors.

[0011] Step 2: Calculate the north angle of the radar sensor and automatically correct it: Select the latitude and longitude of the initial position of the device and the lane point set on the high-precision map, outline the polyline corresponding to the detection area in the radar coordinate system, calculate the north angle of each sensor, and automatically correct the north angle;

[0012] Step 3: Based on step 1 and step 2, transform the video sensor to the earth coordinate system;

[0013] Step 4: Perform decision association of the radar sensor: Use multi-frame feature information of the video to determine target occlusion, and use motion feature association for non-occluded targets based on multi-level matching of occluded and non-occluded data. Use radar data to verify occluded targets. Use video IoU association for targets that fail the verification, and use radar observation data for targets that pass the verification.

[0014] Furthermore, the formula for data synchronization in step 1 is:

[0015]

[0016] Where N is the data synchronization frequency in frames per second, T is the radar update time, and INT represents an integer.

[0017] Furthermore, in step 1, the Yolo deep learning network is used to extract the target pixel position and target confidence in the video frame, and the point cloud clustering method is used to analyze the original point cloud data collected by the radar to obtain the target position information and speed information.

[0018] Furthermore, in step 1, the video target and the radar target are tracked based on the SORT algorithm, specifically including:

[0019] (1) Use Kalman filtering to predict radar / video targets separately:

[0020]

[0021] The tracking target at time t-1 and time t is a 6-dimensional vector [X t-1 ,Y t-1 ,Vx t-1 ,Vy t-1 ,ax t-1 ,ay t-1 ]、[X t ,Y t ,Vx t ,Vy t ,ax t ,ay t ], where X t-1 , X t They represent the x-coordinates and Y-coordinates of the radar / video coordinate system at time t-1 and time t, respectively. t-1 , Y t Respectively represent the y-direction coordinates of the radar / video coordinate system at time t-1 and time t, Vx t-1 、Vx t Respectively represent the x-direction velocity in the radar / video coordinate system at time t-1 and time t, Vy t-1 , Vy tRespectively represent the y-direction speed in the radar / video coordinate system at time t-1 and time t, ax t-1 ,ax t They represent the lateral acceleration at time t-1 and time t respectively, and ay t-1 、ay t They represent the longitudinal acceleration at time t-1 and time t respectively; Indicates the difference between the timestamps of the previous and next two frames of radar or video;

[0022] (2) The prediction results and radar / video detection results are associated respectively through the Hungarian matching method.

[0023] Furthermore, in step 1, the frequency graph of tracking targets of radar and video is statistically analyzed, and then the seed filling method is used for analysis to obtain the effective detection area of ​​the sensor. Based on the statistical analysis of the high-frequency trajectory of the radar data, high-frequency point pairs are automatically selected to perform automatic calibration between the radar and video sensors, specifically including:

[0024] Step 1-1, transform the radar data (x1, y1) coordinates to the canvas coordinates (pt) of size (H, W) X1 ,pt Y1 ), the transformation formula is:

[0025] pt X1 =20+(x2+30) / 60.0*(W-40)

[0026] pt Y2 =(H-20)-(y2-0) / 300.0*(H-40)

[0027] Step 1-2, transform the video detection center point coordinates (x2, y2) to the canvas coordinates (pt) of size (H, W) X2 ,pt Y2 ), the transformation formula is:

[0028] pt X2 =x2 / frameWidth*W

[0029] pt Y2 =y2 / frameHeight*H

[0030] Where frameWidth and frameHeight represent the width and height of the video frame used for model inference, respectively, and W and H represent the width and height of the canvas, respectively;

[0031] Step 1-3: Calculate the heat map of historical trajectories of radar and video data, and assign values ​​to the frequency values ​​near the canvas coordinates calculated in steps 1-1 and 1-2. The rules are as follows:

[0032]

[0033] In the formula, Indicates pt X1 or pt X2 ,pt Y1 or pt Y2 , Near(ptX*,PtY*) is a point The coordinate values ​​of the N×N neighborhood at , where N is an odd number; It represents the frequency value corresponding to Near(ptX*,PtY*), Vx and Vy represent the lateral speed and longitudinal speed respectively;

[0034] The frequency of the multi-time radar tracking target set is counted, the frequency is distinguished according to the radial velocity, the neighborhood frequency is assigned, and the radar tracking target frequency map of H×W×1 size and the video tracking target frequency map of H×W×1 size are generated;

[0035] Steps 1-4, normalize the frequency graph and make the normalized result nonlinear. The calculation formula is:

[0036]

[0037] Where Value(ptX*,ptY*) represents the frequency value at the point (ptX*,ptY*), and nonLinearR is the final nonlinear result;

[0038] Steps 1-5, visualize the frequency graph;

[0039] Steps 1-6, generate radar and video target trajectory diagrams respectively: convert the video and radar tracking targets to the H×W range, and record the coordinate values ​​of the targets at each sampling moment;

[0040] Steps 1-7, based on the seed filling method, analyze the radar and video tracking target frequency graphs to automatically obtain the effective detection area of ​​the radar vision sensor, including:

[0041] A foreground point in the frequency graph is selected as a seed, and then the foreground points adjacent to the seed are merged into the same set according to the position proximity condition of the connected area. The final set is a connected area.

[0042] By analyzing the tracking target frequency graph of radar and video in the above manner, the radar effective detection area DETAradar and the video sensor effective detection area DETAcamera are obtained;

[0043] Step 1-8, automatically select the high-frequency coordinate point pairs of radar vision in the effective monitoring area, specifically including:

[0044] Filter the high-frequency point set greater than the preset threshold in the radar effective detection area DETAradar, and filter the high-frequency point set greater than the preset threshold in the video sensor effective detection area DETAcamera;

[0045] Sort the two high-frequency point sets in the Cartesian coordinate x and y directions respectively, and select the high-frequency points with the maximum and minimum values ​​in the x and y directions;

[0046] According to the time t corresponding to the radar / video raw data at the four high-frequency points and the coordinates on the frequency graph, the corresponding target attributes of the video and radar target trajectory graphs are extracted respectively as the corresponding points between the radar and video sensors;

[0047] Step 1-9, calculate the transformation matrix between the radar and video sensors, specifically including:

[0048] Substitute the four sets of coordinates obtained in steps 1-8 into the following formula to solve the transformation matrix between the radar and video sensor:

[0049]

[0050] In the formula, x radar and radar Respectively represent the horizontal and vertical coordinates in the radar coordinate system, x img and img They represent the horizontal and vertical coordinates in the video coordinate system respectively, and M1 is the perspective transformation matrix, that is, the transformation matrix between the radar and the video sensor.

[0051] Furthermore, step 2 uses a multi-linear regression method to calculate and automatically correct the north angle based on the high-precision map and the radar sensor data, specifically including:

[0052] Step 2-1, select the initial equipment longitude and latitude in the high-precision map, the equipment includes radar and video sensors, and select the longitude and latitude coordinate points {P1, P2, P3, ..., Pn} from near to far in a specific lane, Pi represents the i-th longitude and latitude coordinate point, i = 1, 2, ..., n; outline the corresponding point selection line segment LINEradar in the radar tracking target frequency diagram;

[0053] Step 2-2, construct the translation matrix T and rotation matrix R from the radar coordinate system to the world coordinate system:

[0054]

[0055] in,

[0056]

[0057] Where LAT Pi Indicates the latitude of the high-precision map point Pi, LONPi Indicates the longitude of the high-precision map point Pi, LAT radar and LON radar They represent the latitude and longitude of the radar equipment used in the high-precision map, respectively. θ is the angle between the normal of the millimeter-wave radar and the north direction of the geodetic coordinate system, i.e., the north angle.

[0058] Step 2-3, based on the translation matrix T and the rotation matrix R, inversely transform the high-precision map coordinates to the radar coordinate system to obtain the world coordinate mapping point set r corresponding to {P1, P2, P3, ..., Pn} set , where the coordinate correspondence is:

[0059]

[0060] In the formula, r i_x 、r i_y Respectively represent the latitude and longitude of the i-th longitude and latitude coordinate point Pi in the radar coordinate system, P i_x , P i_y Respectively represent the latitude and longitude of the i-th longitude and latitude coordinate point Pi in the high-precision map coordinates;

[0061] Step 2-4, calculate the world coordinate mapping point set r set The error between the line segment formed by the midpoint from near to far and the line segment outlined in step 2-1 is used to calculate the optimal north angle and the longitude and latitude position of the radar / video sensor.

[0062] Furthermore, in steps 2-4, the north angle θ of the high-precision map point and the position T of the radar / video sensor on the high-precision map are optimized in stages, and the optimal north angle and longitude and latitude position of the radar / video sensor are calculated respectively;

[0063] Stage 1: Fix the position T on the high-precision map and use the least squares method to solve the following equation:

[0064]

[0065] In the formula, LINE k is the slope of the line segment in the radar tracking target frequency diagram, P i+1_x , P i+1_y Respectively represent the latitude and longitude of the i+1th longitude and latitude coordinate point in the high-precision map coordinates;

[0066] Stage 2: Fix the north angle optimized in stage 1, optimize the position T on the high-precision map, and use the least squares method to solve the following formula:

[0067]

[0068] In the formula, LINE b Represents the world coordinate mapping point set rset The intercept of the line segment from the nearest midpoint to the farthest midpoint.

[0069] Furthermore, the transformation relationship from the video sensor to the earth coordinate system in step 3 is:

[0070]

[0071] In the formula, x world and world They represent the horizontal and vertical coordinates in the geodetic coordinate system respectively.

[0072] Furthermore, step 4 specifically includes:

[0073] Step 4-1, establishing a target detection result set, including several fusion targets, wherein the attribute information of the fusion target includes video signal ID, radar signal ID, world coordinates, speed, category, yaw angle, and whether it is blocked;

[0074] Step 4-2, calculate the confidence of the target detected in a single frame of video;

[0075] Step 4-3, based on the confidence level, determines the target occlusion through multiple frames of video information, specifically including:

[0076] The multi-frame confidence of the target tracked by the video sensor is formalized as {T1_conf, T2_conf, T3_conf, ..., Tn_Conf}, where Tn_Conf represents the confidence at time Tn;

[0077] For each tracking target, traverse from time Tn to time T1. If the confidence of m consecutive frames is lower than the preset threshold, the tracking target is determined as an occluded target.

[0078] Step 4-4, associate the first level ID of the radar target, including:

[0079] The ID of the radar / video sensor is compared and matched with the ID in the attribute corresponding to each fused target in the target detection result set. If they match, the attribute information of the corresponding fused target is replaced with the data information of the radar / video sensor.

[0080] Step 4-5, verifying the obstructed target through radar data, specifically includes:

[0081] Count the unmatched fusion target set in the first-level ID association in step 4-4;

[0082] Using radar data to verify the occluded targets of the fused target set, specifically: performing Hungarian matching in the video coordinate system based on the similarity of position and velocity information, if there is a match, it means that the verification is passed, otherwise it means that the verification fails;

[0083] For targets that pass the verification, execute steps 4-6; for targets that fail the verification, execute steps 4-7;

[0084] Step 4-6, perform the second level motion feature association: motion feature matching is divided into matching between fusion target and radar observation, and matching between fusion target and video observation. The similarity based on position and speed is adopted to match in the video coordinate system and radar coordinate system respectively.

[0085] Step 4-7, perform single video IoU association: in the video coordinate system, association is performed based on the predicted and observed IoU distances;

[0086] Step 4-8, if the radar / video sensor observation is not associated with the fusion target after the above two levels of matching, then a new fusion target is created using the observation and added to the target detection result set;

[0087] Step 4-9: Output the target detection result set.

[0088] Furthermore, step 4 also includes, after steps 4-8, executing:

[0089] If the fused target does not match the video / radar observation within the preset q frames, the fused target is erased.

[0090] Compared with the prior art, the present invention has the following significant advantages:

[0091] (1) The use of high-frequency radar tracking of target trajectories and high-precision map data reduces the tedious process of calibration between radar sensors and between the radar sensors and the world coordinate system. At the same time, it reduces the accuracy error of coordinate transformation between radar sensors and world coordinate transformation of radar data. Compared with the existing technology, it is easier to operate, more accurate and more robust in data association in complex traffic roadside vehicle detection scenarios.

[0092] (2) By using the high-frequency trajectory data of radar vision and analyzing it, the radar vision sensor can be calibrated quickly and accurately.

[0093] (3) By utilizing high-precision map data and an automatic correction algorithm based on linear regression, accurate mapping of the sensor to the world coordinate system can be achieved.

[0094] (4) By using multi-frame video occlusion analysis and millimeter-wave radar data verification, the dynamic and static scene advantages of radar sensors and video sensors can be fully utilized to perform data association.

[0095] (5) Based on high-frequency trajectory data and high-precision map data, the sensor coordinate system transformation and sensor-to-world coordinate transformation are automatically corrected, and the radar-visual correlation method with occlusion verification is adopted to finally provide high-precision fusion data. Compared with the existing solutions, the complexity of manual calibration and calibration errors are reduced, the accuracy of automatic calibration is improved, the characteristics of radar and video data are more fully explored, and accurate and real-time multi-target detection results are provided.

[0096] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 Schematic diagram of a flow chart of a target detection method based on multi-source high-frequency trajectories and high-precision maps in one embodiment.

[0098] Figure 2 Schematic diagram of automatic coordinate conversion between radar sensors in one embodiment.

[0099] Figure 3 A schematic diagram of a millimeter wave radar tracking target trajectory in one embodiment.

[0100] Figure 4 A schematic diagram of a video sensor tracking a target trajectory in an embodiment.

[0101] Figure 5 Schematic diagram of obtaining a lane point set under a high-precision map in one embodiment.

[0102] Figure 6 Schematic diagram of radar lane line extraction and high-precision map point regression in one embodiment.

[0103] Figure 7 Schematic diagram of video lane curve and high-precision map point regression in one embodiment.

[0104] Figure 8 Schematic diagram of sensor fusion results in one embodiment. DETAILED DESCRIPTION

[0105] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0106] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, left, right, front, back, etc.), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0107] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0108] In one embodiment, in combination Figure 1 , a multi-target detection method based on multi-source high-frequency trajectories and high-precision maps is provided, the method comprising:

[0109] Step 1, Combine Figure 2 , automatic coordinate conversion between radar sensors: synchronize video data and radar data, extract features of video targets and radar targets, track video targets and radar targets, count the frequency graphs of radar and video tracking targets, and then use the seed filling method for analysis to obtain the effective detection area of ​​the sensor. Based on the statistical analysis of the high-frequency trajectory of the radar data, high-frequency point pairs are automatically selected to perform automatic calibration between the radar and video sensors;

[0110] Step 2: Calculate the north angle of the radar sensor and automatically correct it: Select the latitude and longitude of the initial position of the device and the lane point set on the high-precision map, outline the polyline corresponding to the detection area in the radar coordinate system, calculate the north angle of each sensor, and automatically correct the north angle;

[0111] Step 3: Based on step 1 and step 2, transform the video sensor to the earth coordinate system;

[0112] Step 4: Perform decision association of the radar sensor: Use multi-frame feature information of the video to determine target occlusion, and use motion feature association for non-occluded targets based on multi-level matching of occluded and non-occluded data. Use radar data to verify occluded targets. Use video IoU association for targets that fail the verification, and use radar observation data for targets that pass the verification.

[0113] The present invention uses an 80Ghz millimeter-wave radar and a 2-megapixel monocular camera as sensors, which can simultaneously obtain the road target position coordinate data relative to the ground, the target's speed data, and the pixel information of the target and the scene; the edge computing processor is selected as a 4-core ARM Cortex-A7.

[0114] In an embodiment of the present invention, radar target information is structured information, including speed, position, and state. The convolutional model used for target detection and recognition in video frames is Darknet, combined with the Yolov3-tiny algorithm. The identified targets include car, bus, bike, and truck, and the TensorRT acceleration model is used to accelerate reasoning in target detection.

[0115] Further, in one embodiment, the formula for data synchronization in step 1 is:

[0116]

[0117] Where N is the data synchronization frequency in frames per second, T is the radar update time, and INT represents an integer.

[0118] Furthermore, in one of the embodiments, in step 1, a Yolo deep learning network is used to extract the target pixel position and target confidence in the video frame, and a point cloud clustering method is used to analyze the original point cloud data collected by the radar to obtain the target position information and speed information.

[0119] Further, in one embodiment, in step 1, tracking the video target and the radar target based on the SORT algorithm specifically includes:

[0120] (1) Use Kalman filtering to predict radar / video targets separately:

[0121]

[0122] The tracking target at time t-1 and time t is a 6-dimensional vector [X t-1 ,Y t-1 ,Vx t-1 ,Vy t-1 ,ax t-1 ,ay t-1 ]、[X t ,Y t ,Vx t ,Vy t ,ax t ,ay t ], where X t-1 , X t They represent the x-coordinates and Y-coordinates of the radar / video coordinate system at time t-1 and time t, respectively. t-1 , Y t Respectively represent the y-direction coordinates of the radar / video coordinate system at time t-1 and time t, Vx t-1 、Vx t Respectively represent the x-direction velocity in the radar / video coordinate system at time t-1 and time t, Vyt-1 , Vy t Respectively represent the y-direction speed in the radar / video coordinate system at time t-1 and time t, ax t-1 ,ax t They represent the lateral acceleration at time t-1 and time t respectively, and ay t-1 、ay t They represent the longitudinal acceleration at time t-1 and time t respectively; Indicates the difference between the timestamps of the previous and next two frames of radar or video;

[0123] (2) The prediction results and radar / video detection results are associated with each other through the Hungarian matching method, as shown in the following formula. The covariance of the historical data of the tracking target is statistically analyzed to construct the Mahalanobis distance between the prediction and all observations. The maximum match between the prediction and the observation is found through the Hungarian matching method.

[0124]

[0125] Further, in one embodiment, the statistical frequency graph of the tracking target of the radar and the video in step 1 is then analyzed using a seed filling method to obtain an effective detection area of ​​the sensor, and based on the statistical analysis of the high-frequency trajectory of the radar data, high-frequency point pairs are automatically selected to perform automatic calibration between the radar and video sensors, specifically including:

[0126] Step 1-1, transform the radar data (x1, y1) coordinates to the canvas coordinates (pt) of size (H, W) X1 ,pt Y1 ), the transformation formula is:

[0127] pt X1 =20+(x2+30) / 60.0*(W-40)

[0128] pt Y2 =(H-20)-(y2-0) / 300.0*(H-40)

[0129] Step 1-2, transform the video detection center point coordinates (x2, y2) to the canvas coordinates (pt) of size (H, W) X2 ,pt Y2 ), the transformation formula is:

[0130] pt X2 =x2 / frameWidth*W

[0131] pt Y2 =y2 / frameHeight*H

[0132] Where frameWidth and frameHeight represent the width and height of the video frame used for model inference, respectively, and W and H represent the width and height of the canvas, respectively;

[0133] Step 1-3: Calculate the heat map of historical trajectories of radar and video data, and assign values ​​to the frequency values ​​near the canvas coordinates calculated in steps 1-1 and 1-2. The rules are as follows:

[0134]

[0135] In the formula, Indicates pt X1 or pt X2 ,pt Y1 or pt Y2 , Near(ptX*,PtY*) is a point The coordinate values ​​of the N×N neighborhood at , where N is an odd number; It represents the frequency value corresponding to Near(ptX*,PtY*), Vx and Vy represent the lateral speed and longitudinal speed respectively;

[0136] The frequency of the multi-time radar tracking target set is counted, the frequency is distinguished according to the radial velocity, the neighborhood frequency is assigned, and the radar tracking target frequency map of H×W×1 size and the video tracking target frequency map of H×W×1 size are generated;

[0137] Steps 1-4, normalize the frequency graph and make the normalized result nonlinear. The calculation formula is:

[0138]

[0139] Where Value(ptX*,ptY*) represents the frequency value at the point (ptX*,ptY*), and nonLinearR is the final nonlinear result;

[0140] Steps 1-5, visualize the frequency graph;

[0141] Preferably, the number of currently recorded tracking targets is counted, the scale factor sca is calculated according to the current number of targets, and nonLinearR is normalized and other operations are performed. The calculation process is as follows:

[0142]

[0143]

[0144] Convert the result to RGB value for visualization.

[0145] Steps 1-6 generate radar and video target trajectory maps respectively: convert the video and radar tracking targets to the H×W range, and record the coordinate values ​​of the target at each sampling time, and store them in the form of HxWxT, where T represents the sampling time. Figure 3 For the overlay visualization of the radar track map, Figure 4 Overlay visualization of the video trajectory graph.

[0146] Steps 1-7, based on the seed filling method, analyze the radar and video tracking target frequency graphs to automatically obtain the effective detection area of ​​the radar vision sensor, including:

[0147] A foreground point in the frequency graph is selected as a seed, and then the foreground points adjacent to the seed are merged into the same set according to the position proximity condition of the connected area. The final set is a connected area.

[0148] By analyzing the tracking target frequency graph of radar and video in the above manner, the radar effective detection area DETAradar and the video sensor effective detection area DETAcamera are obtained;

[0149] Step 1-8, automatically select the high-frequency coordinate point pairs of radar vision in the effective monitoring area, specifically including:

[0150] Filter the high-frequency point set greater than the preset threshold in the radar effective detection area DETAradar, and filter the high-frequency point set greater than the preset threshold in the video sensor effective detection area DETAcamera;

[0151] Sort the two high-frequency point sets in the Cartesian coordinate x and y directions respectively, and select the high-frequency points with the maximum and minimum values ​​in the x and y directions;

[0152] According to the time t corresponding to the radar / video raw data at the four high-frequency points and the coordinates on the frequency graph, the corresponding target attributes of the video and radar target trajectory graphs are extracted respectively as the corresponding points between the radar and video sensors;

[0153] Step 1-9, calculate the transformation matrix between the radar and video sensors, specifically including:

[0154] Substitute the four sets of coordinates obtained in steps 1-8 into the following formula to solve the transformation matrix between the radar and video sensor:

[0155]

[0156] In the formula, x radar and radar Respectively represent the horizontal and vertical coordinates in the radar coordinate system, x img and imgThey represent the horizontal and vertical coordinates in the video coordinate system respectively, and M1 is the perspective transformation matrix, that is, the transformation matrix between the radar and the video sensor.

[0157] Furthermore, in one embodiment, step 2 uses a multi-linear regression method to calculate and automatically correct the north angle based on the high-precision map and the radar sensor data. Figure 6 This is the radar view regression effect. This step specifically includes:

[0158] Step 2-1, select the initial device longitude and latitude in the high-precision map, such as Figure 5 As shown, the device includes a radar and a video sensor, and simultaneously selects the longitude and latitude coordinate points {P1, P2, P3, ..., Pn} of a specific lane from near to far, Pi represents the i-th longitude and latitude coordinate point, i = 1, 2, ..., n; outlines the corresponding point selection line segment LINEradar in the radar tracking target frequency diagram, such as Figure 6 As shown;

[0159] Step 2-2, construct the translation matrix T and rotation matrix R from the radar coordinate system to the world coordinate system:

[0160]

[0161] in,

[0162]

[0163] Where LAT Pi Indicates the latitude of the high-precision map point Pi, LON Pi Indicates the longitude of the high-precision map point Pi, LAT radar and LON radar They represent the latitude and longitude of the radar equipment used in the high-precision map, respectively. θ is the angle between the normal of the millimeter-wave radar and the north direction of the geodetic coordinate system, i.e., the north angle.

[0164] Step 2-3, based on the translation matrix T and the rotation matrix R, inversely transform the high-precision map coordinates to the radar coordinate system to obtain the world coordinate mapping point set r corresponding to {P1, P2, P3, ..., Pn} set , where the coordinate correspondence is:

[0165]

[0166] In the formula, r i_x 、r i_y Respectively represent the latitude and longitude of the i-th longitude and latitude coordinate point Pi in the radar coordinate system, P i_x , P i_y Respectively represent the latitude and longitude of the i-th longitude and latitude coordinate point Pi in the high-precision map coordinates;

[0167] Step 2-4, calculate the world coordinate mapping point set r set The error between the line segment formed by the midpoint from near to far and the line segment outlined in step 2-1 is used to calculate the optimal north angle and the longitude and latitude position of the radar / video sensor.

[0168] Preferably, in some embodiments, in steps 2-4, the north angle θ of the high-precision map point and the position T of the radar / video sensor on the high-precision map are optimized in stages, and the optimal north angle and longitude and latitude position of the radar / video sensor are calculated respectively;

[0169] Stage 1: Fix the position T on the high-precision map and use the least squares method to solve the following equation:

[0170]

[0171] In the formula, LINE k is the slope of the line segment in the radar tracking target frequency diagram, P i+1_x , P i+1_y Respectively represent the latitude and longitude of the i+1th longitude and latitude coordinate point in the high-precision map coordinates;

[0172] Stage 2: Fix the north angle optimized in stage 1, optimize the position T on the high-precision map, and use the least squares method to solve the following formula:

[0173]

[0174] In the formula, LINE b Represents the world coordinate mapping point set r set The intercept of the line segment from the nearest midpoint to the farthest midpoint.

[0175] Furthermore, in one embodiment, the transformation relationship from the video sensor to the earth coordinate system in step 3 is:

[0176]

[0177] In the formula, x world and world They represent the horizontal and vertical coordinates in the geodetic coordinate system respectively.

[0178] like Figure 7 The figure shows the point set mapped by the high-precision map and the marking effect mapped by the radar coordinate system.

[0179] Further, in one embodiment, step 4 specifically includes:

[0180] Step 4-1, establishing a target detection result set, including several fusion targets, wherein the attribute information of the fusion target includes video signal ID, radar signal ID, world coordinates, speed, category, yaw angle, and whether it is blocked;

[0181] Step 4-2, calculate the confidence of the target detected in a single frame of video;

[0182] Step 4-3, based on the confidence level, determines the target occlusion through multiple frames of video information, specifically including:

[0183] The multi-frame confidence of the target tracked by the video sensor is formalized as {T1_conf, T2_conf, T3_conf, ..., Tn_Conf}, where Tn_Conf represents the confidence at time Tn;

[0184] For each tracking target, traverse from time Tn to time T1. If the confidence of m consecutive frames is lower than the preset threshold, the tracking target is determined to be an occluded target; preferably, m is 5 frames;

[0185] Step 4-4, associate the first level ID of the radar target, including:

[0186] The ID of the radar / video sensor is compared and matched with the ID in the attribute corresponding to each fused target in the target detection result set. If they match, the attribute information of the corresponding fused target is replaced with the data information of the radar / video sensor.

[0187] Step 4-5, verifying the obstructed target through radar data, specifically includes:

[0188] Count the unmatched fusion target set in the first-level ID association in step 4-4;

[0189] Using radar data to verify the occluded targets of the fused target set, specifically: performing Hungarian matching in the video coordinate system based on the similarity of position and velocity information, if there is a match, it means that the verification is passed, otherwise it means that the verification fails;

[0190] For targets that pass the verification, execute steps 4-6; for targets that fail the verification, execute steps 4-7;

[0191] Step 4-6, perform the second level motion feature association: motion feature matching is divided into matching between fusion target and radar observation, and matching between fusion target and video observation. The similarity based on position and speed is adopted to match in the video coordinate system and radar coordinate system respectively.

[0192] Step 4-7, perform single video IoU association: in the video coordinate system, association is performed based on the predicted and observed IoU distances;

[0193] Step 4-8, if the radar / video sensor observation is not associated with the fusion target after the above two levels of matching, then a new fusion target is created using the observation and added to the target detection result set;

[0194] Step 4-9: Output the target detection result set.

[0195] Furthermore, in some embodiments, step 4 further includes, after steps 4-8, executing:

[0196] If the fused target does not match the video / radar observation within the preset q frames, the fused target is erased. Preferably, q is 10 frames.

[0197] Figure 8 This is the correlation effect of the radar data decision of the present invention.

[0198] Preferably, in some embodiments, the confidence of the target in step 4-2 is determined by two factors, one is the probability that the pixel is a foreground target, and the other is the probability of the category to which it belongs. The present invention uses the product of the two as the confidence of the target detected in a single frame video.

[0199] In the final output of the YoloV3 network, three bounding boxes are generated for each grid cell. The output of each bounding box has three types of parameters: one is the box parameter of the object, which is a total of four values, namely the coordinates of the center point of the box (x, y) and the width and height of the box; one is the confidence between [0,1]; and the last one is a set of conditional category probabilities. Assuming that an image is divided into SxS grid cells and there are B anchor boxes, the final output dimension is: SxSx(Bx(4+1+classes)). The network output result can be used to solve the confidence Conf_obj of the target detected in a single frame of video.

[0200] In one embodiment, a multi-target detection system based on multi-source high-frequency trajectories and high-precision maps is provided, the system comprising:

[0201] The first module is used for automatic coordinate conversion between radar sensors: synchronizing video data and radar data, extracting features of video targets and radar targets, tracking video targets and radar targets, and statistically analyzing the frequency graphs of radar and video tracking targets. After that, the seed filling method is used for analysis to obtain the effective detection area of ​​the sensor. Based on the statistical analysis of the high-frequency trajectory of the radar data, high-frequency point pairs are automatically selected to perform automatic calibration between the radar and video sensors.

[0202] The second module is used to calculate the north angle of the radar sensor and automatically correct it: select the latitude and longitude of the initial position of the device and the lane point set on the high-precision map, outline the polyline corresponding to the detection area in the radar coordinate system, calculate the north angle of each sensor, and automatically correct the north angle;

[0203] The third module is used to transform the video sensor into the earth coordinate system;

[0204] The fourth module is used for decision association of radar vision sensors: using multi-frame feature information of video to determine target occlusion, based on multi-level matching of occlusion and non-occlusion data, non-occluded targets are associated with motion features, and radar data is used to verify occluded targets. Targets that fail the verification are associated with video IoU, and targets that pass the verification are associated with radar observation data.

[0205] For the specific limitations of the multi-target detection system based on multi-source high-frequency trajectories and high-precision maps, please refer to the limitations of the multi-target detection method based on multi-source high-frequency trajectories and high-precision maps above, which will not be repeated here. Each module in the above-mentioned multi-target detection system based on multi-source high-frequency trajectories and high-precision maps can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0206] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following is achieved:

[0207] Step 1: Automatic coordinate conversion between radar sensors: synchronize video data and radar data, extract features of video targets and radar targets, track video targets and radar targets, count radar and video tracking target frequency graphs, and then use seed filling method for analysis to obtain the effective detection area of ​​the sensor. Based on the statistical analysis of high-frequency trajectories of radar data, automatically select high-frequency point pairs to perform automatic calibration between radar and video sensors.

[0208] Step 2: Calculate the north angle of the radar sensor and automatically correct it: Select the latitude and longitude of the initial position of the device and the lane point set on the high-precision map, outline the polyline corresponding to the detection area in the radar coordinate system, calculate the north angle of each sensor, and automatically correct the north angle;

[0209] Step 3: Based on step 1 and step 2, transform the video sensor to the earth coordinate system;

[0210] Step 4: Perform decision association of the radar sensor: Use multi-frame feature information of the video to determine target occlusion, and use motion feature association for non-occluded targets based on multi-level matching of occluded and non-occluded data. Use radar data to verify occluded targets. Use video IoU association for targets that fail the verification, and use radar observation data for targets that pass the verification.

[0211] For the specific limitations of each step, please refer to the limitations of the multi-target detection method based on multi-source high-frequency trajectories and high-precision maps mentioned above, which will not be repeated here.

[0212] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the computer program implements:

[0213] Step 1: Automatic coordinate conversion between radar sensors: synchronize video data and radar data, extract features of video targets and radar targets, track video targets and radar targets, count radar and video tracking target frequency graphs, and then use seed filling method for analysis to obtain the effective detection area of ​​the sensor. Based on the statistical analysis of high-frequency trajectories of radar data, automatically select high-frequency point pairs to perform automatic calibration between radar and video sensors.

[0214] Step 2: Calculate the north angle of the radar sensor and automatically correct it: Select the latitude and longitude of the initial position of the device and the lane point set on the high-precision map, outline the polyline corresponding to the detection area in the radar coordinate system, calculate the north angle of each sensor, and automatically correct the north angle;

[0215] Step 3: Based on step 1 and step 2, transform the video sensor to the earth coordinate system;

[0216] Step 4: Perform decision association of the radar sensor: Use multi-frame feature information of the video to determine target occlusion, and use motion feature association for non-occluded targets based on multi-level matching of occluded and non-occluded data. Use radar data to verify occluded targets. Use video IoU association for targets that fail the verification, and use radar observation data for targets that pass the verification.

[0217] For the specific limitations of each step, please refer to the limitations of the multi-target detection method based on multi-source high-frequency trajectories and high-precision maps mentioned above, which will not be repeated here.

[0218] The present invention utilizes high-frequency radar tracking of target trajectories and high-precision map data, which reduces the tediousness of calibration between sensors and between sensors and the world coordinate system, while also reducing the accuracy error of coordinate transformation between radar sensors and world coordinate transformation of radar data. Compared with the existing technology, the present invention has higher accuracy and more robust data association in complex traffic roadside vehicle detection scenario applications.

[0219] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-target detection method based on multi-source high-frequency trajectories and high-precision maps, characterized in that: The method comprises: Step 1: Automatic coordinate conversion between radar sensors: synchronize video data and radar data, extract features of video targets and radar targets, track video targets and radar targets, count radar and video tracking target frequency graphs, and then use seed filling method for analysis to obtain the effective detection area of ​​the sensor. Based on the statistical analysis of high-frequency trajectories of radar data, automatically select high-frequency point pairs to perform automatic calibration between radar and video sensors. Step 2: Calculate the north angle of the radar sensor and automatically correct it: Select the latitude and longitude of the initial position of the device and the lane point set on the high-precision map, outline the polyline corresponding to the detection area in the radar coordinate system, calculate the north angle of each sensor, and automatically correct the north angle; Step 3: Based on step 1 and step 2, transform the video sensor to the earth coordinate system; Step 4: Perform decision association of the radar sensor: Use multi-frame feature information of the video to determine target occlusion, and use motion feature association for non-occluded targets based on multi-level matching of occluded and non-occluded data. Use radar data to verify occluded targets. Use video IoU association for targets that fail the verification, and use radar observation data for targets that pass the verification.

2. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 1 is characterized in that: The formula for data synchronization in step 1 is: Where N is the data synchronization frequency in frames per second, T is the radar update time, and INT represents an integer.

3. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 1 is characterized in that: In step 1, the Yolo deep learning network is used to extract the target pixel position and target confidence in the video frame, and the point cloud clustering method is used to analyze the original point cloud data collected by the radar to obtain the target position information and speed information.

4. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 1 is characterized in that: In step 1, the video target and the radar target are tracked based on the SORT algorithm, which includes: (1) Use Kalman filtering to predict radar / video targets separately: The tracking target at time t-1 and time t is a 6-dimensional vector [X t-1 ,Y t-1 ,Vx t-1 ,Vy t-1 ,ax t-1 ,ay t-1 ]、[X t ,Y t ,Vx t ,Vy t ,ax t ,ay t ], where X t-1 , X t They represent the x-coordinates and Y-coordinates of the radar / video coordinate system at time t-1 and time t, respectively. t-1 , Y t Respectively represent the y-direction coordinates of the radar / video coordinate system at time t-1 and time t, Vx t-1 、Vx t Respectively represent the x-direction velocity in the radar / video coordinate system at time t-1 and time t, Vy t-1 , Vy t Respectively represent the y-direction speed in the radar / video coordinate system at time t-1 and time t, ax t-1 ,ax t They represent the lateral acceleration at time t-1 and time t respectively, and ay t-1 、ay t They represent the longitudinal acceleration at time t-1 and time t respectively; Indicates the difference between the timestamps of the previous and next two frames of radar or video; (2) The prediction results and radar / video detection results are associated respectively through the Hungarian matching method.

5. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 1 is characterized in that: The frequency graph of tracking targets of radar and video is statistically analyzed in step 1, and then the seed filling method is used for analysis to obtain the effective detection area of ​​the sensor. Based on the statistical analysis of the high-frequency trajectory of the radar data, high-frequency point pairs are automatically selected to perform automatic calibration between the radar and video sensors, which specifically includes: Step 1-1, transform the radar data (x1, y1) coordinates to the canvas coordinates (pt) of size (H, W) X1 ,pt Y1 ), the transformation formula is: pt X1 =20+(x2+30) / 60.0*(W-40) pt Y2 =(H-20)-(y2-0) / 300.0*(H-40) Step 1-2, transform the video detection center point coordinates (x2, y2) to the canvas coordinates (pt) of size (H, W) X2 ,pt Y2 ), the transformation formula is: <h2 style=";text-align:left;direction:ltr">pt<h2 style=";text-align:left;direction:ltr"> X2 <h2 style=";text-align:left;direction:ltr"> =x2 / frameWidth*W pt Y2 =y2 / frameHeight*H Where frameWidth and frameHeight represent the width and height of the video frame used for model inference, respectively, and W and H represent the width and height of the canvas, respectively; Step 1-3: Calculate the heat map of historical trajectories of radar and video data, and assign values ​​to the frequency values ​​near the canvas coordinates calculated in steps 1-1 and 1-2. The rules are as follows: In the formula, Indicates pt X1 or pt X2 ,pt Y1 or pt Y2 , Near(ptX*,PtY*) is a point The coordinate values ​​of the N×N neighborhood at , where N is an odd number; It represents the frequency value corresponding to Near(ptX*,PtY*), Vx and Vy represent the lateral speed and longitudinal speed respectively; The frequency of the multi-time radar tracking target set is counted, the frequency is distinguished according to the radial velocity, the neighborhood frequency is assigned, and the radar tracking target frequency map of H×W×1 size and the video tracking target frequency map of H×W×1 size are generated; Steps 1-4, normalize the frequency graph and make the normalized result nonlinear. The calculation formula is: Where Value(ptX*,ptY*) represents the frequency value at the point (ptX*,ptY*), and nonLinearR is the final nonlinear result; Steps 1-5, visualize the frequency graph; Steps 1-6, generate radar and video target trajectory diagrams respectively: convert the video and radar tracking targets to the H×W range, and record the coordinate values ​​of the targets at each sampling moment; Steps 1-7, based on the seed filling method, analyze the radar and video tracking target frequency graphs to automatically obtain the effective detection area of ​​the radar vision sensor, including: A foreground point in the frequency graph is selected as a seed, and then the foreground points adjacent to the seed are merged into the same set according to the position proximity condition of the connected area. The final set is a connected area. By analyzing the tracking target frequency graph of radar and video in the above manner, the radar effective detection area DETAradar and the video sensor effective detection area DETAcamera are obtained; Step 1-8, automatically select the high-frequency coordinate point pairs of radar vision in the effective monitoring area, specifically including: Filter the high-frequency point set greater than the preset threshold in the radar effective detection area DETAradar, and filter the high-frequency point set greater than the preset threshold in the video sensor effective detection area DETAcamera; Sort the two high-frequency point sets in the Cartesian coordinate x and y directions respectively, and select the high-frequency points with the maximum and minimum values ​​in the x and y directions; According to the time t corresponding to the radar / video raw data at the four high-frequency points and the coordinates on the frequency graph, the corresponding target attributes of the video and radar target trajectory graphs are extracted respectively as the corresponding points between the radar and video sensors; Step 1-9, calculate the transformation matrix between the radar and video sensors, specifically including: Substitute the four sets of coordinates obtained in steps 1-8 into the following formula to solve the transformation matrix between the radar and video sensor: In the formula, x radar and radar Respectively represent the horizontal and vertical coordinates in the radar coordinate system, x img and img They represent the horizontal and vertical coordinates in the video coordinate system respectively, and M1 is the perspective transformation matrix, that is, the transformation matrix between the radar and the video sensor.

6. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 1 is characterized in that: Step 2: Based on the high-precision map and radar sensor data, multi-linear regression is used to calculate and automatically correct the north angle, including: Step 2-1, select the initial equipment longitude and latitude in the high-precision map, the equipment includes radar and video sensors, and select the longitude and latitude coordinate points {P1, P2, P3, ..., Pn} from near to far in a specific lane, Pi represents the i-th longitude and latitude coordinate point, i = 1, 2, ..., n; outline the corresponding point selection line segment LINEradar in the radar tracking target frequency diagram; Step 2-2, construct the translation matrix T and rotation matrix R from the radar coordinate system to the world coordinate system: in, Where LAT Pi Indicates the latitude of the high-precision map point Pi, LON Pi Indicates the longitude of the high-precision map point Pi, LAT radar and LON radar They represent the latitude and longitude of the radar equipment used in the high-precision map, respectively. θ is the angle between the normal of the millimeter-wave radar and the north direction of the geodetic coordinate system, i.e., the north angle. Step 2-3, based on the translation matrix T and the rotation matrix R, inversely transform the high-precision map coordinates to the radar coordinate system to obtain the world coordinate mapping point set r corresponding to {P1, P2, P3, ..., Pn} set , where the coordinate correspondence is: In the formula, r i_x 、r i_y Respectively represent the latitude and longitude of the i-th longitude and latitude coordinate point Pi in the radar coordinate system, P i_x , P i_y Respectively represent the latitude and longitude of the i-th longitude and latitude coordinate point Pi in the high-precision map coordinates; Step 2-4, calculate the world coordinate mapping point set r set The error between the line segment formed by the midpoint from near to far and the line segment outlined in step 2-1 is used to calculate the optimal north angle and the longitude and latitude position of the radar / video sensor.

7. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 6 is characterized in that: In steps 2-4, the north angle θ of the HD map point and the position T of the radar / video sensor on the HD map are optimized in stages, and the optimal north angle and longitude and latitude position of the radar / video sensor are calculated respectively; Stage 1: Fix the position T on the high-precision map and use the least squares method to solve the following equation: In the formula, LINE k is the slope of the line segment in the radar tracking target frequency diagram, P i+1_x , P i+1_y Respectively represent the latitude and longitude of the i+1th longitude and latitude coordinate point in the high-precision map coordinates; Stage 2: Fix the north angle optimized in stage 1, optimize the position T on the high-precision map, and use the least squares method to solve the following formula: In the formula, LINE b Represents the world coordinate mapping point set r set The intercept of the line segment from the nearest midpoint to the farthest midpoint.

8. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 5 or 6, characterized in that: The transformation relationship from the video sensor to the earth coordinate system in step 3 is: In the formula, x world and world They represent the horizontal and vertical coordinates in the geodetic coordinate system respectively.

9. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 1 is characterized in that: Step 4 specifically includes: Step 4-1, establishing a target detection result set, including several fusion targets, wherein the attribute information of the fusion target includes video signal ID, radar signal ID, world coordinates, speed, category, yaw angle, and whether it is blocked; Step 4-2, calculate the confidence of the target detected in a single frame of video; Step 4-3, based on the confidence level, determines the target occlusion through multiple frames of video information, specifically including: The multi-frame confidence of the target tracked by the video sensor is formalized as {T1_conf, T2_conf, T3_conf, ..., Tn_Conf}, where Tn_Conf represents the confidence at time Tn; For each tracking target, traverse from time Tn to time T1. If the confidence of m consecutive frames is lower than the preset threshold, the tracking target is determined as an occluded target. Step 4-4, associate the first level ID of the radar target, including: The ID of the radar / video sensor is compared and matched with the ID in the attribute corresponding to each fused target in the target detection result set. If they match, the attribute information of the corresponding fused target is replaced with the data information of the radar / video sensor. Step 4-5, verifying the obstructed target through radar data, specifically includes: Count the unmatched fusion target set in the first-level ID association in step 4-4; Using radar data to verify the occluded targets of the fused target set, specifically: performing Hungarian matching in the video coordinate system based on the similarity of position and velocity information, if there is a match, it means that the verification is passed, otherwise it means that the verification fails; For targets that pass the verification, execute steps 4-6; for targets that fail the verification, execute steps 4-7; Step 4-6, perform the second level motion feature association: motion feature matching is divided into matching between fusion target and radar observation, and matching between fusion target and video observation. The similarity based on position and speed is adopted to match in the video coordinate system and radar coordinate system respectively. Step 4-7, perform single video IoU association: in the video coordinate system, association is performed based on the predicted and observed IoU distances; Step 4-8, if the radar / video sensor observation is not associated with the fusion target after the above two levels of matching, then a new fusion target is created using the observation and added to the target detection result set; Step 4-9: Output the target detection result set.

10. The multi-target detection method based on multi-source high-frequency trajectories and high-precision maps according to claim 9 is characterized in that: Step 4 also includes, after steps 4-8, executing: If the fused target does not match the video / radar observation within the preset q frames, the fused target is erased.

Images