A radar and visual automatic calibration method based on trajectory tracing

Abstract

The application discloses a radar and video automatic calibration method based on trajectory tracing, which obtains images through a video monitoring device and inputs the images into a convolution model to identify targets and obtain state information of each detection target; an initial transformation matrix is obtained by correcting the images of the target detection area; multiple target state information of multiple frames of radars is taken and trajectory statistics is performed; lane information is inversely deduced by analyzing the trajectory information of video targets and radar targets; two coordinate systems are associated according to the lane information and the longitudinal relative position information of the targets, the module area is divided according to the lane information, calculation is performed according to the associated information and the module area, and a radar and video perspective transformation matrix is obtained. According to the lane information of the trajectory tracing detection target, and according to the relative positions of the targets before and after, two targets of different coordinate systems are accurately associated, the shortcomings of low efficiency, narrow range and point drift of manual calibration point selection are improved, and fast and efficient automatic calibration is realized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent traffic detection, especially the field of multi-view area correction, and particularly relates to a radar-based automatic correction method based on trajectory tracing. Background Technology

[0002] In recent years, the increasing number of various modes of transportation has led to road saturation. Consequently, the importance of "smart city" construction has risen, making real-time monitoring and control of urban road traffic particularly crucial. Road detectors combining radar and video have emerged to address the limitations of single detectors due to environmental constraints. However, the key to radar-video detectors lies in the fusion of information from both, and accurate correlation between their coordinate systems is a prerequisite for accurate fusion.

[0003] The existing correlation methods are basically as follows: acquire radar and image acquisition videos over a period of time, select target point pairs through human observation and manual point selection, and then calculate the perspective transformation matrix. This method has the following drawbacks: it is more accurate for selecting points for a small number of targets, but when multiple targets are involved, due to different detection perspectives and the radar's inaccurate detection of stationary targets, the number of target points cannot be matched, making it impossible to accurately perform point pair correspondence; when manually clicking to select points, point pair offsets are prone to occur, causing errors in the solution of the coordinate plane; the solution of the perspective transformation matrix is ​​obtained by fitting multiple sets of point pairs, and the point pairs are required to be non-collinear, which cannot be fully covered manually, resulting in unsatisfactory calibration results, repeated calibration, wasted time, and inaccurate results. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of manual calibration by providing an automatic radar / visual calibration method based on trajectory tracing. This method employs a deep learning-based detection approach for video target detection, uses image region correction to prepare for image trajectory tracing, performs image segmentation on the target trajectory information from both radar and video to inversely deduce target lane information, associates the relative positions of targets within the same lane between radar and video, sets thresholds for lane sub-module regions, and calculates the corresponding perspective transformation matrix. This invention ensures full coverage of associated points, non-drifting of selected point positions, and accurate point-to-point correspondence, achieving automated, efficient, and highly accurate radar / visual calibration.

[0005] The technical solution to achieve the purpose of this invention is: an automatic calibration method for radar based on trajectory tracing, the method comprising the following steps:

[0006] Step 1: Collect radar target detection point status information, including position, velocity, and status;

[0007] Step 2: Acquire video images from the radar simultaneously, load the target detection model, and input the video images into the target detection model to perform target detection, and obtain the target status information, including location, category, and size;

[0008] Step 3: Select the video detection area, use perspective transformation to correct the detection area to obtain the first transformation matrix, and use the perspective transformation matrix to transform the video detection target to the first correction coordinate system;

[0009] Step 4: Statistically analyze the video and radar target trajectories within the threshold time t, analyze the radar and video target trajectory maps respectively to perform lane inversion, calculate the lane baseline and detection area boundary of the radar coordinate system, and divide the radar and video lanes into equal modules.

[0010] Step 5: Select point pairs in the radar-view coordinate system based on the lane number and longitudinal queuing position, and end the point pair selection according to the coverage of the lane module area. Then, calculate the final transformation matrix based on the point pair group.

[0011] Furthermore, the radar target radar detection point status information in step 1 is structured data obtained after point cloud clustering analysis.

[0012] Furthermore, in step 2, the target detection framework is Darknet, and the target detection model is the YOLOv3-tiny model trained using the training set. The training dataset comes from different traffic data scenarios and is calibrated according to the type of target to be detected. In the inference process, i.e. model validation, the TensorRT inference acceleration framework is used to accelerate inference.

[0013] Further, step 3 specifically involves: selecting an irregular detection area with the lane line as the front reference, calculating the first transformation matrix using the corner points of this area, and using this matrix to realize the mapping transformation of the detection area;

[0014] The formula for solving the first transformation matrix is:

[0015]

[0016] In the formula, x img1 and y img1 The corrected coordinates are given by inputting the coordinates of the four vertices of the image frame, x. img0 and y img0 The coordinates of the selected detection area are denoted by M1, and the input data are the coordinates of the four vertices of the detection area. M1 is the first transformation matrix.

[0017] Furthermore, step 4, analyzing the trajectory map to perform lane deduction, specifically includes:

[0018] Step 5-1: Binarize the radar and video target trajectory maps;

[0019] Step 5-2: Perform further image processing (close) on the binarized image from Step 5-1;

[0020] Step 5-3: Use the Canny operator and particle area filtering to divide the lane regions. Number the lanes from left to right and divide each lane into n region modules.

[0021] Further, in step 4, the lane baseline and detection area boundary of the radar coordinate system are calculated: the longitudinal trajectory of the radar target is accumulated to obtain the accumulated curve, and the accumulated curve is smoothed using the cubic B-spline curve equation. The superimposed data is used to take extreme points from both sides towards the center to determine the left and right boundaries and the lower boundary of the radar detection area.

[0022] The formula for vertical accumulation is:

[0023]

[0024] The image size is m×n, and when stacked vertically, it becomes an m×1 one-dimensional data, (x i ,y i (x) represents the superimposed data in the i-th column. i ,y i () represents the pixel value in the i-th row and j-th column of the original image;

[0025] The left and right boundaries and the lower boundary are determined as follows: the x-values ​​of the left and right extreme points are used as the x-values ​​of the left and right boundaries and the lower boundary, respectively, and the smaller of the y-values ​​of the left and right extreme points is used as the y-values ​​of the lower boundary and the left and right boundaries.

[0026] Furthermore, in step 5, point pairs in the radar-view coordinate system are selected and the final point pairs are selected based on the lane number and the longitudinal queuing position. Specifically, the data targets collected in each frame are classified into lanes, and the detected targets in the video lane and radar lane are longitudinally sorted respectively. The detection area module is matched with the detection target area. If and only if the lane number and module number of the detected target in both coordinate systems match, a point pair is selected and inserted into the target pair queue. When each area module has a corresponding target point pair, the target point pair selection ends, and the final perspective transformation matrix is ​​calculated.

[0027] Compared with the prior art, the significant advantages of this invention are:

[0028] (1) The method of trajectory reverse in this invention selects points based on the condition of the relative position of the target, which overcomes the inability to accurately obtain the radar point corresponding to the predetermined video target point when multiple targets appear in manual point selection.

[0029] (2) The present invention selects the lower edge of the center of the video detection frame as the detection target point, which fully considers the problem of target drift and lane change caused by the fact that the selected points of the video detection target are not on the same plane due to the size of the target. Compared with the manual selection of points as video target detection points, the accuracy of perspective transformation is enhanced.

[0030] (3) When the radar coordinate system reverses the lane, the present invention takes the stop line as the front edge of the lane, fully considers the inconsistency of the target caused by the viewing angle of the video and the radar, and ensures the one-to-one correspondence of the target.

[0031] (4) The types of targets detected by video are taken into consideration. When a radar module area corresponds to multiple targets, while the video detection module area has only one target, redundant points can be proposed according to the type.

[0032] Compared to existing technologies, this method selects precise point pairs based on the relative position of the target, making it highly targeted and robust.

[0033] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating an automatic calibration method for radar based on trajectory tracing in one embodiment.

[0035] Figure 2 This is a schematic diagram of the radar target trajectory in one embodiment of the automatic radar-visual calibration method based on trajectory tracing.

[0036] Figure 3 This is a schematic diagram of the longitudinal cumulative smoothing curve of the radar target trajectory in an embodiment of the automatic calibration method for radar vision based on trajectory tracing.

[0037] Figure 4 This is a schematic diagram of the radar lane backpropagation results of the radar-based automatic calibration method based on trajectory tracing in one embodiment.

[0038] Figure 5 This is a schematic diagram of the video detection area selection result of the automatic calibration method for radar vision based on trajectory tracing in one embodiment.

[0039] Figure 6 This is a schematic diagram of the detection area correction result of the automatic calibration method for radar vision based on trajectory tracing in one embodiment.

[0040] Figure 7 This is a schematic diagram of the video detection target recognition result of the automatic calibration method based on trajectory tracing in one embodiment.

[0041] Figure 8 This is a schematic diagram of the radar calibration result of a radar automatic calibration method based on trajectory tracing in one embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0043] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0044] In one embodiment, combined Figure 1 This paper provides an automatic radar calibration method based on trajectory tracing, which includes the following steps:

[0045] Step 1: Collect radar target detection point status information, including position, velocity, and status;

[0046] Step 2: Acquire video images from the radar simultaneously, load the target detection model, and input the video images into the target detection model to perform target detection, and obtain the target status information, including location, category, and size;

[0047] Step 3: Select the video detection area, use perspective transformation to correct the detection area to obtain the first transformation matrix, and use the perspective transformation matrix to transform the video detection target to the first correction coordinate system;

[0048] Step 4: Statistically analyze the video and radar target trajectories within the threshold time t, analyze the radar and video target trajectory maps respectively to perform lane inversion, calculate the lane baseline and detection area boundary of the radar coordinate system, and divide the radar and video lanes into equal modules.

[0049] Step 5: Select point pairs in the radar-view coordinate system based on the lane number and longitudinal queuing position, and end the point pair selection according to the coverage of the lane module area. Then, calculate the final transformation matrix based on the point pair group.

[0050] In embodiments of the present invention, the radar target information acquired is structured information, including speed, position, and state; the convolutional model used for target detection and recognition in video frames is Darknet, combined with the Yolov3-tiny algorithm, and the identified targets include: car, bus, bike, and truck, and TensorRT acceleration model is used to accelerate inference in target detection; the video detection area is manually selected once starting from the stop line and image correction is performed; the target is transformed to the first coordinate system for trajectory statistics to avoid errors in target lane statistical segmentation in the original image; when the radar lane is reverse-engineered, the accumulated trajectory information is analyzed and the stop line position is calculated to unify the baseline of the video and radar lanes; both the radar lane and the video lane are divided into n module regions, and the target selection conditions are based on region, target relative position, and target type, achieving precise target positioning.

[0051] This invention addresses the requirements of a comprehensive, real-time, accurate, and efficient integrated transportation management system in intelligent transportation, aiming to improve transportation safety and alleviate traffic congestion. It proposes an automatic radar-visual calibration method based on trajectory tracing. This method starts from the actual traffic flow trajectory, comprehensively considers the characteristics of radar and video targets, and achieves automatic target selection and calculation between two coordinate systems, as well as accurate association between the two coordinate systems. Specifically, the method first acquires radar target signals; obtains image-level target detection results through video information; performs image correction based on the detection area to obtain a first correction coordinate system, transforming the video-detected target to the first correction coordinate system; reverse-engineers the lane based on the target trajectory in the radar coordinate system and the first correction coordinate system; divides the lane into module regions, and selects points using lane numbers and the relative position information of the module regions and targets; when all module regions have selected points, calculates the perspective transformation matrix, associates the video and radar coordinate systems, and completes the calibration task for both video and radar.

[0052] Radar lane reverse push

[0053] The radar lane reversal processing specifically involves: acquiring structured target data from the radar, including position, speed, and status; filtering out static targets based on status information; and plotting the radar target trajectory over a time period t, such as... Figure 2 As shown, the trajectory map undergoes vertical pixel accumulation processing; the accumulation formula is as follows:

[0054]

[0055] The accumulated data is then smoothed using cubic spline interpolation, as shown in the following formula:

[0056] y = a0 + a1x + a2x 2 +a3x 3

[0057] Smoothing results as follows Figure 3 As shown, a threshold is set for the smoothing result, and the first minimum point is found from both ends of the curve inwards to determine the front edge line of each lane and the left and right boundary lines of the radar detection area. Image processing is performed on the trajectory within the detection area, specifically: the trajectory is binarized, then a close operation is performed to remove small cracks, Canny edge extraction is performed, and particle surface filtering is applied to the closed area to obtain the lane boundary range. The resulting lanes are displayed in the original image as shown below. Figure 4 The resulting lanes are numbered from left to right, and each lane is divided into regional modules to obtain radar-based grid lane information.

[0058] Video lane reverse push

[0059] Mark the detection area on the acquired image, such as Figure 5 As shown, due to imaging, viewing angle, and other factors, lane convergence at a distance will converge to a single point. To change this, image correction is performed using the coordinate information of the detection area. The correction formula is as follows:

[0060]

[0061] The coordinates of the four original detection region points and the four vertices of the image are used as inputs into the above formula to calculate the first perspective transformation matrix. All pixels in the detection region are then transformed to the first correction coordinate system using this matrix, completing the initial transformation. The transformation result is as follows: Figure 6 As shown.

[0062] To perform object detection on the original image, a convolutional detection model is first required. This embodiment selects the Darknet model and trains it using the Yolov3-tiny algorithm. The COCO dataset is used to train the convolutional model's weight parameters. The parameter configuration file's batchsize is set to 64, and the learning rate is set to 0.001. More training samples are generated by setting rotation angles, adjusting saturation, exposure, and hue. The learning rate adjustment strategy is set to steps, and the learning rate decays after a certain number of training iterations. The trained convolutional model is a combination of the network model and weight parameters. The model file is passed to the TensorRT deep learning framework for parsing. The generated ONNX general model is then converted into a TRT model for accelerated inference and deployment. Figure 7 The target identification results provide information on the target's location, size, category, and probability.

[0063] The detected targets are filtered according to their type: car, bus, bike, and truck. The filtered targets are then transformed to the first calibration coordinate system using a first perspective transformation matrix. The target point coordinates are selected from the lower edge of the detection box, as shown in the following formula:

[0064]

[0065] The system statistically analyzes target trajectories over a period of time. Based on these trajectories, image processing is performed, including image binarization, close operation, Canny edge detection, image segmentation, and particle area filtering to extract lane information. The extracted lanes are then sorted from left to right, and further divided into module regions to obtain video-based grid lane information.

[0066] Selecting target points using radar

[0067] At the same time, the lane number and module area number of the radar target and video target are determined. Starting from the radar target, the lanes and module areas containing the target are traversed. When the number of targets in the same module area is the same, point-to-point matching is performed according to the longitudinal relative position. When the number of points does not match, and if the number of video points in the same area is less than the number of radar points, the detection type of the video is considered. If there are only one bus and truck type, the one-to-many phenomenon of large targets is considered. The video module areas are sorted longitudinally, and the smaller targets are selected point-to-point with the radar target in relative order. The remaining radar points are sorted longitudinally from smallest to largest, and the smaller points are selected to correspond to the large targets in the video. In other cases, the corresponding module areas are not associated with targets. When all module areas are covered by targets, the point-to-point selection stops, and the selected point-to-point queue is input into the perspective transformation matrix calculation formula to calculate the final perspective transformation matrix. Figure 8 The target association effect of this method is shown.

[0068] In one embodiment, a radar-based automatic calibration system based on trajectory tracing is provided, the system comprising sequentially executing:

[0069] The first module is used to collect status information of radar target detection points, including position, velocity, and status;

[0070] The second module is used to acquire video images from the radar at the same time, load the target detection model, input the video images into the target detection model for target detection, and obtain the target status information, including position, category and size.

[0071] The third module is used to select the video detection area, and to correct the detection area using perspective transformation to obtain the first transformation matrix;

[0072] The fourth module is used to statistically analyze the video and radar target trajectories within a threshold time t, analyze the trajectory maps of radar and video targets respectively to perform lane inversion, divide the radar and video lanes into equal modules, and calculate the lane baseline and detection area boundary of the radar coordinate system.

[0073] The fifth module is used to select point pairs and end point pairs in the radar-view coordinate system based on lane number and longitudinal queuing position, and then calculate the final transformation matrix based on the point pair group.

[0074] Specific limitations regarding the trajectory-based automatic radar calibration system can be found in the limitations of the trajectory-based automatic radar calibration method described above, and will not be repeated here. Each module in the aforementioned trajectory-based automatic radar calibration system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0075] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A radar automatic calibration method based on trajectory tracing, characterized in that, The method includes the following steps: Step 1: Collect radar target detection point status information, including position, velocity, and status; Step 2: Acquire video images from the radar simultaneously, load the target detection model, and input the video images into the target detection model to perform target detection, and obtain the target status information, including location, category, and size; Step 3: Select the video detection area, use perspective transformation to correct the detection area to obtain the first transformation matrix, and use the perspective transformation matrix to transform the video detection target to the first correction coordinate system; Step 4: Statistically analyze the video and radar target trajectories within the threshold time t, analyze the radar and video target trajectory maps respectively to perform lane inversion, calculate the lane baseline and detection area boundary of the radar coordinate system, and divide the radar and video lanes into equal modules. Step 5: Select point pairs in the radar-view coordinate system based on the lane number and longitudinal queuing position, and end the point pair selection according to the coverage of the lane module area. Then, calculate the final transformation matrix based on the point pair group.

2. The automatic radar calibration method based on trajectory tracing according to claim 1, characterized in that, In step 1, the radar target radar detection point status information is structured data obtained after point cloud clustering analysis.

3. The automatic radar calibration method based on trajectory tracing according to claim 1, characterized in that, In step 2, the target detection framework is Darknet, and the target detection model is the YOLOv3-tiny model trained using the training set. The training dataset comes from different traffic data scenarios and is calibrated according to the type of target to be detected. In the inference process, i.e. model validation, the TensorRT inference acceleration framework is used to accelerate inference.

4. The automatic radar calibration method based on trajectory tracing according to claim 1, characterized in that, Step 3 specifically involves: selecting an irregular detection area based on the lane line as the front reference, calculating the first transformation matrix using the corner points of this area, and using this matrix to achieve the mapping transformation of the detection area; The formula for solving the first transformation matrix is: In the formula, x img1 and y img1 The corrected coordinates are given by inputting the coordinates of the four vertices of the image frame, x. img0 and y img0 The coordinates of the selected detection area are given, and the input data consists of the coordinates of the four vertices of the detection area. This is the first transformation matrix.

5. The automatic radar calibration method based on trajectory tracing according to claim 1, characterized in that, Step 4, analyzing the trajectory map to perform lane deduction, specifically includes: Step 5-1: Binarize the radar and video target trajectory maps; Step 5-2: Perform further image processing (close) on the binarized image from Step 5-1; Step 5-3: Use the Canny operator and particle area filtering to divide the lane regions. Number the lanes from left to right and divide each lane into n region modules.

6. The automatic radar calibration method based on trajectory tracing according to claim 1, characterized in that, In step 4, the lane baseline and detection area boundary of the radar coordinate system are calculated: the longitudinal trajectory of the radar target is accumulated to obtain the accumulated curve. The accumulated curve is smoothed using the cubic B-spline curve equation. The superimposed data is used to take extreme points from both sides towards the center to determine the left and right boundaries and the lower boundary of the radar detection area. The formula for vertical accumulation is: The image size is m×n, and when stacked vertically, it becomes a one-dimensional data of size m×1, (x i ,Y i (x) represents the superimposed data in the i-th column. i ,y j () represents the pixel value in the i-th row and j-th column of the original image; The left and right boundaries and the lower boundary are determined as follows: the x-values ​​of the left and right extreme points are used as the x-values ​​of the left and right boundaries and the lower boundary, respectively, and the smaller of the y-values ​​of the left and right extreme points is used as the y-values ​​of the lower boundary and the left and right boundaries.

7. The automatic radar calibration method based on trajectory tracing according to claim 1, characterized in that, In step 5, point pairs in the radar coordinate system are selected based on lane number and longitudinal queuing position, and the point pair selection ends based on the coverage of the lane module area. Specifically, the data targets collected in each frame are classified into lanes, and the detected targets in the video lane and radar lane are longitudinally sorted respectively. The detection area module is matched with the detection target area. If and only if the lane number and module number of the detected target in both coordinate systems match, a point pair is selected and inserted into the target pair queue. When each area module has a corresponding target point pair, the target point pair selection ends, and the final perspective transformation matrix is ​​calculated.

8. A radar-based automatic calibration system based on trajectory tracing according to any one of claims 1 to 7, characterized in that, The system includes sequential execution of: The first module is used to collect status information of radar target detection points, including position, velocity, and status; The second module is used to acquire video images from the radar at the same time, load the target detection model, input the video images into the target detection model for target detection, and obtain the target status information, including position, category and size. The third module is used to select the video detection area, and to correct the detection area using perspective transformation to obtain the first transformation matrix; The fourth module is used to statistically analyze the video and radar target trajectories within a threshold time t, analyze the trajectory maps of radar and video targets respectively to perform lane inversion, divide the radar and video lanes into equal modules, and calculate the lane baseline and detection area boundary of the radar coordinate system. The fifth module is used to select point pairs and end point pairs in the radar-view coordinate system based on lane number and longitudinal queuing position, and then calculate the final transformation matrix based on the point pair group.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images