A method for radar and visual automation calibration based on hierarchical window mapping

Abstract

This invention discloses an automated radar-visual target calibration method based on hierarchical window mapping, comprising the following steps: acquiring radar data and camera image data, and performing time calibration and alignment on the radar and camera; performing target detection and tracking on the radar data to obtain radar target trajectory data, which includes at least a timestamp, target ID, target position, and target velocity information; performing target detection and tracking on the camera image data to obtain visual target trajectory data, which includes at least a timestamp, target ID, and detection box information; performing trajectory filtering and alignment on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs between radar targets and visual targets; forming a matching point set between radar coordinates and image pixel coordinates, and establishing a mapping relationship from radar coordinates to image pixel coordinates using a hierarchical window mapping strategy, thereby outputting the calibration result.

Description

Technical Field

[0001] This invention is applied to the fields of intelligent sensing and intelligent transportation, specifically a radar-visual automated calibration method based on hierarchical window mapping. Background Technology

[0002] With the rapid development of intelligent transportation systems, multimodal sensor fusion technology has been widely applied in traffic scenarios such as highways, ordinary roads, and tunnels, mainly for key tasks such as traffic flow monitoring, abnormal event detection, and vehicle trajectory reconstruction. Among them, millimeter-wave radar and cameras have become the mainstream sensor combination solution due to their complementary sensing characteristics and cost advantages: millimeter-wave radar has the characteristics of accurate ranging and strong resistance to environmental interference, while cameras can provide rich texture semantic information. However, in complex scenarios such as tunnels, millimeter-wave radar is susceptible to multipath reflection interference, and cameras have difficulty accurately estimating the target's trajectory. Therefore, sensor fusion is needed to improve the system's robustness.

[0003] The core foundation of current radar-camera fusion technology lies in sensor calibration. Traditional calibration methods mainly rely on manually set calibration objects (such as calibration boards, corner reflectors, etc.), and require data acquisition in a controlled environment, which has the following significant drawbacks:

[0004] (1) Cumbersome operation and safety risks: The calibration process requires interrupting traffic or closing roads, resulting in high actual deployment costs and affecting normal traffic operation;

[0005] (2) Insufficient flexibility and adaptability: When the sensor position shifts or the environment changes, it needs to be recalibrated. Traditional methods are difficult to support real-time adjustment in dynamic scenarios.

[0006] Limitations of existing targetless calibration schemes: Although there are some calibration algorithms that do not depend on the calibration object, their operation process is complex, requiring offline manual data alignment, or self-calibration based on a specific model. They have strict requirements on computing resources and scenario constraints, making it difficult to meet the needs of actual engineering applications. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide an automated calibration method for radar vision based on hierarchical window mapping, which addresses the shortcomings of the prior art.

[0008] To solve the above-mentioned technical problems, the present invention provides an automated radar calibration method based on hierarchical window mapping, comprising the following steps:

[0009] Acquire radar data and camera image data, and perform time calibration and time alignment on the radar and camera;

[0010] Target detection and target tracking are performed on the radar data to obtain radar target trajectory data, which includes at least timestamp, target ID, target position and target speed information;

[0011] Target detection and target tracking are performed on the camera image data to obtain visual target trajectory data, which includes at least a timestamp, target ID, and detection box information;

[0012] Trajectory filtering and alignment are performed on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs between the radar target and the visual target;

[0013] Based on the matching trajectory pair, a set of matching points is formed between radar coordinates and image pixel coordinates, and a hierarchical window mapping strategy is used to establish the mapping relationship from radar coordinates to image pixel coordinates, thereby outputting the calibration result.

[0014] As one possible implementation, the steps of acquiring radar data and camera image data, and performing time calibration and time alignment on the radar and camera are as follows:

[0015] The radar and camera are time-calibrated via an NTP time synchronization server. This invention also employs a linear interpolation method to achieve time synchronization between the radar and camera.

[0016]

[0017] and These are the radar coordinates at adjacent time points. For camera timestamps, Implemented by interpolation Radar coordinates at any given time.

[0018] As one possible implementation, further, target detection and target tracking are performed on the radar data to obtain radar target trajectory data. The radar target trajectory data at least includes a timestamp, target ID, target position, and target velocity information. The target detection step specifically involves:

[0019] The radar signal is used to extract the target point cloud through range-Doppler FFT and CFAR detection, and DBSCAN clustering is used to remove noise points.

[0020] As one possible implementation, further, target detection and target tracking are performed on the radar data to obtain radar target trajectory data. The radar target trajectory data at least includes a timestamp, target ID, target position, and target velocity information. The target tracking step specifically involves:

[0021] Based on Kalman filter-based target trajectory tracking, the state vector is defined as follows: The state transition model is as follows:

[0022]

[0023] in The sampling interval is... This is process noise. The output trajectory data includes timestamp, target ID, and location. and speed ( )information.

[0024] As a possible implementation, further, target detection and target tracking are performed on the camera image data to obtain visual target trajectory data. The visual target trajectory data includes at least a timestamp, target ID, and detection box information. The target detection step uses the YOLOX algorithm, and the target tracking step uses the ByteTrack algorithm. The output data includes at least a timestamp, target ID, and the center coordinates of the detection box. ,width ,high and confidence level .

[0025] As a possible implementation, further, trajectory filtering and trajectory alignment are performed on the radar target trajectory data and the visual target trajectory data to obtain a matching trajectory pair between the radar target and the visual target. The trajectory filtering in the step includes at least a vehicle type constraint: only car targets with confidence levels meeting preset conditions are retained, and the total number of car targets in the scene is limited to no more than 10.

[0026] As a possible implementation, further, trajectory filtering and trajectory alignment are performed on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs of radar targets and visual targets. The trajectory filtering in the step includes spatial range constraints: based on the characteristic that radar and camera trajectories must match in a single target scene, the effective area is roughly estimated, and the radar distance range and image pixel coordinate range are limited to narrow the spatial search range.

[0027] As one possible implementation, further, trajectory filtering and trajectory alignment are performed on the radar target trajectory data and the visual target trajectory data to obtain a matching trajectory pair between the radar target and the visual target. The trajectory filtering in the step includes trajectory length constraints: retaining trajectories whose duration exceeds a preset duration threshold.

[0028] As a possible implementation, further, trajectory filtering and trajectory alignment are performed on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs of radar targets and visual targets. The trajectory alignment in the step includes: aligning the radar trajectory and camera trajectory based on timestamps and setting a time deviation threshold of 0.1s; and spatial association based on the spatial distribution order of the radar coordinate system and the image coordinate system to achieve automatic sorting and alignment of trajectories in the case of multiple targets at the same time.

[0029] As a possible implementation, further, based on the matching trajectory pair, a set of matching points is formed between radar coordinates and image pixel coordinates, and a hierarchical window mapping strategy is adopted to establish the mapping relationship from radar coordinates to image pixel coordinates, thereby outputting the calibration result. The hierarchical window mapping strategy in the step includes a precise window matching stage and an extended window interpolation stage, wherein: in the precise window matching stage, a precise window is defined around the mapped radar point, and when there is at least one matching point within the precise window, the average value of the pixel coordinates corresponding to the matching point within the precise window is output as the mapping result.

[0030] The present invention adopts the above technical solution and has the following beneficial effects:

[0031] 1. This invention has been applied in actual tunnel scenarios, with an average calibration error of less than 5 pixels, and can achieve high-precision fusion of tunnel millimeter-wave radar and camera at 200m.

[0032] 2. By using this invention, targetless automated calibration can be achieved, saving labor costs and reducing safety hazards. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below.

[0034] This invention provides an automated radar calibration method based on hierarchical window mapping, comprising the following steps:

[0035] Acquire radar data and camera image data, and perform time calibration and time alignment on the radar and camera;

[0036] Target detection and target tracking are performed on the radar data to obtain radar target trajectory data, which includes at least timestamp, target ID, target position and target speed information;

[0037] Target detection and target tracking are performed on the camera image data to obtain visual target trajectory data, which includes at least a timestamp, target ID, and detection box information;

[0038] Trajectory filtering and alignment are performed on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs between the radar target and the visual target;

[0039] Based on the matching trajectory pair, a set of matching points is formed between radar coordinates and image pixel coordinates, and a hierarchical window mapping strategy is used to establish the mapping relationship from radar coordinates to image pixel coordinates, thereby outputting the calibration result.

[0040] As one possible implementation, the steps of acquiring radar data and camera image data, and performing time calibration and time alignment on the radar and camera are as follows:

[0041] The radar and camera are time-calibrated via an NTP time synchronization server. This invention also employs a linear interpolation method to achieve time synchronization between the radar and camera.

[0042]

[0043] and These are the radar coordinates at adjacent time points. For camera timestamps, Implemented by interpolation The radar coordinates at that moment. Target detection and tracking are performed on the radar data to obtain radar target trajectory data, which includes at least a timestamp, target ID, target position, and target velocity information. The target detection step specifically involves:

[0044] The radar signal is used to extract target point cloud through range-Doppler FFT and CFAR detection, and DBSCAN clustering is used to remove noise points. Target detection and tracking are then performed on the radar data to obtain radar target trajectory data. This radar target trajectory data includes at least a timestamp, target ID, target position, and target velocity information. The target tracking step specifically involves:

[0045] Based on Kalman filter-based target trajectory tracking, the state vector is defined as follows: The state transition model is as follows:

[0046]

[0047] in The sampling interval is... This is process noise. The output trajectory data includes timestamp, target ID, and location. and speed ( Information. The camera image data is subjected to target detection and target tracking to obtain visual target trajectory data. This visual target trajectory data includes at least a timestamp, target ID, and detection box information. The target detection step uses the YOLOX algorithm, and the target tracking step uses the ByteTrack algorithm. The output data includes at least a timestamp, target ID, and the center coordinates of the detection box. ,width ,high and confidence level The process involves performing trajectory filtering and alignment on the radar target trajectory data and the visual target trajectory data to obtain matching trajectories between the radar target and the visual target. The trajectory filtering in this step includes at least a vehicle type constraint: only car targets with confidence levels meeting preset conditions are retained, and the total number of car targets in the scene is limited to no more than 10. The trajectory filtering also includes a spatial range constraint: based on the characteristic that radar and camera trajectories inevitably match in a single-target scene, the effective area is roughly estimated, and the radar distance range and image pixel coordinate range are limited to narrow the spatial search range. Finally, the trajectory filtering also includes a trajectory length constraint: trajectories with a duration exceeding a preset duration threshold are retained. The process of performing trajectory filtering and alignment on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs between radar targets and visual targets includes: aligning radar trajectories and camera trajectories based on timestamps and setting a time deviation threshold of 0.1s; and spatially associating them based on the spatial distribution order of the radar coordinate system and the image coordinate system to achieve automatic trajectory sorting and alignment in the case of multiple targets at the same time. Based on the matching trajectory pairs, a set of matching points for radar coordinates and image pixel coordinates is formed, and a hierarchical window mapping strategy is used to establish the mapping relationship from radar coordinates to image pixel coordinates, thereby outputting the calibration result. The hierarchical window mapping strategy includes a precise window matching stage and an extended window interpolation stage, wherein: in the precise window matching stage, a precise window is defined around the mapped radar point; when there is at least one matching point within the precise window, the average value of the pixel coordinates corresponding to the matching point within the precise window is output as the mapping result.

[0048] Example:

[0049] An automated radar calibration method based on hierarchical window mapping, the method includes:

[0050] 1. Design an automated trajectory acquisition module;

[0051] 2. Intelligent trajectory filtering and alignment strategies are employed to acquire radar and image target pairs;

[0052] 3. Use hierarchical grid window mapping technology to obtain calibration results.

[0053] The specific implementation method is explained in detail below:

[0054] Design an automated trajectory acquisition module

[0055] Radar camera data synchronization

[0056] The radar and camera are time-calibrated via an NTP time synchronization server. This invention also employs a linear interpolation method to achieve time synchronization between the radar and camera.

[0057]

[0058] and These are the radar coordinates at adjacent time points. For camera timestamps, Implemented by interpolation Radar coordinates at any given time.

[0059] Radar target detection and tracking

[0060] Radar signals are used to extract target point clouds through range-Doppler FFT and CFAR detection, and DBSCAN clustering is used to remove noise points. Target trajectory is tracked based on a Kalman filter, and the state vector is defined as follows: The state transition model is as follows:

[0061]

[0062] in The sampling interval is... This is process noise. The output trajectory data includes timestamp, target ID, and location. and speed ( )information.

[0063] Visual target detection and tracking

[0064] The visual target detection module uses the YOLOX algorithm, and the tracking module uses the ByteTrack algorithm. The output data includes timestamps, target IDs, and coordinates of the center of the detection box. ,width ,high and confidence level .

[0065] Intelligent trajectory filtering and alignment strategies are employed to acquire radar and image target pairs.

[0066] Trajectory Filtering Criteria

[0067] Vehicle type constraint: Only retain confidence level The target vehicles are small cars, excluding large vehicles such as trucks, because they are prone to splitting in tunnel scenes, which affects the effectiveness of matching; at the same time, in order to ensure the accuracy of matching, the total number of small car targets in the scene is limited to no more than 10.

[0068] Spatial range constraint: The effective area is roughly estimated based on the occurrence of individual cars in the recorded data. The radar range is... The image pixel coordinate range is , This rough estimate utilizes the characteristic that radar and camera trajectories inevitably match in a single-target scenario to initially limit the spatial search range.

[0069] Trajectory length constraint: Retention duration exceeds The trajectory.

[0070] Spatiotemporal correlation algorithm

[0071] Time correlation: Align radar and camera trajectories based on timestamps, with a time deviation threshold set to... =0.1s;

[0072] Spatial correlation: Matching is performed by the spatial distribution order of trajectory points in the radar coordinate system and the image coordinate system. For multiple targets at the same time, if the radar coordinates satisfy... and Then the image coordinates must satisfy and (Due to the image coordinate system) (Axis downwards). This sorting rule is based on the geometric correspondence between the radar coordinate system and the image coordinate system, enabling automatic trajectory alignment.

[0073] Collect radar trajectory

[0074] The automated trajectory acquisition module and intelligent trajectory filtering and alignment strategies are used to acquire radar-matched trajectories. To ensure calibration accuracy, a sufficient number of matching trajectories must be acquired, at least more than 100 pairs.

[0075] Calibration results are obtained using hierarchical grid window mapping technology.

[0076] Precise window matching stage

[0077] For radar points to be mapped First, define the size around it as The precise window. If one or more pre-defined matching points exist within the window, the mean of the pixel coordinates corresponding to these points is directly calculated as the output:

[0078]

[0079] in, To accurately count the number of matching points within the window, These are the corresponding pixel coordinates. At this stage, high local precision is prioritized.

[0080] Expanded window interpolation stage: If there is no matching point within the exact window, the search range is expanded to a size of [size missing]. Extended window ( Based on the number of matching points within the expanded window. Choose an interpolation strategy:

[0081] Direct Linear Transformation (DLT) Mapping: When When selecting distance The four most recent matching points are used to construct a DLT mapping model based on four point pairs. This model is achieved by solving a system of linear equations. (in It is an 8×9 coefficient matrix. The projection matrix is ​​obtained from the 9-dimensional mapping vector. Then, the pixel coordinates are calculated through projection:

[0082]

[0083] Inverse Distance Weighted (IDW) Interpolation: When IDW interpolation is used at that time. According to... To each matching point Euclidean distance Calculate weights ( To prevent division by zero (smaller values), a weighted sum is used to obtain the pixel coordinates:

[0084]

[0085] The above description is an embodiment of the present invention. For those skilled in the art, any equivalent changes, modifications, substitutions and variations made in accordance with the teachings of the present invention without departing from the principles and spirit of the present invention should be covered by the present invention.

Claims

1. A method for automated radar calibration based on hierarchical window mapping, characterized in that, Includes the following steps: Acquire radar data and camera image data, and perform time calibration and time alignment on the radar and camera; Target detection and target tracking are performed on the radar data to obtain radar target trajectory data, which includes at least timestamp, target ID, target position and target speed information; Target detection and target tracking are performed on the camera image data to obtain visual target trajectory data, which includes at least a timestamp, target ID, and detection box information; Trajectory filtering and alignment are performed on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs between the radar target and the visual target; Based on the matching trajectory pair, a set of matching points is formed between radar coordinates and image pixel coordinates. A hierarchical window mapping strategy is used to establish the mapping relationship between radar coordinates and image pixel coordinates, thereby outputting the calibration result.

2. The automatic calibration method for radar vision based on hierarchical window mapping according to claim 1, characterized in that: The steps of acquiring radar data and camera image data, and performing time calibration and time alignment on the radar and camera are as follows: The radar and camera are time-calibrated via an NTP time synchronization server. This invention also employs a linear interpolation method to achieve time synchronization between the radar and camera.

3. and These are the radar coordinates at adjacent time points. For camera timestamps, Implemented by interpolation Radar coordinates at any given time.

4. The automatic calibration method for radar vision based on hierarchical window mapping according to claim 1, characterized in that: The step of performing target detection and target tracking on the radar data to obtain radar target trajectory data, wherein the radar target trajectory data includes at least timestamp, target ID, target position, and target velocity information, specifically involves the following: The radar signal is used to extract the target point cloud through range-Doppler FFT and CFAR detection, and DBSCAN clustering is used to remove noise points.

5. The automatic calibration method for radar vision based on hierarchical window mapping according to claim 1, characterized in that: The step of performing target detection and target tracking on the radar data to obtain radar target trajectory data, wherein the radar target trajectory data includes at least timestamp, target ID, target position, and target velocity information, specifically involves the following: Based on Kalman filter-based target trajectory tracking, the state vector is defined as follows: The state transition model is as follows:

6. Among them The sampling interval is... This is process noise. The output trajectory data includes timestamp, target ID, and location. and speed ( )information.

7. The automatic calibration method for radar vision based on hierarchical window mapping according to claim 1, characterized in that: The step of performing target detection and target tracking on the camera image data to obtain visual target trajectory data, wherein the visual target trajectory data includes at least a timestamp, target ID, and detection box information, employs the YOLOX algorithm for target detection and the ByteTrack algorithm for target tracking, and the output data includes at least a timestamp, target ID, and the center coordinates of the detection box. ,width ,high and confidence level .

8. The automatic calibration method for radar vision based on hierarchical window mapping according to claim 1, characterized in that: The step of performing trajectory filtering and trajectory alignment on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs between radar targets and visual targets includes at least vehicle type constraints: only retaining car targets with confidence levels meeting preset conditions, and limiting the total number of car targets in the scene to no more than 10.

9. The automatic calibration method for radar vision based on hierarchical window mapping according to claim 1, characterized in that: The step of performing trajectory filtering and trajectory alignment on the radar target trajectory data and the visual target trajectory data to obtain matching trajectory pairs between the radar target and the visual target includes spatial range constraints: based on the characteristic that the radar and camera trajectories must match in a single target scene, the effective area is roughly estimated, and the radar distance range and image pixel coordinate range are limited to narrow the spatial search range.

10. The automatic calibration method for radar vision based on hierarchical window mapping according to claim 1, characterized in that: The step of performing trajectory filtering and trajectory alignment on the radar target trajectory data and the visual target trajectory data to obtain a matching trajectory pair between the radar target and the visual target includes trajectory length constraints: retaining trajectories whose duration exceeds a preset duration threshold.

11. The automatic radar calibration method based on hierarchical window mapping according to claim 1, characterized in that: The step of performing trajectory filtering and trajectory alignment on the radar target trajectory data and the visual target trajectory data to obtain a matching trajectory pair between the radar target and the visual target includes: aligning the radar trajectory and the camera trajectory based on the timestamp and setting a time deviation threshold of 0.1s; and performing spatial association based on the spatial distribution order of the radar coordinate system and the image coordinate system to achieve automatic sorting and alignment of trajectories in the case of multiple targets at the same time.

12. The automatic calibration method for radar vision based on hierarchical window mapping according to claim 1, characterized in that: The step of forming a set of matching points between radar coordinates and image pixel coordinates based on the matching trajectory, and establishing a mapping relationship from radar coordinates to image pixel coordinates using a hierarchical window mapping strategy to output calibration results includes a precise window matching stage and an extended window interpolation stage. In the precise window matching stage, a precise window is defined around the mapped radar point. When there is at least one matching point within the precise window, the average value of the pixel coordinates corresponding to the matching point within the precise window is output as the mapping result.

Images