A radar and visual combined automatic calibration method and system thereof

Abstract

The application discloses a radar and camera combined automatic calibration method and system, which belongs to the technical field of vehicle-road cooperative roadside perception, and specifically relates to a radar and camera combined automatic calibration method and system. First, the original data collected by the radar and the camera are preprocessed and time-synchronized, and a completed data cache pool is constructed, and the synchronized data passing the output verification are output. Then, based on the installation prior parameter initialization, the synchronized data passing the verification are combined, the initial calibration is carried out by using the road static target, the initial external parameter is solved, the solvability is verified, and the initial conversion matrix is output. Based on this, the application has the effects of breaking away from the dependence on the calibration board, completing the initial calibration by using the common static artificial facilities in the road environment, automatically detecting and compensating the long-term installation drift of the sensor by periodic error monitoring, and reducing the frequency of manual road closure recalibration.

Description

Technical Field

[0001] This invention relates to the field of vehicle-road cooperative roadside perception technology, specifically to a radar-visual joint automatic calibration method and system. Background Technology

[0002] Vehicle-road cooperative roadside perception systems are a core component of intelligent transportation systems. The fusion perception of millimeter-wave radar and cameras can simultaneously leverage the advantages of radar's high accuracy in ranging and speed measurement and the rich semantic information of cameras. As a fundamental pre-processing step in fusion perception, the accuracy of radar-camera joint calibration directly determines the reliability of subsequent target detection, tracking, and trajectory prediction.

[0003] However, the existing joint calibration of radar and vision systems has the following problems:

[0004] 1. Static calibration suffers from poor robustness and lacks online correction: Existing static calibration methods rely on manual placement of specific calibration plates, which significantly reduces accuracy under conditions of drastic changes in outdoor lighting and fluctuations in temperature and humidity. Furthermore, roadside sensors are susceptible to installation drift due to wind, sun, and vehicle vibrations; for example, after several months of installation, the pitch angle drift can reach tens of degrees, causing significant deviations in the projection of distant targets in the image. Recalibrating manually requires closing the road, resulting in high costs and low efficiency. Current technologies lack an automatic detection and correction mechanism for long-term sensor installation drift.

[0005] 2. Dynamic calibration requires high target density and does not make sufficient use of static information: Some existing dynamic calibration methods require a large number of densely moving targets as calibration references. In scenarios where targets are sparse, it is difficult to complete effective calibration within a reasonable time. For example, when traffic flow on suburban roads is extremely low in the early morning, some methods require a long time to accumulate data. At the same time, existing methods do not make full use of the geometric constraint information provided by common static man-made facilities in the road environment (such as street lamp poles and traffic signs), resulting in loss of calibration capability or insufficient accuracy during periods of target sparseness.

[0006] 3. Insufficient sensor type adaptability: Existing calibration methods are mostly designed for lidar, relying on dense point clouds and rich geometric features for matching. However, the point clouds of traditional 3D millimeter-wave radars commonly used on roadside radars are sparse and lack height information. Directly applying lidar calibration methods will lead to difficulties in feature matching or insufficient well-posedness in solving extrinsic parameters. Although the point cloud density of 4D imaging radars has improved, existing methods still lack differentiated adaptation strategies for the physical characteristics of different types of millimeter-wave radars. The generalized processing in the fine calibration stage cannot fully leverage the data advantages of various radar models. Summary of the Invention

[0007] To address the aforementioned technical problems of poor robustness and inability to correct online static calibration, high target density requirements and insufficient utilization of static information in dynamic calibration, and insufficient sensor type compatibility, this invention provides the following technical solution:

[0008] A method for automatic calibration of radar and vision systems includes the following steps:

[0009] S1, Multi-source data preprocessing and time synchronization: Preprocess and synchronize the raw data collected by radar and camera, build a complete data cache pool, and output the verified synchronized data;

[0010] S2, Static Target-Assisted Initial Calibration: Based on the initialization of prior extrinsic parameters, combined with the verified synchronous data, the road static target is used for initial calibration, the initial extrinsic parameters are solved, the well-posedness of the solution is verified, and the initial transformation matrix is ​​output.

[0011] S3, Dynamic Target Trajectory Matching and Fine Calibration:

[0012] S31, Dynamic Target Detection: Based on the verified synchronization data, perform dynamic target detection on both the radar side and the image side, and output the radar dynamic target set and the image dynamic target set;

[0013] S32, Radar Trajectory Generation: Based on the radar dynamic target set, the Kalman filter algorithm is used to track each dynamic target, generate a continuous motion trajectory in the radar coordinate system, and output the radar target tracking trajectory set;

[0014] S33, Image Trajectory Generation: Based on the dynamic target set of the image, the Kalman filter algorithm is used to track each dynamic target, generate a continuous motion trajectory in the image coordinate system, and output the image target tracking trajectory set;

[0015] S34, Trajectory Association: Based on the initial transformation matrix, each trajectory in the radar target tracking trajectory set is projected onto the image coordinate system, matched with the image target tracking trajectory set, and output a set of trajectory association pairs;

[0016] S35, Fine calibration optimization: Based on the set of trajectory association pairs, with the objective function of minimizing reprojection error, the initial transformation matrix is ​​iteratively optimized using a multi-initial-value strategy, a robust loss function, and external parameter physical constraints to output a fine calibration transformation matrix;

[0017] S4, Periodic Drift Detection and Extrinsic Parameter Correction: Based on the precise calibration transformation matrix, the calibration error is periodically monitored. When drift is detected, the data validity is checked, the amount of drift is estimated and the extrinsic parameters are corrected. If the correction fails, a degradation strategy is executed.

[0018] As a preferred embodiment of the automatic calibration method for combined radar and vision as described in this invention, the specific steps of step S1 are as follows:

[0019] S11, Raw data acquisition: Simultaneously acquire raw point cloud data from roadside millimeter-wave radar and raw image frame data from cameras, record the hardware timestamp of each data frame, and output the raw radar point cloud set, raw image set, and hardware timestamp sequence.

[0020] S12, Radar point cloud preprocessing: Based on the original radar point cloud set, outlier removal and ground point filtering are performed to output a denoised radar target observation set;

[0021] S13, Image Distortion Correction: Based on the original image set, distortion correction is performed using a pre-calibrated camera intrinsic parameter matrix and distortion coefficients, and a corrected image set is output.

[0022] S14, Timestamp Alignment: Based on the hardware timestamp sequence, the denoised radar target observation set and the corrected image set, the radar data is time-aligned with the camera frame timestamp as the reference, and the time-synchronized radar target observation sequence and image frame sequence are output.

[0023] S15, Synchronization Verification: Based on the time-synchronized radar target observation sequence and image frame sequence, perform time consistency checks on radar target observations corresponding to adjacent image frames. If the proportion of qualified data exceeds the preset ratio, output the verified synchronization data; otherwise, trigger a synchronization anomaly alarm.

[0024] S16, Data Cache Pool Construction: Based on the verified synchronized data, construct a multi-level data cache pool that includes real-time frame cache, short-term trajectory cache, and baseline time period cache, and output the completed data cache pool.

[0025] As a preferred embodiment of the automatic calibration method for combined radar and vision as described in this invention, the specific steps of S2 are as follows:

[0026] S21, Installation of prior extrinsic parameter initialization: Based on the physical installation and configuration information of the millimeter-wave radar and camera, construct a coarse transformation matrix from the radar to the camera coordinate system, and output the denoised coarse transformation matrix;

[0027] S22, Static Target Detection and Semantic Recognition: Based on the image frame sequence in the verified synchronous data, identify and locate static man-made facilities on both sides of the road, and output a list of static target detection results;

[0028] S23, Static Target Radar Detectability Verification: Based on the denoised coarse transformation matrix, the static target detection result list, and the synchronous radar target observation sequence, radar detectability verification is performed on each static target, and the radar detectable static target screening list and the corresponding radar observation point set are output.

[0029] S24, Radar Static Target Observation Extraction: Based on the radar-detectable static target screening list and the corresponding radar observation point set, extract the radar observation information of each detectable static target and output the radar static target observation set;

[0030] S25, Image Static Target Observation Extraction: Based on the list of static target detection results, extract image observation information for each static target and output the image static target observation set;

[0031] S26, Static Target Cross-Modal Association: Based on the radar static target observation set and the image static target observation set, a multi-level joint matching strategy is adopted to establish the correspondence between static targets in the radar coordinate system and the image coordinate system, and output the set of static target cross-modal association pairs;

[0032] S27, Initial extrinsic parameter constraint solution: Based on the set of static target cross-modal correlation pairs, distinguish between 4D imaging radar and traditional 3D millimeter-wave radar types, adopt differentiated solution strategies to solve the initial extrinsic parameter matrix of millimeter-wave radar to camera coordinate system, and output the initial transformation matrix;

[0033] S28, Solve the well-posedness verification: Based on the initial transformation matrix, calculate the reprojection error of the associated pairs. If the error meets the threshold requirement, output the initial transformation matrix; otherwise, return to re-association or trigger a failure alarm.

[0034] As a preferred embodiment of the automatic calibration method for combined radar and vision as described in this invention, the specific steps of S4 are as follows:

[0035] S41, Periodic monitoring of reprojection error: Based on the precise calibration transformation matrix and real-time acquired dynamic target data, the reprojection error is calculated at a fixed period to form a periodic reprojection error statistical sequence.

[0036] S42, Drift Detection: Based on the periodic reprojection error statistical sequence, a dual threshold strategy is used to determine whether the sensor has experienced installation drift, and a drift trigger signal is output.

[0037] S43, Data Validity Verification: Based on the drift trigger signal, extract dynamic target trajectory data and static target baseline data from the constructed data cache pool, perform validity verification, and output the verified dataset;

[0038] S44, Drift estimation: Based on the validated dataset, the sensor installation drift is estimated using an optimization method based on an incremental extrinsic model, and the sensor installation drift estimation result is output.

[0039] S45, External parameter incremental correction: Based on the sensor installation drift estimation result, the currently used external parameter matrix is ​​dynamically updated, and the amplitude of the rotation increment is limited, and the updated transformation matrix or manual verification alarm signal is output.

[0040] S46, Result Verification and Degradation Strategy: Based on the updated transformation matrix or manual review of alarm signals, the reprojection error is recalculated using dynamic target data from subsequent consecutive frames. If the error is lower than the verification threshold, the final transformation matrix that has passed verification is output and stored in the baseline buffer; otherwise, the drift amount is re-estimated or a degradation strategy is triggered.

[0041] A radar-visual joint automatic calibration system, comprising:

[0042] The multi-source data preprocessing and time synchronization module preprocesses and synchronizes the raw data collected by radar and cameras, builds a complete data cache pool, and outputs verified synchronized data.

[0043] The static target-assisted initial calibration module initializes based on prior installation extrinsic parameters, combines verified synchronous data, uses road static targets for initial calibration, solves for initial extrinsic parameters, verifies the well-posedness of the solution, and outputs the initial transformation matrix.

[0044] The dynamic target trajectory matching and fine calibration module includes a dynamic target detection unit, a radar trajectory generation unit, an image trajectory generation unit, a trajectory association unit, a fine calibration optimization unit, and a periodic drift detection and extrinsic parameter correction module;

[0045] The dynamic target detection unit performs dynamic target detection on both the radar side and the image side based on the verified synchronization data, and outputs radar dynamic target sets and image dynamic target sets.

[0046] The radar trajectory generation unit, based on the radar dynamic target set, uses the Kalman filter algorithm to track each dynamic target, generates a continuous motion trajectory in the radar coordinate system, and outputs a radar target tracking trajectory set;

[0047] The image trajectory generation unit, based on the image dynamic target set, uses the Kalman filter algorithm to track each dynamic target, generates a continuous motion trajectory in the image coordinate system, and outputs an image target tracking trajectory set;

[0048] The trajectory association unit, based on the initial transformation matrix, projects each trajectory in the radar target tracking trajectory set onto the image coordinate system, matches it with the image target tracking trajectory set, and outputs a set of trajectory association pairs.

[0049] The fine calibration optimization unit, based on the set of trajectory association pairs, takes minimizing the reprojection error as the objective function, and employs a multi-initial-value strategy, a robust loss function, and external parameter physical constraints to iteratively optimize the initial transformation matrix and output the fine calibration transformation matrix.

[0050] The periodic drift detection and extrinsic parameter correction module periodically monitors calibration errors based on the fine calibration transformation matrix. When drift is detected, it performs data validity verification, estimates the amount of drift, and corrects the extrinsic parameters. If the correction fails, it executes a degradation strategy.

[0051] As a preferred embodiment of the radar-visual joint automatic calibration system described in this invention, the multi-source data preprocessing and time synchronization module includes:

[0052] The raw data acquisition unit synchronously acquires raw point cloud data from the roadside millimeter-wave radar and raw image frame data from the camera, records the hardware timestamp of each data frame, and outputs the raw radar point cloud set, the raw image set, and the hardware timestamp sequence.

[0053] The radar point cloud preprocessing unit performs outlier removal and ground point filtering based on the original radar point cloud set, and outputs a denoised radar target observation set.

[0054] The image distortion correction unit performs distortion correction based on the original image set using a pre-calibrated camera intrinsic parameter matrix and distortion coefficients, and outputs a corrected image set.

[0055] The timestamp alignment unit, based on the hardware timestamp sequence, the denoised radar target observation set and the corrected image set, uses the camera frame timestamp as a reference to perform time alignment on the radar data and outputs the time-synchronized radar target observation sequence and image frame sequence.

[0056] The synchronization verification unit performs a time consistency check on the radar target observations corresponding to adjacent image frames based on the time-synchronized radar target observation sequence and image frame sequence. If the proportion of qualified data exceeds the preset proportion, the unit outputs the synchronized data that has passed the verification; otherwise, it triggers a synchronization anomaly alarm.

[0057] The data cache pool construction unit, based on verified synchronized data, constructs a multi-level data cache pool including real-time frame cache, short-term trajectory cache, and baseline time period cache, and outputs the completed data cache pool.

[0058] As a preferred embodiment of the radar-visual joint automatic calibration system described in this invention, the static target-assisted initial calibration module includes:

[0059] Install the prior extrinsic parameter initialization unit, construct a coarse transformation matrix from the radar to the camera coordinate system based on the physical installation and configuration information of the millimeter-wave radar and camera, and output the denoised coarse transformation matrix;

[0060] The static target detection and semantic recognition unit identifies and locates static man-made facilities on both sides of the road based on the image frame sequence in the verified synchronous data, and outputs a list of static target detection results.

[0061] The static target radar detectability verification unit verifies the radar detectability of each static target based on the denoised coarse transformation matrix, the static target detection result list, and the synchronous radar target observation sequence, and outputs a radar-detectable static target screening list and the corresponding radar observation point set.

[0062] The radar static target observation extraction unit extracts radar observation information for each detectable static target based on the radar-detectable static target screening list and the corresponding radar observation point set, and outputs the radar static target observation set.

[0063] The image static target observation and extraction unit extracts image observation information for each static target based on the static target detection result list and outputs an image static target observation set;

[0064] The static target cross-modal association unit, based on the radar static target observation set and the image static target observation set, adopts a multi-level joint matching strategy to establish the correspondence between static targets in the radar coordinate system and the image coordinate system, and outputs a set of static target cross-modal association pairs;

[0065] The initial extrinsic parameter constraint solving unit, based on the static target cross-modal correlation pair set, distinguishes between 4D imaging radar and traditional 3D millimeter-wave radar types, and uses a differentiated solving strategy to solve the initial extrinsic parameter matrix of the millimeter-wave radar to the camera coordinate system, and outputs the initial transformation matrix;

[0066] Solve the well-posed verification unit. Based on the initial transformation matrix, calculate the reprojection error of the associated pairs. If the error meets the threshold requirement, output the initial transformation matrix; otherwise, return to re-association or trigger a failure alarm.

[0067] As a preferred embodiment of the radar-visual joint automatic calibration system described in this invention, the periodic drift detection and extrinsic parameter correction module includes:

[0068] The reprojection error periodic monitoring unit calculates the reprojection error at fixed intervals based on the precise calibration transformation matrix and real-time acquired dynamic target data, forming a periodic reprojection error statistical sequence.

[0069] The drift determination unit, based on the periodic reprojection error statistical sequence, uses a dual threshold strategy to determine whether the sensor has experienced installation drift and outputs a drift trigger signal.

[0070] The data validity verification unit, based on the drift trigger signal, extracts dynamic target trajectory data and static target baseline data from the constructed data cache pool, performs validity verification, and outputs the verified dataset.

[0071] The drift estimation unit estimates the sensor installation drift based on a validated dataset using an optimization method based on an incremental extrinsic model, and outputs the sensor installation drift estimation result.

[0072] The extrinsic parameter incremental correction unit dynamically updates the currently used extrinsic parameter matrix based on the sensor installation drift estimation results, limits the amplitude of the rotation increment, and outputs an updated transformation matrix or a manual verification alarm signal.

[0073] The result verification and degradation strategy unit, based on the updated transformation matrix or the manual review alarm signal, recalculates the reprojection error using dynamic target data from subsequent consecutive frames. If the error is lower than the verification threshold, it outputs the final transformation matrix that has passed verification and stores it in the baseline buffer; otherwise, it returns to re-estimate the drift amount or triggers the degradation strategy.

[0074] Compared with existing technologies:

[0075] 1. By constructing a hierarchical calibration architecture that includes initialization of prior external parameters, initial calibration assisted by static targets, fine calibration of dynamic target trajectories, and incremental correction of external parameters linked by periodic error monitoring, it can achieve the effect of eliminating the dependence on calibration boards, completing initial calibration using common static artificial facilities in the road environment, and simultaneously automatically detecting and compensating for long-term sensor installation drift through periodic error monitoring, thereby reducing the frequency of manual road closures for recalibration.

[0076] 2. A two-stage solution model, which uses static target observation to provide geometric constraints for initial calibration and combines dynamic target trajectory for fine-tuning, enables the maintenance of basic calibration capabilities using static facilities in low-target-density scenarios, while gradually accumulating effective matching data and improving calibration accuracy when dynamic targets appear. Simultaneously, a static target radar detectability verification mechanism ensures that only static targets stably observable by the radar are used for calibration, avoiding interference from invalid targets.

[0077] 3. By distinguishing the data characteristics of 4D imaging radar and traditional 3D millimeter-wave radar, a differentiated static target observation extraction strategy and extrinsic parameter solution strategy (including degree-of-freedom constraints, minimum correlation pair requirements, and slope adaptive processing) are designed in the initial calibration stage. In the fine calibration stage, a differentiated error weighting strategy is adopted. This approach can adapt to the physical characteristics of different millimeter-wave radar models and improve the problem of insufficient well-posedness of extrinsic parameter solution caused by directly applying lidar calibration methods.

[0078] 4. By introducing convergence guarantee mechanisms such as multiple initial value strategies, robust loss functions, and physical constraints on extrinsic parameters, combined with data validity verification and failure correction degradation strategies, the robustness of the calibration optimization process is improved, and reliable calibration results can still be maintained even when there are erroneous correlations, data pollution, or changes in the scenario. Attached Figure Description

[0079] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0080] Figure 2 This is a schematic diagram of the multi-source data preprocessing and time synchronization module framework of the present invention;

[0081] Figure 3 This is a schematic diagram of the static target-assisted initial calibration module framework of the present invention;

[0082] Figure 4 This is a schematic diagram of the dynamic target trajectory matching and fine calibration module framework of the present invention;

[0083] Figure 5 This is a schematic diagram of the periodic drift detection and extrinsic parameter correction module framework of the present invention. Detailed Implementation

[0084] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0085] This invention provides a method for automatic calibration of radar and vision systems. Please refer to [link / reference]. Figures 1-5 This includes the following steps:

[0086] S1, Multi-source data preprocessing and time synchronization: Preprocess and synchronize the raw data collected by radar and camera, build a complete data cache pool, and output the verified synchronized data;

[0087] S11, Raw Data Acquisition: Synchronously acquire raw point cloud data from the roadside millimeter-wave radar and raw image frame data from the camera, recording the hardware timestamp for each data frame. For 4D imaging radar, acquire its output point cloud data, with each point containing range, azimuth, elevation, radial velocity, and reflection intensity information; for traditional 3D millimeter-wave radar, acquire its periodically output target list or sparse point cloud data, with each point containing range, azimuth, and radial velocity information. Output the raw radar point cloud set, raw image set, and hardware timestamp sequence.

[0088] S12, Radar Point Cloud Preprocessing: Based on the original radar point cloud set, outlier removal is first performed using a joint thresholding method based on distance and reflection intensity to remove outliers that significantly exceed the road range or have abnormal reflection intensity. Then, ground point filtering is performed, utilizing prior information about ground height in the road scene to filter out clutter points caused by ground reflection. For 4D imaging radar point clouds, valid target point clouds and their height information are retained. For traditional 3D millimeter-wave radar target lists, valid targets and their range-azimuth observations are retained. The denoised radar target observation set is output.

[0089] S13, Image Distortion Correction: Based on the original image set, radial and tangential distortion corrections are performed on each frame using a pre-calibrated camera intrinsic parameter matrix and distortion coefficients to obtain a distortion-free image. Intrinsic parameter calibration is completed at the factory using a checkerboard calibration method, and the calibration results include focal length, principal point coordinates, and distortion coefficients. A corrected image set is output.

[0090] S14, Timestamp Alignment: Based on the hardware timestamp sequence, the denoised radar target observation set, and the corrected image set, the radar data is time-aligned using the camera frame timestamp as a reference. Since the output method of millimeter-wave radar data varies depending on the model, the alignment strategy is differentiated: For radars that output point clouds in frame format (such as 4D imaging radar), the nearest neighbor matching method is used to select the radar frame with the smallest time difference; for radars that periodically output target lists (such as traditional 3D millimeter-wave radar, with a typical output period of 50ms), a sliding time window method is used to accumulate all radar target observations within the window period (e.g., 100ms), using the midpoint of the window as the timestamp of that group of observations. Multiple observations of the same target within the window period are deduplicated using ID or position continuity. The time-synchronized radar target observation sequence and image frame sequence are output.

[0091] S15, Synchronization Verification: Based on the time-synchronized radar target observation sequence and image frame sequence, perform a time consistency check on the radar target observations corresponding to adjacent image frames. Calculate the time difference between adjacent frames for the same physical target. If the time difference is less than the set synchronization tolerance threshold (e.g., 80ms), the data set is considered synchronized successfully. Statistically calculate the percentage of successful data. If the percentage exceeds a preset ratio (e.g., 90%), output the verified synchronized data; otherwise, trigger a synchronization anomaly alarm, prompting a check of the hardware synchronization mechanism.

[0092] S16, Data Cache Pool Construction: Based on validated synchronization data, a multi-level data cache pool is constructed. The cache pool consists of three levels: the first level is a real-time frame cache, storing all synchronization data from the most recent few seconds (e.g., 10 seconds) for fine calibration and error monitoring; the second level is a short-term trajectory cache, storing dynamic target tracking trajectories and static target observation records from the most recent few minutes (e.g., 5 minutes) for drift estimation; the third level is a baseline time period cache, storing baseline data (including static target association pairs, dynamic trajectory association pairs, etc.) from the calibration initialization phase or the most recent successful correction, serving as a clean reference dataset for drift estimation. The cache pool employs a cyclic overwrite mechanism, automatically overwriting the oldest data when the capacity reaches its limit. Output the completed data cache pool.

[0093] S2, Static Target-Assisted Initial Calibration: Based on the initialization of prior extrinsic parameters, combined with the verified synchronous data, the road static target is used for initial calibration, the initial extrinsic parameters are solved, the well-posedness of the solution is verified, and the initial transformation matrix is ​​output.

[0094] S21, Initialization of Prior Extrinsic Parameters: Based on the physical installation configuration information of the millimeter-wave radar and camera, a coarse transformation matrix is ​​constructed from the radar to the camera coordinate system. The physical installation configuration information includes: equipment installation height difference (obtained through measurement or design drawings), horizontal installation offset (obtained through measurement), and nominal values ​​of installation pitch and yaw angles (obtained through installation inclinometers or design drawings). These physical parameters are converted into initial estimates of the rotation matrix and translation vector, forming the coarse transformation matrix. The accuracy requirement for this coarse extrinsic parameter is not high (pitch angle error within ±5°, translation error within ±0.5m is sufficient), and it is only used for coarse spatial correlation of cross-modal targets in subsequent steps. The denoised coarse transformation matrix is ​​output.

[0095] S22, Static Object Detection and Semantic Recognition: Based on the image frame sequence in the validated synchronous data, a deep learning-based semantic segmentation network or object detection network is used to identify and locate static man-made facilities on both sides of the road, including vertical columnar structures such as streetlights, traffic signs, traffic signal poles, and utility poles. For each detected static object, its two-dimensional bounding box coordinates in the image, semantic category label, and confidence score are output to form a static object detection result list. Only static objects with a confidence score higher than a set threshold (e.g., 0.7) and located within the effective perception area of ​​the road are retained. The static object detection result list is output.

[0096] S23, Static Target Radar Detectability Verification: Based on the denoised coarse transformation matrix, the static target detection result list, and the synchronous radar target observation sequence, radar detectability verification is performed for each static target. The verification method is as follows: the center point of the bottom edge of the two-dimensional bounding box of the static target in the image is projected backward onto the radar coordinate system using the coarse transformation matrix to obtain the theoretical azimuth and range range of the target in the radar coordinate system;

[0097] Let the normalized image coordinates of the center point of the bottom edge of the bounding box of the static target in the image be... The camera intrinsic parameter matrix is The rough rotation matrix is The rough translation vector is The direction of the ray projected back onto the radar coordinate system is:

[0098]

[0099] Let the equation of the road plane in the radar coordinate system be: ( (Given the radar installation height), the intersection point of the ray and the road plane is:

[0100]

[0101] in Direction vector of Quantity.

[0102] From this, the theoretical distance to the target can be obtained. and theoretical azimuth .

[0103] The system searches for radar points within the specified azimuth and range range in the radar target observation sequence. If at least one radar point falls within this range, the static target is determined to be detectable by the radar and retained for subsequent calibration. If no corresponding radar point is detected for several consecutive frames (e.g., 5 frames), the static target is determined to be currently undetectable (possibly due to low material RCS, obstruction, or radar viewing angle issues). It is marked as temporarily unavailable and not included in the current calibration round, but its image detection results are retained for re-verification in subsequent rounds. The system outputs a list of detectable static targets and the corresponding set of radar observation points.

[0104] S24, Radar Static Target Observation Extraction: Based on the radar-detectable static target screening list and the corresponding radar observation point set, for each detectable static target, the following radar observation information is extracted: range, azimuth, elevation (4D radar), and reflection intensity of the radar point corresponding to the static target. If the same static target corresponds to multiple radar points, the average range, average azimuth, and average reflection intensity of these points are calculated as the radar observation characterization of the target. The output is a radar static target observation set, where each observation includes the target ID, radar range-azimuth observation, elevation observation (4D radar), and reflection intensity statistics.

[0105] S25, Image Static Target Observation Extraction: Based on the static target detection result list, for each static target's 2D bounding box region in the image, extract the following observation information: image coordinates of the center point of the bottom edge of the bounding box (as an estimate of the target's projected position in the image), vertical pixel coordinates of the center point of the bottom edge of the bounding box (used to assist in distance estimation), and semantic category. Output image static target observation set, where each observation contains the target ID, image coordinates, and semantic category.

[0106] S26, Static Target Cross-Modal Association: Based on radar static target observation sets and image static target observation sets, establish the correspondence between static targets in the radar coordinate system and the image coordinate system. The association strategy adopts multi-level joint matching:

[0107] Let the radar static target observation set be The image static target observation set is The rough transformation matrix is .

[0108] First layer (distance coarse screening): Converts radar observations into three-dimensional points in the radar coordinate system. Projected onto the image coordinate system:

[0109]

[0110] Calculate the image coordinate distance between the projection point and each static target in the image, and only retain those distances less than a set threshold. Candidate matching pairs (e.g., 20% of the image width):

[0111]

[0112] Second layer (angle fine screening): For candidate pairs that have passed the first layer of screening Using camera intrinsic parameters and coarse extrinsic parameters, the vertical pixel coordinates of static targets in the image are determined. The theoretical distance is calculated by reverse calculation. Let the ordinate of the camera's principal point be... focal length is The theoretical pitch angle is:

[0113]

[0114] Theoretical distance is ,in This is a rough pitch angle. Candidate pairs with distance deviations within a reasonable range (e.g., ±30%) are retained.

[0115]

[0116] The third layer (reflection intensity-assisted differentiation): For candidate pairs that pass the first two layers, reflection intensity information is used as an auxiliary differentiation feature. When multiple candidate pairs are spatially close, candidate pairs whose reflection intensity matches the expected target material are prioritized for matching.

[0117] A set of cross-modal correlation pairs for static targets is formed. Each correlation pair contains a correspondence between a radar static target observation and an image static target observation. If the number of correlation pairs is less than the minimum required for subsequent solutions, a static target insufficiency alarm is triggered, prompting the system to wait for more targets to enter the sensing area or to check the sensor status.

[0118] S27, Initial Extrinsic Parameter Constraint Solution: Based on the set of static target cross-modal correlation pairs, solve for the initial extrinsic parameter matrix of the millimeter-wave radar reaching the camera coordinate system. The solution process distinguishes between radar types and considers scene geometric characteristics.

[0119] Let the set of associative pairs be ,in A three-dimensional point in the radar coordinate system. These are the corresponding image coordinates. The extrinsic parameter to be determined is the rotation matrix. Translation vector .

[0120] For 4D imaging radar:

[0121] The objective function to be optimized is:

[0122]

[0123] in For robust loss functions (such as Huber loss). Let be the intrinsic parameter matrix of the camera. Each association pair provides 3 spatial constraints (X,Y,Z) and 2 image observation constraints (u,v), with 6 degrees of freedom for the extrinsic parameters. Theoretically, a minimum of 2 association pairs are required to solve the problem, but to ensure robustness, at least 3 association pairs are required and they must be distributed in different distance intervals.

[0124] For traditional 3D millimeter-wave radar:

[0125] Let the known pitch angle be... Then the rotation matrix It can be expressed as containing only the yaw angle. and roll angle The parameterized form of radar observation. (High-level information missing). The reprojection equation is:

[0126]

[0127] An optimization method based on distance-azimuth consistency is used to solve for the initial extrinsic parameters, with the optimization objective being to minimize the reprojection error.

[0128]

[0129] For non-ideal planar road scenarios with slopes or curves, this step introduces adaptive processing: by analyzing the three-dimensional distribution (4D radar) or range-azimuth distribution (3D radar) of multiple static targets, the local slope angle of the road is estimated; if the slope angle exceeds a set threshold (e.g., 3°), a slope compensation term is added to the optimization objective, or a piecewise planar assumption is used to replace the global planar assumption in order to reduce the impact of planar assumption deviation on the solution of external parameters.

[0130] Output the initial transformation matrix.

[0131] S28, Solving for well-posedness verification: Based on the initial transformation matrix, calculate the reprojection error of all associated pairs, and statistically analyze the average reprojection error and the maximum reprojection error. If the average reprojection error is lower than a set threshold (e.g., 20 pixels) and the maximum reprojection error is lower than a set upper limit (e.g., 50 pixels), the initial extrinsic parameter solution is deemed well-posed, and the initial transformation matrix is ​​output to the dynamic target trajectory matching and fine calibration module; otherwise, the solution is deemed to have failed. Possible reasons include insufficient number of associated pairs, poor distribution, or erroneous associations. In this case, return to the static target cross-modal association and re-associate (using stricter screening conditions). If multiple retries still fail, an initial calibration failure alarm is triggered, prompting a check of sensor installation or environmental conditions.

[0132] S3, Dynamic Target Trajectory Matching and Fine Calibration: Based on the initial transformation matrix, fine calibration is performed using the motion trajectory of the dynamic target, external parameters are optimized, and a fine calibration transformation matrix is ​​output.

[0133] S31, Dynamic Target Detection: Based on validated synchronous data, dynamic target detection is performed on both the radar and image sides. On the radar side, a radar target detection algorithm is used to extract dynamic targets such as vehicles and pedestrians from the denoised radar target observation set, outputting the radar observation (range, azimuth, velocity) for each dynamic target. On the image side, an image target detection algorithm is used to detect dynamic targets such as vehicles and pedestrians from the corrected image, outputting the image bounding box coordinates and semantic category for each dynamic target. Both the radar dynamic target set and the image dynamic target set are output.

[0134] S32, Radar Trajectory Generation: Based on a dynamic radar target set, a Kalman filter algorithm is used to track each dynamic target. The state vector contains the target's position (range, azimuth) and velocity (radial velocity, azimuth rate of change) in the radar coordinate system, while the observation vector represents the target position detected by the radar in each frame. Kalman filtering is used for state prediction and observation updates to generate a continuous motion trajectory for each target in the radar coordinate system, outputting a set of radar target tracking trajectories. Each trajectory contains the target ID, a temporal position sequence, and a velocity sequence.

[0135] S33, Image Trajectory Generation: Based on a dynamic target set in the image, a Kalman filter algorithm with the same structure is used to track each dynamic target. The state vector contains the target's position (center coordinates of the bottom edge of the bounding box) and velocity (pixel displacement velocity) in the image coordinate system, and the observation vector is the target position detected in each frame of the image. A continuous motion trajectory of each target in the image coordinate system is generated, and a set of image target tracking trajectories is output. Each trajectory contains the target ID, a temporal position sequence, and a velocity sequence.

[0136] S34, Trajectory Association: Based on the initial transformation matrix, each trajectory in the radar target tracking trajectory set is projected onto the image coordinate system to obtain the projected trajectory. During projection, for 4D imaging radar trajectories, the complete three-dimensional coordinates are used for projection; for traditional 3D millimeter-wave radar trajectories, height information is supplemented using the road plane assumption before projection. The projected trajectory is matched with the image target tracking trajectory set, and a joint cost function based on motion similarity and spatial proximity is used to achieve a one-to-one association between the radar trajectory and the image trajectory using the Hungarian algorithm.

[0137] Set radar trajectory Image trajectory ,in A three-dimensional point in the radar coordinate system. Image coordinates. Project the radar trajectory onto the image coordinate system:

[0138]

[0139] The joint cost function comprehensively considers the projection position deviation, velocity direction consistency, and trajectory length difference between trajectories:

[0140]

[0141] in For time overlap intervals, , The angle between the direction of the trajectory velocity, These are the weighting coefficients.

[0142] The optimal association is solved using the Hungarian algorithm, i.e., the solution is:

[0143]

[0144] in This is a matching mapping from radar trajectory to image trajectory.

[0145] Output the set of trajectory association pairs.

[0146] S35, Fine Calibration Optimization: Based on the set of trajectory association pairs, the initial transformation matrix is ​​iteratively optimized with the objective function of minimizing the reprojection error. The optimization process employs the following strategies to ensure convergence and robustness:

[0147] (1) Multiple initial value strategy: The initial transformation matrix is ​​used as the main initial value, and multiple perturbation initial values ​​are generated at the same time (a small range of random perturbation is added to the rotation and translation components of the initial extrinsic parameters), and optimization is performed separately. The result with the smallest final reprojection error is selected as the optimal solution to avoid getting trapped in local minima.

[0148] (2) Robust loss function: The Huber loss function is used instead of the traditional L2 loss function to reduce the negative impact of abnormal correlation points on the optimization results and enhance the tolerance to erroneous trajectory correlation;

[0149] (3) External parameter physical constraints: The optimization variables are represented by the axis angle of the rotation matrix (to avoid Euler angle gimbal lock problem), and physical constraint regularization terms are introduced, including the orthogonality constraint of the rotation matrix, the reasonable range constraint of the translation vector (such as the difference in installation height between the camera and the radar, and the horizontal offset should be within the physical installation range), and the amplitude limit of the rotation increment (the single optimization rotation change does not exceed 5°) to prevent the optimization results from deviating from the actual installation configuration;

[0150] (4) Differentiated processing for different radar types: For 4D imaging radar, the error term is constructed using the complete three-dimensional coordinates of all corresponding points on the trajectory; for traditional 3D millimeter-wave radar, the height uncertainty weight is added to the error term to reduce the excessive influence of the height dimension on the optimization result, while more trajectory points are accumulated to compensate for the lack of single-point height information.

[0151] Output the fine calibration transformation matrix.

[0152] S4, Periodic Drift Detection and Extrinsic Parameter Correction: Based on the fine calibration transformation matrix, the calibration error is periodically monitored. When drift is detected, the data validity is checked, the amount of drift is estimated and the extrinsic parameters are corrected. If the correction fails, a degradation strategy is executed.

[0153] S41, Periodic Monitoring of Reprojection Error: Based on the precise calibration transformation matrix and real-time acquired dynamic target data, for each newly arrived frame of data, the position of the dynamic target detected by the radar is projected onto the image coordinate system, and the reprojection error between the projected point and the corresponding target position detected in the image is calculated. The average and maximum reprojection errors of all dynamic targets within multiple consecutive frames (e.g., 30 frames, corresponding to approximately 1 second) are statistically analyzed to form a periodic reprojection error statistical sequence. The monitoring process performs a complete statistical analysis at a fixed period (e.g., every 30 seconds) rather than processing each frame in real time to reduce the computational load.

[0154] S42, Drift Detection: Sensor installation drift is determined based on a periodic reprojection error statistical sequence. A dual-threshold strategy is employed: when the average reprojection error over multiple consecutive monitoring periods (e.g., 3 periods, corresponding to approximately 90 seconds) exceeds the first threshold (e.g., 10 pixels), an early warning state is entered; when the early warning state persists and the average reprojection error exceeds the second threshold (e.g., 15 pixels), sensor installation drift is formally determined, and a drift trigger signal is output. This dual-threshold strategy effectively avoids false triggers caused by instantaneous noise or sporadic detection errors. Furthermore, if the number of valid dynamic targets in a monitoring period is insufficient (e.g., less than 3), the drift determination for that period is skipped, preventing misjudgments during periods of target sparseness.

[0155] S43, Data validity check: Based on the drift trigger signal, data is extracted from the completed data cache pool for drift estimation. Data extraction follows the following principles: (1) Dynamic target trajectory data is taken from the most recent time period (e.g., 5 minutes) before drift judgment in the second-level short-term trajectory cache, but validity check is required - remove trajectory points with significantly abnormal reprojection errors (more than 3 times the standard deviation of historical benchmark errors) in this time period. These abnormal points may have been severely affected by drift; (2) Static target observation data is taken from the benchmark data after the most recent successful calibration or correction in the third-level benchmark time period cache. This benchmark data is regarded as a clean reference that is not contaminated by the current drift; (3) If the data in the benchmark time period cache becomes invalid due to the long time (e.g., more than 30 days) or scene changes (e.g., road construction causes the static target position to change), the benchmark data update process is triggered, and the benchmark association pair is re-established using the current extrinsic parameters.

[0156] Output the dynamic trajectory dataset and static target benchmark dataset after validity verification.

[0157] S44, Drift Estimation: Based on a validated dataset, estimate the sensor installation drift. The estimation method employs an optimization approach based on an incremental extrinsic parameter model: assuming the drift primarily manifests as small changes in the extrinsic parameter matrix, the current extrinsic parameter is represented as a composite of the most recently validated extrinsic parameter and an incremental transformation. Utilizing valid trajectory points retained in the dynamic trajectory dataset and benchmark pairs in the static target benchmark dataset, the incremental transformation is solved with the objective of minimizing reprojection error. The incremental transformation is decomposed into rotational increments around three coordinate axes (corresponding to pitch, yaw, and roll drift) and translational increments in three directions. The optimization process employs the same robust optimization strategy as the fine calibration optimization (multiple initial values, Huber loss, physical constraints). Output: Sensor installation drift estimation results.

[0158] Specifically, let the most recently validated external parameter be... The current extrinsic parameters are represented as incremental transformations. Combination with reference extrinsic parameters:

[0159]

[0160] The incremental transformation is decomposed into rotational increments about three coordinate axes and translational increments in three directions. Let the rotational increments be represented by axis angles. ,but .

[0161] The optimization objective is to minimize the reprojection error:

[0162]

[0163] in For valid trajectory points or reference pairs.

[0164] S45, Extrinsic Parameter Incremental Correction: Based on the sensor installation drift estimation results, the currently used extrinsic parameter matrix is ​​dynamically updated. The update adopts an incremental superposition method, applying the estimated rotation and translation increments to the current extrinsic parameter matrix to obtain the updated transformation matrix. During the update process, the amplitude of the rotation increment is limited. If the estimated rotation increment exceeds a preset upper limit (e.g., 2°), it is judged as abnormal drift. The upper limit is set based on the following: according to field experience, normal installation drift is usually on the order of ten degrees within several months, and the 2° threshold can cover the normal drift range while excluding sudden and drastic changes (such as traffic accident collisions, human sabotage, etc.). If it is judged as abnormal drift, a manual review alarm is triggered instead of automatic update to prevent serious deviation of extrinsic parameters due to extreme conditions. The updated transformation matrix or manual review alarm signal is output.

[0165] S46, Result Verification and Degradation Strategy: Based on the updated transformation matrix or manual review alarm signal, the reprojection error is recalculated using dynamic target data from subsequent consecutive frames (e.g., 60 frames, corresponding to approximately 2 seconds). If the updated average reprojection error is lower than the verification threshold (e.g., 8 pixels), the correction is deemed effective, and the updated transformation matrix is ​​stored in the third-level reference time period cache as a new reference extrinsic parameter. The final transformation matrix that has passed verification is output for subsequent radar-visual fusion perception. If the error is still higher than the threshold, the correction is deemed insufficient, and the drift estimation is returned to re-estimate the drift amount (using a stricter outlier removal strategy or expanding the data window). If multiple consecutive corrections (e.g., 3 times) still fail to meet the standard, a degradation strategy is triggered: temporarily reverting to the most recently verified reference extrinsic parameter, and simultaneously triggering a system-level alarm, indicating the need for manual on-site recalibration. However, the system can continue to run based on the reference extrinsic parameter to avoid completely losing fusion perception capabilities.

[0166] A radar-visual joint automatic calibration system, comprising:

[0167] The multi-source data preprocessing and time synchronization module preprocesses and synchronizes the raw data collected by radar and cameras, builds a complete data cache pool, and outputs verified synchronized data.

[0168] The raw data acquisition unit simultaneously acquires raw point cloud data from the roadside millimeter-wave radar and raw image frame data from the camera, recording a hardware timestamp for each data frame. For 4D imaging radar, it acquires the output point cloud data, with each point containing range, azimuth, elevation, radial velocity, and reflection intensity information. For traditional 3D millimeter-wave radar, it acquires the periodically output target list or sparse point cloud data, with each point containing range, azimuth, and radial velocity information. The unit outputs the raw radar point cloud set, the raw image set, and the hardware timestamp sequence.

[0169] The radar point cloud preprocessing unit, based on the original radar point cloud set, first performs outlier removal, using a joint thresholding method based on distance and reflection intensity to remove outliers that are significantly outside the road area or have abnormal reflection intensity. Then, it performs ground point filtering, utilizing prior information about ground height in the road scene to filter out clutter points caused by ground reflection. For 4D imaging radar point clouds, it retains the valid target point cloud and its height information; for traditional 3D millimeter-wave radar target lists, it retains the valid targets and their range-azimuth observations. The output is a denoised radar target observation set.

[0170] The image distortion correction unit, based on the original image set, uses a pre-calibrated camera intrinsic parameter matrix and distortion coefficients to correct radial and tangential distortion in each frame, resulting in a distortion-free image. Intrinsic parameter calibration is performed at the factory using a checkerboard calibration method, and the calibration results include focal length, principal point coordinates, and distortion coefficients. The unit outputs a corrected image set.

[0171] The timestamp alignment unit, based on the hardware timestamp sequence, the denoised radar target observation set, and the corrected image set, performs time alignment of radar data using camera frame timestamps as a reference. Since the output method of millimeter-wave radar data varies depending on the model, the alignment strategy is differentiated: for radars that output point clouds in frame format (such as 4D imaging radar), the nearest neighbor matching method is used to select the radar frame with the smallest time difference; for radars that periodically output target lists (such as traditional 3D millimeter-wave radar, with a typical output period of 50ms), a sliding time window method is used to accumulate all radar target observations within the window period (e.g., 100ms), using the midpoint of the window as the timestamp of that group of observations. Multiple observations of the same target within the window period are deduplicated using ID or positional continuity. The system outputs the time-synchronized radar target observation sequence and image frame sequence.

[0172] The synchronization verification unit, based on the time-synchronized radar target observation sequence and image frame sequence, performs a time consistency check on radar target observations corresponding to adjacent image frames. It calculates the time difference of the same physical target between adjacent frames. If the time difference is less than the set synchronization tolerance threshold (e.g., 80ms), the data set is deemed to be synchronized successfully. The percentage of successful data is counted. If the percentage exceeds a preset proportion (e.g., 90%), the verified synchronized data is output; otherwise, a synchronization anomaly alarm is triggered, prompting a check of the hardware synchronization mechanism.

[0173] The data cache pool construction unit builds a multi-level data cache pool based on validated synchronized data. The cache pool has three levels: the first level is a real-time frame cache, storing all synchronized data from the most recent few seconds (e.g., 10 seconds) for fine calibration and error monitoring; the second level is a short-term trajectory cache, storing dynamic target tracking trajectories and static target observation records from the most recent few minutes (e.g., 5 minutes) for drift estimation; the third level is a baseline time period cache, storing baseline data (including static target association pairs, dynamic trajectory association pairs, etc.) from the calibration initialization phase or the most recent successful correction, serving as a clean reference dataset for drift estimation. The cache pool uses a cyclic overwrite mechanism, automatically overwriting the oldest data when the capacity reaches its limit. The completed data cache pool is output.

[0174] The static target-assisted initial calibration module initializes based on prior installation extrinsic parameters, combines verified synchronous data, uses road static targets for initial calibration, solves for initial extrinsic parameters, verifies the well-posedness of the solution, and outputs the initial transformation matrix.

[0175] A priori extrinsic parameter initialization unit is installed. Based on the physical installation configuration information of the millimeter-wave radar and camera, a coarse transformation matrix is ​​constructed from the radar to the camera coordinate system. The physical installation configuration information includes: the equipment installation height difference (obtained through measurement or design drawings), the horizontal installation offset (obtained through measurement), and the nominal values ​​of the installation pitch and yaw angles (obtained through an inclinometer or design drawings). These physical parameters are converted into initial estimates of the rotation matrix and translation vector, forming the coarse transformation matrix. The accuracy requirement for this coarse extrinsic parameter is not high (pitch angle error within ±5°, translation error within ±0.5m is sufficient), and it is only used for coarse spatial correlation of cross-modal targets in subsequent steps. The denoised coarse transformation matrix is ​​output.

[0176] The static object detection and semantic recognition unit, based on image frame sequences from validated synchronous data, employs a deep learning-based semantic segmentation network or object detection network to identify and locate static man-made structures on both sides of the road, including vertical columnar structures such as streetlights, traffic signs, traffic signal poles, and utility poles. For each detected static object, it outputs its two-dimensional bounding box coordinates in the image, semantic category label, and confidence score, forming a static object detection result list. Only static objects with a confidence score higher than a set threshold (e.g., 0.7) and located within the effective perception area of ​​the road are retained. The static object detection result list is then output.

[0177] The static target radar detectability verification unit verifies the radar detectability of each static target based on the denoised coarse transformation matrix, the static target detection result list, and the synchronous radar target observation sequence. The verification method is as follows: the center point of the bottom edge of the two-dimensional bounding box of the static target in the image is projected backward onto the radar coordinate system using the coarse transformation matrix to obtain the theoretical azimuth and range range of the target in the radar coordinate system;

[0178] Let the normalized image coordinates of the center point of the bottom edge of the bounding box of the static target in the image be... The camera intrinsic parameter matrix is The rough rotation matrix is The rough translation vector is The direction of the ray projected back onto the radar coordinate system is:

[0179]

[0180] Let the equation of the road plane in the radar coordinate system be: ( (Given the radar installation height), the intersection point of the ray and the road plane is:

[0181]

[0182] in Direction vector of Quantity.

[0183] From this, the theoretical distance to the target can be obtained. and theoretical azimuth .

[0184] The system searches for radar points within the specified azimuth and range range in the radar target observation sequence. If at least one radar point falls within this range, the static target is determined to be detectable by the radar and retained for subsequent calibration. If no corresponding radar point is detected for several consecutive frames (e.g., 5 frames), the static target is determined to be currently undetectable (possibly due to low material RCS, obstruction, or radar viewing angle issues). It is marked as temporarily unavailable and not included in the current calibration round, but its image detection results are retained for re-verification in subsequent rounds. The system outputs a list of detectable static targets and the corresponding set of radar observation points.

[0185] The radar static target observation extraction unit, based on the radar-detectable static target screening list and the corresponding radar observation point set, extracts the following radar observation information for each detectable static target: range, azimuth, elevation (4D radar), and reflection intensity of the radar point corresponding to the static target. If the same static target corresponds to multiple radar points, the average range, average azimuth, and average reflection intensity of these points are calculated as the radar observation characterization of the target. The unit outputs a radar static target observation set, where each observation includes the target ID, radar range-azimuth observation, elevation (4D radar) observation, and reflection intensity statistics.

[0186] The image static target observation extraction unit, based on the static target detection result list, extracts the following observation information for each static target's two-dimensional bounding box region in the image: image coordinates of the center point of the bottom edge of the bounding box (as an estimate of the target's projected position in the image), vertical pixel coordinates of the center point of the bottom edge of the bounding box (used to assist in distance estimation), and semantic category. The output is an image static target observation set, where each observation contains the target ID, image coordinates, and semantic category.

[0187] The static target cross-modal association unit establishes the correspondence between static targets in the radar coordinate system and the image coordinate system based on radar static target observation sets and image static target observation sets. The association strategy employs multi-level joint matching:

[0188] Let the radar static target observation set be The image static target observation set is The rough transformation matrix is .

[0189] First layer (distance coarse screening): Converts radar observations into three-dimensional points in the radar coordinate system. Projected onto the image coordinate system:

[0190]

[0191] Calculate the image coordinate distance between the projection point and each static target in the image, and only retain those distances less than a set threshold. Candidate matching pairs (e.g., 20% of the image width):

[0192]

[0193] Second layer (angle fine screening): For candidate pairs that have passed the first layer of screening Using camera intrinsic parameters and coarse extrinsic parameters, the vertical pixel coordinates of static targets in the image are determined. The theoretical distance is calculated by reverse calculation. Let the ordinate of the camera's principal point be... focal length is The theoretical pitch angle is:

[0194]

[0195] Theoretical distance is ,in This is a rough pitch angle. Candidate pairs with distance deviations within a reasonable range (e.g., ±30%) are retained.

[0196]

[0197] The third layer (reflection intensity-assisted differentiation): For candidate pairs that pass the first two layers, reflection intensity information is used as an auxiliary differentiation feature. When multiple candidate pairs are spatially close, candidate pairs whose reflection intensity matches the expected target material are prioritized for matching.

[0198] A set of cross-modal correlation pairs for static targets is formed. Each correlation pair contains a correspondence between a radar static target observation and an image static target observation. If the number of correlation pairs is less than the minimum required for subsequent solutions, a static target insufficiency alarm is triggered, prompting the system to wait for more targets to enter the sensing area or to check the sensor status.

[0199] The initial extrinsic parameter constraint solving unit, based on the set of static target cross-modal correlation pairs, solves for the initial extrinsic parameter matrix of the millimeter-wave radar reaching the camera coordinate system. The solution process distinguishes between radar types and considers scene geometric characteristics.

[0200] Let the set of associative pairs be ,in A three-dimensional point in the radar coordinate system. These are the corresponding image coordinates. The extrinsic parameter to be determined is the rotation matrix. Translation vector .

[0201] For 4D imaging radar:

[0202] The objective function to be optimized is:

[0203]

[0204] in For robust loss functions (such as Huber loss). Let be the intrinsic parameter matrix of the camera. Each association pair provides 3 spatial constraints (X,Y,Z) and 2 image observation constraints (u,v), with 6 degrees of freedom for the extrinsic parameters. Theoretically, a minimum of 2 association pairs are required to solve the problem, but to ensure robustness, at least 3 association pairs are required and they must be distributed in different distance intervals.

[0205] For traditional 3D millimeter-wave radar:

[0206] Let the known pitch angle be... Then the rotation matrix It can be expressed as containing only the yaw angle. and roll angle The parameterized form of radar observation. (High-level information missing). The reprojection equation is:

[0207]

[0208] An optimization method based on distance-azimuth consistency is used to solve for the initial extrinsic parameters, with the optimization objective being to minimize the reprojection error.

[0209]

[0210] For non-ideal planar road scenarios with slopes or curves, this step introduces adaptive processing: by analyzing the three-dimensional distribution (4D radar) or range-azimuth distribution (3D radar) of multiple static targets, the local slope angle of the road is estimated; if the slope angle exceeds a set threshold (e.g., 3°), a slope compensation term is added to the optimization objective, or a piecewise planar assumption is used to replace the global planar assumption in order to reduce the impact of planar assumption deviation on the solution of external parameters.

[0211] Output the initial transformation matrix.

[0212] The well-posedness verification unit calculates the reprojection error of all associated pairs based on the initial transformation matrix, and statistically analyzes the average and maximum reprojection errors. If the average reprojection error is below a set threshold (e.g., 20 pixels) and the maximum reprojection error is below a set upper limit (e.g., 50 pixels), the initial extrinsic parameter solution is deemed well-posed, and the initial transformation matrix is ​​output to the dynamic target trajectory matching and fine calibration module. Otherwise, the solution is deemed to have failed, possibly due to insufficient number of associated pairs, poor distribution, or incorrect associations. In this case, the system returns to the static target cross-modal association and re-associates (using stricter screening conditions). If multiple retries still fail, an initial calibration failure alarm is triggered, prompting a check of sensor installation or environmental conditions.

[0213] The dynamic target trajectory matching and fine calibration module performs fine calibration based on the initial transformation matrix and the motion trajectory of the dynamic target, optimizes the extrinsic parameters, and outputs the fine calibration transformation matrix.

[0214] The dynamic target detection unit performs dynamic target detection on both the radar and image sides based on validated synchronization data. On the radar side, a radar target detection algorithm is used to extract dynamic targets such as vehicles and pedestrians from the denoised radar target observation set, outputting the radar observation (range, azimuth, and velocity) for each dynamic target. On the image side, an image target detection algorithm is used to detect dynamic targets such as vehicles and pedestrians from the corrected image, outputting the image bounding box coordinates and semantic category for each dynamic target. Both radar dynamic target sets and image dynamic target sets are output.

[0215] The radar trajectory generation unit, based on a dynamic radar target set, uses a Kalman filter algorithm to track each dynamic target. The state vector contains the target's position (range, azimuth) and velocity (radial velocity, azimuth rate of change) in the radar coordinate system, while the observation vector represents the target position detected by the radar in each frame. Kalman filtering is used for state prediction and observation updates to generate a continuous motion trajectory for each target in the radar coordinate system, outputting a set of radar target tracking trajectories. Each trajectory includes the target ID, a temporal position sequence, and a velocity sequence.

[0216] The image trajectory generation unit tracks each dynamic target based on a set of dynamic targets in the image using a Kalman filter algorithm with the same structure. The state vector contains the target's position (center coordinates of the bottom edge of the bounding box) and velocity (pixel displacement velocity) in the image coordinate system, while the observation vector represents the target position detected in each frame. It generates a continuous motion trajectory for each target in the image coordinate system and outputs a set of image target tracking trajectories. Each trajectory contains the target ID, a temporal position sequence, and a velocity sequence.

[0217] The trajectory association unit, based on the initial transformation matrix, projects each trajectory in the radar target tracking trajectory set onto the image coordinate system to obtain the projected trajectory. During projection, for 4D imaging radar trajectories, the complete 3D coordinates are used for projection; for traditional 3D millimeter-wave radar trajectories, height information is supplemented using a road plane assumption before projection. The projected trajectory is then matched with the image target tracking trajectory set. A joint cost function based on motion similarity and spatial proximity is used, and the Hungarian algorithm is employed to achieve a one-to-one association between the radar trajectory and the image trajectory.

[0218] Set radar trajectory Image trajectory ,in A three-dimensional point in the radar coordinate system. Image coordinates. Project the radar trajectory onto the image coordinate system:

[0219]

[0220] The joint cost function comprehensively considers the projection position deviation, velocity direction consistency, and trajectory length difference between trajectories:

[0221]

[0222] in For time overlap intervals, , The angle between the direction of the trajectory velocity, These are the weighting coefficients.

[0223] The optimal association is solved using the Hungarian algorithm, i.e., the solution is:

[0224]

[0225] in This is a matching mapping from radar trajectory to image trajectory.

[0226] Output the set of trajectory association pairs.

[0227] The fine-calibration optimization unit, based on the set of trajectory association pairs, iteratively optimizes the initial transformation matrix with the objective function of minimizing the reprojection error: The optimization process employs the following strategies to ensure convergence and robustness:

[0228] (1) Multiple initial value strategy: The initial transformation matrix is ​​used as the main initial value, and multiple perturbation initial values ​​are generated at the same time (a small range of random perturbation is added to the rotation and translation components of the initial extrinsic parameters), and optimization is performed separately. The result with the smallest final reprojection error is selected as the optimal solution to avoid getting trapped in local minima.

[0229] (2) Robust loss function: The Huber loss function is used instead of the traditional L2 loss function to reduce the negative impact of abnormal correlation points on the optimization results and enhance the tolerance to erroneous trajectory correlation;

[0230] (3) External parameter physical constraints: The optimization variables are represented by the axis angle of the rotation matrix (to avoid Euler angle gimbal lock problem), and physical constraint regularization terms are introduced, including the orthogonality constraint of the rotation matrix, the reasonable range constraint of the translation vector (such as the difference in installation height between the camera and the radar, and the horizontal offset should be within the physical installation range), and the amplitude limit of the rotation increment (the single optimization rotation change does not exceed 5°) to prevent the optimization results from deviating from the actual installation configuration;

[0231] (4) Differentiated processing for different radar types: For 4D imaging radar, the error term is constructed using the complete three-dimensional coordinates of all corresponding points on the trajectory; for traditional 3D millimeter-wave radar, the height uncertainty weight is added to the error term to reduce the excessive influence of the height dimension on the optimization result, while more trajectory points are accumulated to compensate for the lack of single-point height information.

[0232] Output the fine calibration transformation matrix.

[0233] The periodic drift detection and extrinsic parameter correction module, based on the fine calibration transformation matrix, periodically monitors the calibration error, performs data validity verification when drift is detected, estimates the amount of drift and corrects the extrinsic parameters, and executes a degradation strategy when the correction fails.

[0234] The reprojection error periodic monitoring unit, based on a precise calibration transformation matrix and real-time acquired dynamic target data, projects the radar-detected dynamic target positions onto the image coordinate system for each newly arriving frame of data, calculating the reprojection error between the projected point and the corresponding target position detected in the image. It then statistically analyzes the average and maximum reprojection errors of all dynamic targets over multiple consecutive frames (e.g., 30 frames, corresponding to approximately 1 second), forming a periodic reprojection error statistical sequence. The monitoring process performs a complete statistical analysis at a fixed period (e.g., every 30 seconds) rather than processing each frame in real-time, to reduce computational load.

[0235] The drift determination unit determines sensor installation drift based on a periodic reprojection error statistical sequence. A dual-threshold strategy is employed: when the average reprojection error over multiple consecutive monitoring periods (e.g., 3 periods, corresponding to approximately 90 seconds) exceeds a first threshold (e.g., 10 pixels), an early warning state is activated; if the early warning state persists and the average reprojection error exceeds a second threshold (e.g., 15 pixels), sensor installation drift is formally determined, and a drift trigger signal is output. This dual-threshold strategy effectively avoids false triggers caused by transient noise or sporadic detection errors. Furthermore, if the number of valid dynamic targets in a given monitoring period is insufficient (e.g., less than 3), the drift determination for that period is skipped, preventing misjudgments during periods of target sparseness.

[0236] The data validity verification unit extracts data from the constructed data cache pool based on the drift trigger signal for drift estimation. The data extraction follows the following principles: (1) Dynamic target trajectory data is taken from the most recent time period (e.g., 5 minutes) before drift judgment in the second-level short-term trajectory cache, but validity verification is required - trajectory points with significantly abnormal reprojection errors (more than 3 times the standard deviation of historical benchmark errors) are removed. These abnormal points may have been severely affected by drift; (2) Static target observation data is taken from the benchmark data after the most recent successful calibration or correction in the third-level benchmark time period cache. This benchmark data is regarded as a clean reference that is not contaminated by the current drift; (3) If the data in the benchmark time period cache becomes invalid due to the long time (e.g., more than 30 days) or scene changes (e.g., road construction causes the static target position to change), the benchmark data update process is triggered to re-establish the benchmark association pair using the current extrinsic parameters.

[0237] Output the dynamic trajectory dataset and static target benchmark dataset after validity verification.

[0238] The drift estimation unit estimates the sensor installation drift based on a validated dataset. The estimation method employs an optimization approach based on an incremental extrinsic parameter model: assuming the drift primarily manifests as small changes in the extrinsic parameter matrix, the current extrinsic parameters are represented as a composite of the most recently validated extrinsic parameters and an incremental transformation. Utilizing valid trajectory points retained in the dynamic trajectory dataset and benchmark pairs in the static target benchmark dataset, the incremental transformation is solved with the objective of minimizing reprojection error. The incremental transformation is decomposed into rotational increments around three coordinate axes (corresponding to pitch, yaw, and roll drift) and translational increments in three directions. The optimization process employs the same robust optimization strategy (multiple initial values, Huber loss, and physical constraints) as the fine calibration optimization. The output is the sensor installation drift estimation result.

[0239] Specifically, let the most recently validated external parameter be... The current extrinsic parameters are represented as incremental transformations. Combination with reference extrinsic parameters:

[0240]

[0241] The incremental transformation is decomposed into rotational increments about three coordinate axes and translational increments in three directions. Let the rotational increments be represented by axis angles. ,but .

[0242] The optimization objective is to minimize the reprojection error:

[0243]

[0244] in For valid trajectory points or reference pairs.

[0245] The extrinsic parameter incremental correction unit dynamically updates the currently used extrinsic parameter matrix based on the sensor installation drift estimation results. The update uses an incremental superposition method, applying the estimated rotation and translation increments to the current extrinsic parameter matrix to obtain the updated transformation matrix. During the update process, the amplitude of the rotation increment is limited. If the estimated rotation increment exceeds a preset upper limit (e.g., 2°), it is considered an abnormal drift. This upper limit is set based on industry experience: normal installation drift is typically on the order of ten decimals over several months, and the 2° threshold covers the normal drift range while excluding sudden drastic changes (such as traffic accident impacts, vandalism, etc.). If an abnormal drift is determined, a manual review alarm is triggered instead of automatic updating to prevent severe deviations in extrinsic parameters due to extreme conditions. The updated transformation matrix or a manual review alarm signal is output.

[0246] The result verification and degradation strategy unit, based on the updated transformation matrix or manual review alarm signal, recalculates the reprojection error using dynamic target data from subsequent consecutive frames (e.g., 60 frames, corresponding to approximately 2 seconds). If the updated average reprojection error is lower than the verification threshold (e.g., 8 pixels), the correction is deemed effective, and the updated transformation matrix is ​​stored in the third-level baseline time period cache as a new baseline extrinsic parameter. The final transformation matrix that has passed verification is output for subsequent radar-visual fusion perception. If the error is still higher than the threshold, the correction is deemed insufficient, and the drift estimation is returned to re-estimate the drift amount (using a stricter outlier removal strategy or expanding the data window). If multiple consecutive corrections (e.g., 3 times) still fail to meet the standard, a degradation strategy is triggered: temporarily reverting to the most recently verified baseline extrinsic parameter, while triggering a system-level alarm, indicating the need for manual on-site recalibration. However, the system can continue to operate based on the baseline extrinsic parameter to avoid completely losing its fusion perception capability.

[0247] Although the present invention has been described above with reference to embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of the invention. In particular, as long as there is no structural conflict, the features in the disclosed embodiments can be combined with each other in any manner. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for automatic calibration of radar and vision systems, characterized in that, Includes the following steps: S1, Multi-source data preprocessing and time synchronization: Preprocess and synchronize the raw data collected by radar and camera, build a complete data cache pool, and output the verified synchronized data; S2, Static Target-Assisted Initial Calibration: Based on the initialization of prior extrinsic parameters, combined with the verified synchronous data, the road static target is used for initial calibration, the initial extrinsic parameters are solved, the well-posedness of the solution is verified, and the initial transformation matrix is ​​output. S3, Dynamic Target Trajectory Matching and Fine Calibration: S31, Dynamic Target Detection: Based on the verified synchronization data, perform dynamic target detection on both the radar side and the image side, and output the radar dynamic target set and the image dynamic target set; S32, Radar Trajectory Generation: Based on the radar dynamic target set, the Kalman filter algorithm is used to track each dynamic target, generate a continuous motion trajectory in the radar coordinate system, and output the radar target tracking trajectory set; S33, Image Trajectory Generation: Based on the dynamic target set of the image, the Kalman filter algorithm is used to track each dynamic target, generate a continuous motion trajectory in the image coordinate system, and output the image target tracking trajectory set; S34, Trajectory Association: Based on the initial transformation matrix, each trajectory in the radar target tracking trajectory set is projected onto the image coordinate system, matched with the image target tracking trajectory set, and output a set of trajectory association pairs; S35, Fine calibration optimization: Based on the set of trajectory association pairs, with the objective function of minimizing reprojection error, the initial transformation matrix is ​​iteratively optimized using a multi-initial-value strategy, a robust loss function, and external parameter physical constraints to output a fine calibration transformation matrix; S4, Periodic Drift Detection and Extrinsic Parameter Correction: Based on the precise calibration transformation matrix, the calibration error is periodically monitored. When drift is detected, the data validity is checked, the amount of drift is estimated and the extrinsic parameters are corrected. If the correction fails, a degradation strategy is executed.

2. The automatic calibration method for combined radar and vision as described in claim 1, characterized in that, The specific steps of S1 are as follows: S11, Raw data acquisition: Simultaneously acquire raw point cloud data from roadside millimeter-wave radar and raw image frame data from cameras, record the hardware timestamp of each data frame, and output the raw radar point cloud set, raw image set, and hardware timestamp sequence. S12, Radar point cloud preprocessing: Based on the original radar point cloud set, outlier removal and ground point filtering are performed to output a denoised radar target observation set; S13, Image Distortion Correction: Based on the original image set, distortion correction is performed using a pre-calibrated camera intrinsic parameter matrix and distortion coefficients, and a corrected image set is output. S14, Timestamp Alignment: Based on the hardware timestamp sequence, the denoised radar target observation set and the corrected image set, the radar data is time-aligned with the camera frame timestamp as the reference, and the time-synchronized radar target observation sequence and image frame sequence are output. S15, Synchronization Verification: Based on the time-synchronized radar target observation sequence and image frame sequence, perform time consistency checks on radar target observations corresponding to adjacent image frames. If the proportion of qualified data exceeds the preset ratio, output the verified synchronization data; otherwise, trigger a synchronization anomaly alarm. S16, Data Cache Pool Construction: Based on the verified synchronized data, construct a multi-level data cache pool that includes real-time frame cache, short-term trajectory cache, and baseline time period cache, and output the completed data cache pool.

3. The automatic calibration method for combined radar and vision as described in claim 1, characterized in that, The specific steps of S2 are as follows: S21, Installation of prior extrinsic parameter initialization: Based on the physical installation and configuration information of the millimeter-wave radar and camera, construct a coarse transformation matrix from the radar to the camera coordinate system, and output the denoised coarse transformation matrix; S22, Static Target Detection and Semantic Recognition: Based on the image frame sequence in the verified synchronous data, identify and locate static man-made facilities on both sides of the road, and output a list of static target detection results; S23, Static Target Radar Detectability Verification: Based on the denoised coarse transformation matrix, the static target detection result list, and the synchronous radar target observation sequence, radar detectability verification is performed on each static target, and the radar detectable static target screening list and the corresponding radar observation point set are output. S24, Radar Static Target Observation Extraction: Based on the radar-detectable static target screening list and the corresponding radar observation point set, extract the radar observation information of each detectable static target and output the radar static target observation set; S25, Image Static Target Observation Extraction: Based on the list of static target detection results, extract image observation information for each static target and output the image static target observation set; S26, Static Target Cross-Modal Association: Based on the radar static target observation set and the image static target observation set, a multi-level joint matching strategy is adopted to establish the correspondence between static targets in the radar coordinate system and the image coordinate system, and output the set of static target cross-modal association pairs; S27, Initial extrinsic parameter constraint solution: Based on the set of static target cross-modal correlation pairs, distinguish between 4D imaging radar and traditional 3D millimeter-wave radar types, adopt differentiated solution strategies to solve the initial extrinsic parameter matrix of millimeter-wave radar to camera coordinate system, and output the initial transformation matrix; S28, Solve the well-posedness verification: Based on the initial transformation matrix, calculate the reprojection error of the associated pairs. If the error meets the threshold requirement, output the initial transformation matrix; otherwise, return to re-association or trigger a failure alarm.

4. The automatic calibration method for combined radar and vision as described in claim 1, characterized in that, The specific steps of S4 are as follows: S41, Periodic monitoring of reprojection error: Based on the precise calibration transformation matrix and real-time acquired dynamic target data, the reprojection error is calculated at a fixed period to form a periodic reprojection error statistical sequence. S42, Drift Detection: Based on the periodic reprojection error statistical sequence, a dual threshold strategy is used to determine whether the sensor has experienced installation drift, and a drift trigger signal is output. S43, Data Validity Verification: Based on the drift trigger signal, extract dynamic target trajectory data and static target baseline data from the constructed data cache pool, perform validity verification, and output the verified dataset; S44, Drift estimation: Based on the validated dataset, the sensor installation drift is estimated using an optimization method based on an incremental extrinsic model, and the sensor installation drift estimation result is output. S45, External parameter incremental correction: Based on the sensor installation drift estimation result, the currently used external parameter matrix is ​​dynamically updated, and the amplitude of the rotation increment is limited, and the updated transformation matrix or manual verification alarm signal is output. S46, Result Verification and Degradation Strategy: Based on the updated transformation matrix or manual review of alarm signals, the reprojection error is recalculated using dynamic target data from subsequent consecutive frames. If the error is lower than the verification threshold, the final transformation matrix that has passed verification is output and stored in the baseline buffer; otherwise, the drift amount is re-estimated or a degradation strategy is triggered.

5. A radar-visual combined automatic calibration system, characterized in that, include: The multi-source data preprocessing and time synchronization module preprocesses and synchronizes the raw data collected by radar and cameras, builds a complete data cache pool, and outputs verified synchronized data. The static target-assisted initial calibration module initializes based on prior installation extrinsic parameters, combines verified synchronous data, uses road static targets for initial calibration, solves for initial extrinsic parameters, verifies the well-posedness of the solution, and outputs the initial transformation matrix. The dynamic target trajectory matching and fine calibration module includes a dynamic target detection unit, a radar trajectory generation unit, an image trajectory generation unit, a trajectory association unit, a fine calibration optimization unit, and a periodic drift detection and extrinsic parameter correction module; The dynamic target detection unit performs dynamic target detection on both the radar side and the image side based on the verified synchronization data, and outputs radar dynamic target sets and image dynamic target sets. The radar trajectory generation unit, based on the radar dynamic target set, uses the Kalman filter algorithm to track each dynamic target, generates a continuous motion trajectory in the radar coordinate system, and outputs a radar target tracking trajectory set; The image trajectory generation unit, based on the image dynamic target set, uses the Kalman filter algorithm to track each dynamic target, generates a continuous motion trajectory in the image coordinate system, and outputs an image target tracking trajectory set; The trajectory association unit, based on the initial transformation matrix, projects each trajectory in the radar target tracking trajectory set onto the image coordinate system, matches it with the image target tracking trajectory set, and outputs a set of trajectory association pairs. The fine calibration optimization unit, based on the set of trajectory association pairs, takes minimizing the reprojection error as the objective function, and employs a multi-initial-value strategy, a robust loss function, and external parameter physical constraints to iteratively optimize the initial transformation matrix and output the fine calibration transformation matrix. The periodic drift detection and extrinsic parameter correction module periodically monitors calibration errors based on the fine calibration transformation matrix. When drift is detected, it performs data validity verification, estimates the amount of drift, and corrects the extrinsic parameters. If the correction fails, it executes a degradation strategy.

6. The automatic calibration system for combined radar and vision as described in claim 5, characterized in that, The multi-source data preprocessing and time synchronization module includes: The raw data acquisition unit synchronously acquires raw point cloud data from the roadside millimeter-wave radar and raw image frame data from the camera, records the hardware timestamp of each data frame, and outputs the raw radar point cloud set, the raw image set, and the hardware timestamp sequence. The radar point cloud preprocessing unit performs outlier removal and ground point filtering based on the original radar point cloud set, and outputs a denoised radar target observation set. The image distortion correction unit performs distortion correction based on the original image set using a pre-calibrated camera intrinsic parameter matrix and distortion coefficients, and outputs a corrected image set. The timestamp alignment unit, based on the hardware timestamp sequence, the denoised radar target observation set and the corrected image set, uses the camera frame timestamp as a reference to perform time alignment on the radar data and outputs the time-synchronized radar target observation sequence and image frame sequence. The synchronization verification unit performs a time consistency check on the radar target observations corresponding to adjacent image frames based on the time-synchronized radar target observation sequence and image frame sequence. If the proportion of qualified data exceeds the preset proportion, the unit outputs the synchronized data that has passed the verification; otherwise, it triggers a synchronization anomaly alarm. The data cache pool construction unit, based on verified synchronized data, constructs a multi-level data cache pool including real-time frame cache, short-term trajectory cache, and baseline time period cache, and outputs the completed data cache pool.

7. The automatic calibration system for combined radar and vision as described in claim 5, characterized in that, The static target-assisted initial calibration module includes: Install the prior extrinsic parameter initialization unit, construct a coarse transformation matrix from the radar to the camera coordinate system based on the physical installation and configuration information of the millimeter-wave radar and camera, and output the denoised coarse transformation matrix; The static target detection and semantic recognition unit identifies and locates static man-made facilities on both sides of the road based on the image frame sequence in the verified synchronous data, and outputs a list of static target detection results. The static target radar detectability verification unit verifies the radar detectability of each static target based on the denoised coarse transformation matrix, the static target detection result list, and the synchronous radar target observation sequence, and outputs a radar-detectable static target screening list and the corresponding radar observation point set. The radar static target observation extraction unit extracts radar observation information for each detectable static target based on the radar-detectable static target screening list and the corresponding radar observation point set, and outputs the radar static target observation set. The image static target observation and extraction unit extracts image observation information for each static target based on the static target detection result list and outputs an image static target observation set; The static target cross-modal association unit, based on the radar static target observation set and the image static target observation set, adopts a multi-level joint matching strategy to establish the correspondence between static targets in the radar coordinate system and the image coordinate system, and outputs a set of static target cross-modal association pairs; The initial extrinsic parameter constraint solving unit, based on the static target cross-modal correlation pair set, distinguishes between 4D imaging radar and traditional 3D millimeter-wave radar types, and uses a differentiated solving strategy to solve the initial extrinsic parameter matrix of the millimeter-wave radar to the camera coordinate system, and outputs the initial transformation matrix; Solve the well-posed verification unit. Based on the initial transformation matrix, calculate the reprojection error of the associated pairs. If the error meets the threshold requirement, output the initial transformation matrix; otherwise, return to re-association or trigger a failure alarm.

8. The automatic calibration system for combined radar and vision as described in claim 5, characterized in that, The periodic drift detection and extrinsic parameter correction module includes: The reprojection error periodic monitoring unit calculates the reprojection error at fixed intervals based on the precise calibration transformation matrix and real-time acquired dynamic target data, forming a periodic reprojection error statistical sequence. The drift determination unit, based on the periodic reprojection error statistical sequence, uses a dual threshold strategy to determine whether the sensor has experienced installation drift and outputs a drift trigger signal. The data validity verification unit, based on the drift trigger signal, extracts dynamic target trajectory data and static target baseline data from the constructed data cache pool, performs validity verification, and outputs the verified dataset. The drift estimation unit estimates the sensor installation drift based on a validated dataset using an optimization method based on an incremental extrinsic model, and outputs the sensor installation drift estimation result. The extrinsic parameter incremental correction unit dynamically updates the currently used extrinsic parameter matrix based on the sensor installation drift estimation results, limits the amplitude of the rotation increment, and outputs an updated transformation matrix or a manual verification alarm signal. The result verification and degradation strategy unit, based on the updated transformation matrix or the manual review alarm signal, recalculates the reprojection error using dynamic target data from subsequent consecutive frames. If the error is lower than the verification threshold, it outputs the final transformation matrix that has passed verification and stores it in the baseline buffer; otherwise, it returns to re-estimate the drift amount or triggers the degradation strategy.

Images