Radar and camera hierarchical automatic calibration method and system based on target trajectory analysis

Abstract

The application discloses a radar and camera hierarchical automatic calibration method and system based on target trajectory analysis, and the method comprises the following steps: S1, generating an initial external parameter matrix by measuring the relative position and attitude angle of the radar and the camera; S2, performing multi-frame correlation on the radar point cloud to generate a radar target trajectory with a unique ID; detecting a target bounding box in the camera image and generating a visual target trajectory; S3, projecting the radar target to the image coordinate system by using the initial external parameter matrix, and correlating the radar target and the visual target by a preset constraint condition; S4, screening matching point pairs with a confidence degree greater than a preset value from the successfully correlated trajectories; S5, taking the screened matching point pairs as an optimized sample set, constructing a weighted projection error function, taking the initial external parameter matrix as a starting point, and iteratively optimizing the external parameter matrix by using an L-M algorithm until the error converges to a preset threshold value, and outputting a fine calibration external parameter matrix. The application has the advantages of high calibration precision.

Description

Technical Field

[0001] This invention mainly relates to the field of radar and vision sensor calibration technology, specifically to a radar and camera hierarchical automatic calibration method and system based on target trajectory analysis. Background Technology

[0002] Radar and vision sensor calibration is the core foundation of radar-visual fusion technology in intelligent security systems, and its accuracy directly determines the reliability of upper-level applications such as target tracking and intrusion event analysis. With breakthroughs in 4D millimeter-wave radar's capabilities in stereoscopic spatial perception and high-density point cloud analysis, collaborative perception between radar and vision is increasingly being applied to all-weather monitoring and environmental perception scenarios in the security field. However, the calibration of extrinsic parameters between sensors still faces significant challenges: 4D millimeter-wave radar acquires target distance, velocity, azimuth, and elevation information through electromagnetic wave reflection, while vision systems rely on optical imaging to analyze the semantic and textural features of targets. The data modalities of the two systems differ significantly, requiring high-precision calibration to achieve spatiotemporal alignment. Large calibration errors can lead to the failure of multi-source data fusion; for example, in security intrusion alarm tasks, the deviation between radar ranging and visual target localization directly affects the accuracy of the alarm.

[0003] Existing 4D radar and vision sensor calibration methods can be mainly divided into two categories. One category is the static calibration method, which relies on manually setting up special calibration objects such as checkerboards and corner reflectors. It involves hardware-triggered synchronous acquisition of radar point clouds and images, using visual algorithms to extract the corner coordinates of the images, manually selecting the centroids of the corner reflectors in the radar point cloud, and manually associating image pixels with radar coordinates to form matching pairs. The rigid transformation relationship between the two coordinate systems is then calculated to obtain the extrinsic parameter matrix. This method requires precise control of the size and placement of the calibration objects, and the calibration objects need to cover the joint field of view of the sensors. Large-scale deployment consumes a significant amount of time. Furthermore, when the sensor undergoes slight displacement due to mechanical vibration or changes in ambient temperature, the calibration process needs to be repeated, making it difficult to support the long-term maintenance needs of the equipment.

[0004] Specifically, the static calibration board method requires manually deploying high-density calibration objects to cover the joint field of view. These calibration objects are susceptible to environmental interference (wind and rain erosion, vehicle collisions) and cannot dynamically respond to changes in device pose. When the sensor's pose changes due to mechanical vibration or environmental variations, if the error exceeds the calibration tolerance, a full-process calibration is required, significantly increasing maintenance complexity. Furthermore, the static calibration method requires manual labeling of matching pairs between radar point clouds and image corner points. Each calibration is time-consuming and cannot be automated. Manual matching is also prone to introducing random errors, such as incorrectly selecting outliers, causing the optimized extrinsic parameter matrix to get stuck in a local optimum, requiring repeated iterative corrections and greatly impacting the deployment progress of large-scale equipment.

[0005] Another method involves directly measuring the relative position and attitude angles of the radar and camera using tools such as laser rangefinders and inclinometers. Rotation matrix components are calculated based on Euler angles, and a total rotation matrix is ​​synthesized. The displacement is then converted into translation vectors, constructing a 4×4 homogeneous transformation matrix, which represents the measured calibration parameters. However, manual measurement errors can lead to accumulated errors in the extrinsic parameter matrix. Furthermore, this method has poor installation tolerance and requires strict geometric flatness of the equipment mounting platform, making it prone to calibration failure due to structural deformation during deployment.

[0006] Specifically, the manual measurement method relies on manual operation of laser rangefinders and inclinometers. The measuring tools have inherent errors, and the operators also have subjective biases, such as the bracket mounting surface not being tightly fitted. This leads to a systematic deviation in the external parameter matrix. This error will amplify linearly with the measurement distance. That is, the farther the observed target is, the greater the projection offset caused by the error. This error makes it impossible for the radar target and the visual detection frame to be accurately aligned, thus causing downstream tasks such as zone alarms to fail. Summary of the Invention

[0007] To address the technical problems existing in the prior art, this invention provides a radar and camera hierarchical automatic calibration method and system based on target trajectory analysis with high calibration accuracy.

[0008] To solve the above-mentioned technical problems, the technical solution proposed by this invention is as follows:

[0009] A radar and camera hierarchical automatic calibration method based on target trajectory analysis includes the following steps:

[0010] S1. By measuring the relative position and attitude angle between the radar and the camera, calculate the rotation matrix and translation vector, and combine them to generate the initial extrinsic parameter matrix;

[0011] S2. Simultaneously acquire four-dimensional point cloud data from the radar and camera image streams, and align them using timestamps; perform multi-frame association on the radar point cloud to generate radar target trajectories with unique IDs; detect target bounding boxes in camera images and generate visual target trajectories through cross-frame matching;

[0012] S3. Project the radar target onto the image coordinate system using the initial extrinsic parameter matrix from step S1, and associate the radar with the visual target using preset constraints.

[0013] S4. Filter matching point pairs with a confidence level greater than a preset value from the successfully associated trajectories;

[0014] S5. Using the selected matching point pairs as the optimization sample set, construct a weighted projection error function. Starting from the initial extrinsic parameter matrix of S1, use the LM algorithm to iteratively optimize the extrinsic parameter matrix until the error converges to the preset threshold, and output the finely calibrated extrinsic parameter matrix.

[0015] Preferably, after step S5, step S6 is also included: during online operation, the reprojection error is monitored in real time. If the continuous sampling error exceeds the tolerance threshold, the process is automatically jumped to step S2 to restart the calibration process.

[0016] Preferably, in step S3, the preset constraints include:

[0017] Using spatial consistency constraints, the distance between the radar target and the center or bottom of the image detection box is calculated. If the distance is less than a set threshold, the target is considered to be the same target.

[0018] Preferably, in step S3, the preset constraint conditions further include:

[0019] Using dynamic trajectory constraints, the system determines whether the motion trajectories of radar targets and visual targets are consistent through continuous matching of multiple frames. If they are consistent, they are considered to be the same target.

[0020] Preferably, in step S4, the following three strategies are used sequentially to filter matching point pairs with a confidence level greater than a preset value:

[0021] Curvature extremum point extraction: By numerically differentiating the target motion trajectory and calculating the curvature, the top m trajectory points with the largest curvature values ​​are selected;

[0022] Spatial uniformity constraint: Divide the area into multiple segments according to distance, obtain at least 3 point pairs in each segment, and ensure that the sampling points are evenly distributed in the central and edge areas of the field of view;

[0023] Noise filtering: Remove stationary points, high-speed points, low-confidence visual inspection boxes, and points with excessive projection deviation.

[0024] Preferably, in step S5, the weighted projection error function E is specifically:

[0025]

[0026] in, N Indicates the number of matched point pairs; As weight; i Indicates the index of the matching point pair; K Indicates camera intrinsic parameters; T This represents the extrinsic parameter matrix that needs to be optimized. and These represent the pixel coordinates of the center of the image detection box and the three-dimensional coordinates of the radar target in the radar coordinate system, respectively.

[0027] Preferably, the specific process of step S1 is as follows:

[0028] The relative position and attitude angle between the radar and the camera are measured using tools;

[0029] Then, the orbital distance is calculated using the relative position and attitude angle between the radar and the camera. Z axis, Y axis, X The rotation components of the axis are multiplied by matrix to obtain a 3x3 rotation matrix, which is used to correct the directional difference between the radar and the camera. The measured relative position relationship is then converted into a translation vector. Finally, the rotation matrix and the translation vector are combined to obtain the transformation relationship between the camera coordinate system and the radar coordinate system, and the initial extrinsic parameter matrix is ​​obtained.

[0030] Preferably, the transformation relationship between the camera coordinate system and the radar coordinate system is as follows:

[0031]

[0032] in, , , This represents the coordinates of the observed target in the camera coordinate system. , , This represents the coordinates of the observed target in the radar coordinate system; R , t Represents the rotation matrix and translation vector.

[0033] Preferably, in step S2, the four dimensions of the point cloud data include target distance, horizontal azimuth angle, vertical height, and velocity.

[0034] The present invention also discloses a radar and camera hierarchical automatic calibration system based on target trajectory analysis, including a memory and a processor connected to each other. The memory stores a computer program, which executes the steps of the method described above when run by the processor.

[0035] Compared with the prior art, the advantages of the present invention are as follows:

[0036] This method uses tools such as laser rangefinders and tiltmeters to quickly measure the relative position and angle of the radar and camera, generating an initial extrinsic parameter matrix for coarse calibration. Then, the radar collects four-dimensional information (distance, velocity, azimuth, and elevation angle) of the target in real time, while the camera uses a visual deep learning algorithm to detect, identify, and locate the target, obtaining the target bounding box. Simultaneously, the radar and camera data streams are aligned using timestamps. Based on the coarse calibration parameters, the radar is projected onto the image coordinate system, initially matching the spatiotemporal positions of the radar target and the visual target. Finally, high-confidence matching pairs are selected by calculating the distance between the radar target projection point and the center point of the image target bounding box, combined with constraints such as the consistency of the target's motion trajectory, to eliminate false detections or noise-interfering targets. Using the matching pairs as constraints, a weighted reprojection error function is constructed, and a nonlinear optimization algorithm iteratively optimizes the extrinsic parameter matrix to complete fine calibration.

[0037] The method of this invention can use vehicles or pedestrians in the natural environment as dynamic calibration objects for calibration, without the need for large-scale dense deployment of calibration objects. At the same time, it can automatically match radar and camera targets, use a coarse-fine calibration hierarchical strategy to balance efficiency and accuracy, and support long-term online calibration, greatly reducing manual intervention.

[0038] This invention first employs a coarse-fine calibration grading strategy, where manual measurements are used only for primary calibration. Then, the external parameter matrix is ​​automatically optimized using trajectory data of dynamic targets. Since the trajectory of dynamic targets has spatiotemporal continuity, it can provide high-density, highly consistent matching point pairs. Nonlinear optimization methods are used to suppress manual measurement errors and improve calibration accuracy.

[0039] This invention utilizes dynamic targets in the natural environment, such as vehicles and pedestrians, to replace dedicated calibration objects. It automatically extracts calibration features through a trajectory filtering algorithm. The movement of dynamic targets in the sensor's joint field of view can cover multiple distances and angles, avoiding the spatial limitations of manually setting up calibration objects.

[0040] This invention designs an external parameter drift detection mechanism. When the reprojection error exceeds a threshold, the calibration process is automatically triggered. The continuous data stream of the dynamic target is used to correct the external parameter calibration results in real time, so as to cope with the impact of changes in external conditions such as equipment vibration on the calibration accuracy. Attached Figure Description

[0041] Figure 1 The flowchart below shows an embodiment of the automatic calibration method of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0043] like Figure 1 As shown, the radar and camera hierarchical automatic calibration method based on target trajectory analysis provided in this embodiment of the invention includes the following steps:

[0044] S1. Obtain equipment installation parameters, calculate the external parameter matrix, and complete the coarse calibration.

[0045] The extrinsic parameter matrix represents the transformation from the radar coordinate system to the camera coordinate system. The core of coarse calibration is to quickly establish preliminary alignment rules between the two coordinate systems by manually measuring the physical installation relationship between the radar and the camera. The specific implementation method is as follows:

[0046] First, traditional tools (such as measuring tape, laser rangefinder, and inclinometer) are used to directly measure the relative position and orientation of the two devices, that is, to measure the relative positional relationship and attitude angle of the radar and camera;

[0047] Then, the orbital distance is calculated using the relative positions and attitude angles of the radar and camera. Zaxis, Y axis, X The rotation components of the axis are multiplied by matrix to obtain a 3x3 rotation matrix, which is used to correct the directional difference between the radar and the camera. The measured relative position relationship is then converted into a translation vector. Finally, the rotation matrix and the translation vector are combined to obtain the transformation relationship between the camera coordinate system and the radar coordinate system, and the initial extrinsic parameter matrix is ​​obtained to complete the coarse calibration.

[0048] In practical applications, the extrinsic parameter matrix is ​​multiplied by the camera intrinsic parameters to obtain the projection matrix, which is then used to project the radar target onto the image. The projection point is then observed to see if it is basically aligned with the target position. If there is a significant offset, the measurement error needs to be rechecked.

[0049] Specifically, the transformation relationship between the radar coordinate system and the camera coordinate system is as follows:

[0050]

[0051] in, , , This represents the coordinates of the observed target in the camera coordinate system. , , This represents the coordinates of the observed target in the radar coordinate system; R , t This represents the rotation matrix and translation vector obtained through measurement. Combining them yields the extrinsic parameter matrix, which allows the observed target to be transformed from the radar coordinate system to the camera coordinate system.

[0052] S2. Detection and tracking of radar and visual targets.

[0053] 4D millimeter-wave radar transmits frequency-modulated electromagnetic waves and receives reflected signals from targets, resolving four-dimensional information about the target: range, horizontal azimuth, vertical height, and velocity, forming dense point cloud data. Based on the spatiotemporal continuity of the point cloud, 4D radar can detect dynamic targets such as vehicles and pedestrians, and construct the target's trajectory by associating multiple frames of data (such as Kalman filtering and trajectory prediction), assigning a unique tracking ID to achieve stable radar target tracking.

[0054] The vision system captures environmental images through optical lenses, uses deep learning models to identify target objects in the scene, and outputs detection results with bounding boxes and category labels. It generates continuous trajectories and assigns IDs through cross-frame target matching (such as feature similarity and motion consistency), thereby achieving visual target detection and tracking.

[0055] S3. Based on the coarse calibration results, map the radar target onto the image for matching.

[0056] The initial extrinsic parameter matrix obtained from coarse calibration establishes a preliminary spatial correspondence between the radar and camera coordinate systems. With radar and camera data synchronized spatiotemporally, this matrix converts radar-detected target points (such as the 3D coordinates of a pedestrian's centroid) into coordinates in the camera coordinate system. These coordinates are then projected onto the image plane using camera intrinsic parameters, yielding the estimated pixel position of the target in the image. A series of constraints are applied to the matching process, such as spatial consistency constraints (calculating the distance between the radar target and the center or bottom of the image detection box; if the distance is less than a set threshold, the target is considered the same); and dynamic trajectory constraints (multi-frame continuous matching determines whether the radar and visual target's motion trajectories are consistent; if they are, the target is considered the same). After radar and camera target matching is complete, the radar and camera target trajectories are fused and associated, and a new target ID is assigned.

[0057] S4. Filter suitable matching point pairs

[0058] During the automatic calibration process of radar vision, the target's motion trajectory is optimally S-shaped. At this point, a trajectory point filtering strategy can be used to automatically select points and optimize calibration accuracy. The following three strategies are mainly adopted in sequence for point pair filtering:

[0059] (1) Curvature extreme point extraction: By performing numerical differentiation on the target motion trajectory and calculating the curvature, select multiple trajectory points with the largest curvature values ​​(the first m trajectory points).

[0060] (2) Spatial uniformity constraint: Divide the points into multiple segments according to the distance of the points, and obtain at least 3 points in each segment. At the same time, ensure that the sampling points are evenly distributed in the central and edge areas of the field of view, and ensure that relatively uniform points are obtained in space.

[0061] (3) Remove noise interference and eliminate stationary or excessively fast points. At the same time, filter out points with low confidence based on the matching results. For example, when the radar target is mapped to the image coordinate system, if the distance between the radar and the visual detection box is greater than a certain threshold, it is considered that its confidence is not high enough and is eliminated to avoid interfering with the optimization of calibration accuracy. At the same time, it is also necessary to filter point pairs in combination with the visual detection results. For example, if the confidence score of the visual target detection box itself is not high, it may interfere with the point pair matching due to object occlusion or mismatch. In this case, these point pairs also need to be filtered.

[0062] This invention employs a target trajectory-based matching point pair filtering strategy. It filters matching point pairs based on various constraints. First, it uses extreme points to filter radar points by calculating trajectory curvature. Second, based on spatial uniformity constraints, all points are divided into multiple distance segments to ensure a relatively balanced number of point pairs across each distance segment in the matching point pair set. Finally, it completes the matching point pair filtering by eliminating some mismatched points or point pairs with low confidence.

[0063] S5. Perform nonlinear optimization on the coarse calibration results.

[0064] In selecting suitable matching point pairs, Strategy 1 uses curvature calculation to filter them. Generally, points with greater curvature are more reliable in the calibration and optimization process. Therefore, during optimization, a weighted nonlinear optimization method is used to construct a weighted error function instead of the traditional reprojection error function as the objective function. :

[0065]

[0066] Among them, weight The curvature is positively correlated with the amount of information contained in a point (the greater the curvature, the richer the information it contains), for example, points with greater curvature have a 50% higher weight. i Indicates the index of the matching point pair. K Indicates camera intrinsic parameters. T This represents the objective to be optimized, i.e., the extrinsic parameter matrix. and These represent the pixel coordinates of the image detection box center and the three-dimensional coordinates of the radar target in the radar coordinate system, respectively. N This indicates the number of matching point pairs selected based on the constraints, generally... N The range is 7 to 15.

[0067] Employing nonlinear optimization algorithms, such as LM The algorithm optimizes the target. During the iteration process, the iteration stops when the reprojection error is less than a set threshold. The resulting extrinsic parameter matrix is ​​then obtained. T These are the optimized calibration parameters, and the fine calibration is complete.

[0068] The weighted nonlinear optimization method of this invention weights the reprojection error function according to the curvature of the trajectory at different stages, so that the point pairs with higher reliability occupy a larger weight in the optimization process. The weighted reprojection error function makes the iterative target more reasonable, thereby improving the accuracy of the final calibration parameters.

[0069] In the iterative optimization strategy of weighted projection error, a reinforcement learning model can be constructed, with the reprojection error reduction rate as the reward function, to dynamically select the optimal matching point pair, replacing the inherent rules such as curvature and noncollinearity in the design. Alternatively, an extrinsic physical constraint term can be introduced into the reprojection error function to construct a multi-objective optimization problem. This approach may avoid the extrinsic matrix from violating physical laws during the optimization process. Therefore, the multi-optimization model can replace the single-objective optimization model in this method, which may require mathematical tools including but not limited to nonlinear least squares, reinforcement learning, and multi-objective optimization.

[0070] S6. During online operation, random sampling and monitoring of reprojection error are performed. If there is a significant increasing trend, the above steps need to be repeated for calibration correction.

[0071] This method uses tools such as laser rangefinders and tiltmeters to quickly measure the relative position and angle of the radar and camera, generating an initial extrinsic parameter matrix for coarse calibration. Then, the radar collects four-dimensional information (distance, velocity, azimuth, and elevation angle) of the target in real time, while the camera uses a visual deep learning algorithm to detect, identify, and locate the target, obtaining the target bounding box. Simultaneously, the radar and camera data streams are aligned using timestamps. Based on the coarse calibration parameters, the radar is projected onto the image coordinate system, initially matching the spatiotemporal positions of the radar target and the visual target. Finally, high-confidence matching pairs are selected by calculating the distance between the radar target projection point and the center point of the image target bounding box, combined with constraints such as the consistency of the target's motion trajectory, to eliminate false detections or noise-interfering targets. Using the matching pairs as constraints, a weighted reprojection error function is constructed, and a nonlinear optimization algorithm iteratively optimizes the extrinsic parameter matrix to complete fine calibration.

[0072] The method of this invention can use vehicles or pedestrians in the natural environment as dynamic calibration objects for calibration, without the need for large-scale dense deployment of calibration objects. At the same time, it can automatically match radar and camera targets, use a coarse-fine calibration hierarchical strategy to balance efficiency and accuracy, and support long-term online calibration, greatly reducing manual intervention.

[0073] This invention first employs a coarse-fine calibration grading strategy, where manual measurements are used only for primary calibration. Then, the external parameter matrix is ​​automatically optimized using trajectory data of dynamic targets. Since the trajectory of dynamic targets has spatiotemporal continuity, it can provide high-density, highly consistent matching point pairs. Nonlinear optimization methods are used to suppress manual measurement errors and improve calibration accuracy.

[0074] This invention utilizes dynamic targets in the natural environment, such as vehicles and pedestrians, to replace dedicated calibration objects. It automatically extracts calibration features through a trajectory filtering algorithm. The movement of dynamic targets in the sensor's joint field of view can cover multiple distances and angles, avoiding the spatial limitations of manually setting up calibration objects.

[0075] This invention designs an external parameter drift detection mechanism. When the reprojection error exceeds a threshold, the calibration process is automatically triggered. The continuous data stream of the dynamic target is used to correct the external parameter calibration results in real time, so as to cope with the impact of changes in external conditions such as equipment vibration on the calibration accuracy.

[0076] The hierarchical automatic calibration method of this invention first involves manually pre-determining the spatial installation parameters of the radar and camera. These parameters are then used to calculate the mapping relationship between the radar coordinate system and the camera coordinate system, obtaining an initial extrinsic parameter matrix to complete the coarse calibration of the radar and camera. During the data acquisition phase, spatiotemporally aligned radar and image data pairs are acquired. Targets in the radar and image data are detected and tracked in real time. The extrinsic parameter matrix obtained from the coarse calibration is used to project the radar target onto the image plane. Preliminary association matching is constructed based on the distance between the centroid of the radar target and the target detection box in the image. For successfully matched dynamic targets, the system continuously records their motion trajectory data. In the trajectory filtering stage, ... By designing multi-dimensional evaluation indicators (including spatial uniformity, trajectory curvature characteristics, geometric noncollinearity, etc.), the most representative optimal matching point pair set is selected from the joint field of view as the optimization sample set. Using the coarsely calibrated extrinsic parameter matrix as the initial value, the weighted projection error is used as the objective function for optimization. The optimization sample set is used to perform nonlinear iterative optimization on the coarse calibration result. When the reprojection error converges to a preset threshold, the optimized extrinsic parameter matrix is ​​output as the final calibration result. During online operation and maintenance, random sampling is used to monitor whether there is a significant increasing trend in the reprojection error. If the reprojection error increases significantly, it proves that external conditions such as hardware pose may have changed, and the calibration needs to be corrected.

[0077] This method eliminates the need for specialized equipment such as checkerboard patterns and corner reflectors, significantly reducing time and hardware costs during calibration. Furthermore, when sensors experience slight displacement due to vibration or temperature deformation, the system automatically triggers extrinsic parameter optimization based on real-time dynamic target data, maintaining calibration accuracy. During multi-sensor data fusion, a trajectory filtering algorithm automatically selects the optimal matching point pairs, constructs a weighted projection error function for iterative optimization, and eliminates the influence of human subjectivity on the calibration results, resulting in a higher level of intelligence. It requires no specific calibration objects; only common objects like people and vehicles are used as calibration targets, adapting to different scenarios and enabling rapid deployment, saving hardware and time costs.

[0078] This invention also discloses a radar and camera hierarchical automatic calibration system based on target trajectory analysis, comprising an interconnected memory and a processor. The memory stores a computer program, which, when run by the processor, executes the steps of the method described above. The calibration system of this invention corresponds to the calibration method described above and also possesses the advantages described therein.

[0079] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium includes: any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. The memory is used to store computer programs and / or modules. The processor implements various functions by running or executing the computer programs and / or modules stored in the memory, and by calling data stored in the memory. The memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0080] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A radar and camera hierarchical automatic calibration method based on target trajectory analysis, characterized in that, Including the following steps: S1. By measuring the relative position and attitude angle between the radar and the camera, calculate the rotation matrix and translation vector, and combine them to generate the initial extrinsic parameter matrix; S2. Synchronously acquire four-dimensional point cloud data from the radar and camera image streams, and align them using timestamps; Multi-frame correlation is performed on the radar point cloud to generate radar target trajectories with unique IDs; target bounding boxes in camera images are detected, and visual target trajectories are generated through cross-frame matching. S3. Project the radar target onto the image coordinate system using the initial extrinsic parameter matrix from step S1, and associate the radar with the visual target using preset constraints. S4. Filter matching point pairs with a confidence level greater than a preset value from the successfully associated trajectories; S5. Using the selected matching point pairs as the optimization sample set, construct a weighted projection error function. Starting from the initial extrinsic parameter matrix of S1, use the LM algorithm to iteratively optimize the extrinsic parameter matrix until the error converges to the preset threshold, and output the finely calibrated extrinsic parameter matrix. In step S4, the following three strategies are used sequentially to filter matching point pairs with a confidence level greater than a preset value: Curvature extremum point extraction: By numerically differentiating the target motion trajectory and calculating the curvature, the top m trajectory points with the largest curvature values ​​are selected; Spatial uniformity constraint: Divide the area into multiple segments according to distance, obtain at least 3 point pairs in each segment, and ensure that the sampling points are evenly distributed in the central and edge areas of the field of view; Noise filtering: Remove stationary points, high-speed points, low-confidence visual inspection boxes, and points with excessive projection deviation.

2. The radar and camera hierarchical automatic calibration method based on target trajectory analysis according to claim 1, characterized in that, After step S5, step S6 is also included: during online operation, the reprojection error is monitored in real time. If the continuous sampling error exceeds the tolerance threshold, the process is automatically jumped to step S2 to restart the calibration process.

3. The radar and camera hierarchical automatic calibration method based on target trajectory analysis according to claim 1 or 2, characterized in that, In step S3, the preset constraints include: Using spatial consistency constraints, the distance between the radar target and the center or bottom of the image detection box is calculated. If the distance is less than a set threshold, the target is considered to be the same target.

4. The radar and camera hierarchical automatic calibration method based on target trajectory analysis according to claim 3, characterized in that, In step S3, the preset constraints also include: Using dynamic trajectory constraints, the system determines whether the motion trajectories of radar targets and visual targets are consistent through continuous matching of multiple frames. If they are consistent, they are considered to be the same target.

5. The automatic radar and camera classification calibration method based on target trajectory analysis according to claim 1 or 2, characterized in that, In step S5, the weighted projection error function E is specifically as follows: Where N represents the number of matching point pairs; is the weight; i represents the index of the matching point pair; K represents the camera intrinsic parameters; T represents the extrinsic parameter matrix to be optimized; and These represent the pixel coordinates of the center of the image detection box and the three-dimensional coordinates of the radar target in the radar coordinate system, respectively.

6. The automatic calibration method for radar and camera classification based on target trajectory analysis according to claim 1 or 2, characterized in that, The specific process of step S1 is as follows: The relative position and attitude angle between the radar and the camera are measured using tools; Then, the rotation components around the Z-axis, Y-axis, and X-axis are calculated based on the relative position and attitude angle of the radar and camera, respectively. A 3x3 rotation matrix is ​​obtained by matrix multiplication and used to correct the directional difference between the radar and the camera. The measured relative position relationship is then converted into a translation vector. Finally, the rotation matrix and the translation vector are combined to obtain the transformation relationship between the camera coordinate system and the radar coordinate system, and the initial extrinsic parameter matrix is ​​obtained.

7. The radar and camera hierarchical automatic calibration method based on target trajectory analysis according to claim 6, characterized in that, The transformation relationship between the camera coordinate system and the radar coordinate system is as follows: in, , , This represents the coordinates of the observed target in the camera coordinate system. , , R represents the coordinates of the observed target in the radar coordinate system; R and t represent the rotation matrix and translation vector, respectively.

8. The automatic radar and camera classification calibration method based on target trajectory analysis according to claim 1 or 2, characterized in that, In step S2, the four dimensions of the point cloud data include target distance, horizontal azimuth, vertical height, and velocity.

9. A radar and camera hierarchical automatic calibration system based on target trajectory analysis, comprising an interconnected memory and a processor, wherein the memory stores a computer program, characterized in that, The computer program, when run by a processor, performs the steps of the method as described in any one of claims 1-8.

Images