Microwave radar-camera space calibration method and system based on pose joint optimization

Abstract

This invention provides a microwave radar-camera spatial calibration method and system based on pose joint optimization, comprising: Step S1: constructing a calibration scene, in which a radar, a camera, and a calibration reference are arranged; the radar is rigidly connected to the camera; Step S2: defining a radar coordinate system and constructing an optimization problem, solving the optimization problem to obtain the optimal radar pose, and adjusting the radar; Step S3: acquiring images of the calibration reference through the camera, obtaining the pixel coordinates of the calibration reference, solving for the optimal pose of the camera, and adjusting the camera; Step S4: transforming the coordinates of the radar and the camera, outputting the final radar-camera spatial calibration result, and reducing errors. This invention, by introducing pitch angle constraints, creatively solves the ill-posed problem of pose estimation for linear antenna array radars, providing a practical solution for the precise pose calibration of this type of sensor.

Description

Technical Field

[0001] This invention belongs to the field of sensor fusion technology, specifically relating to a microwave radar-camera spatial calibration method and system based on pose joint optimization. More specifically, it is a method for realizing spatial coordinate calibration between a microwave radar and a visual camera. Background Technology

[0002] In fields such as robotics, autonomous driving, and precision measurement, fusion sensing by integrating microwave radar and vision cameras has become an important technological approach. A prerequisite for effective fusion is accurately obtaining the spatial transformation relationship between the microwave radar coordinate system and the vision camera coordinate system; this process is called heterogeneous sensor spatial calibration.

[0003] The core challenge of existing microwave radar-camera spatial calibration methods stems from the fundamental differences in the observation models and information dimensions of the two sensors.

[0004] Microwave radar, especially LFMCW radar employing linear antenna arrays, typically only provides radial range and one-dimensional angle of arrival information for the target. These two observations cannot fully constrain the radar's pose in all six degrees of freedom in three-dimensional space, posing an inherent challenge to direct, high-precision pose estimation. Visual cameras, on the other hand, can more easily obtain their complete six-degree-of-freedom pose in the world coordinate system using reference objects such as calibration boards (e.g., ChArUco boards) and mature perspective projection models and solution methods.

[0005] Current mainstream calibration methods can be divided into step-by-step independent calibration and fully joint optimization. Step-by-step independent calibration estimates the poses of the radar and camera relative to a common world coordinate system independently, and then indirectly calculates their transformation. Due to the insufficient observation dimensions of linear antenna array radars, pose estimation is inherently an ill-posed problem. Without special constraints, the estimation results are inaccurate and highly uncertain, leading to unreliable extrinsic parameter calibration results. The fully joint optimization method simultaneously uses radar extrinsic parameters, camera extrinsic parameters, and even all acquired poses as optimization variables. This method has a complex model, a large solution space, is sensitive to initial values, is prone to getting trapped in local optima, and is also limited by the inherent bottleneck of insufficient observation information in linear antenna array radars.

[0006] Currently, existing microwave radar-camera spatial calibration methods have the following three significant drawbacks.

[0007] First, there is a lack of effective methods to address the limitation in radar pose estimation accuracy caused by insufficient observation dimensions in linear antenna array radars. These radars can only provide target range and one-dimensional angle of arrival information; these two observations cannot constitute complete constraints to solve for the radar's six degrees of freedom pose in three-dimensional space. Therefore, without introducing additional strong constraints, the radar's own pose estimation is an ill-posed problem, with significant uncertainties and errors in the results. This becomes a theoretical bottleneck restricting the calibration accuracy of the entire system.

[0008] Secondly, the fully joint optimization method has problems such as complex optimization models, large variable space, and extreme sensitivity to initial values ​​of iteration. It not only has a large computational load, but is also very easy to get trapped in local optima, resulting in poor robustness and practicality in real-world applications.

[0009] Finally, in a controlled calibration scenario, available engineering prior knowledge was not effectively utilized. For example, the radar's elevation angle could have been artificially adjusted and constrained to near zero degrees.

[0010] Therefore, existing technologies have failed to effectively solve the fundamental challenge of achieving high-precision pose estimation for linear antenna array radars when the information provided is limited. This leads to problems such as limited accuracy, complex procedures, and poor stability in subsequent radar-camera extrinsic parameter calibration.

[0011] This problem urgently needs to be solved. Summary of the Invention

[0012] To address the shortcomings of existing technologies, the purpose of this invention is to provide a microwave radar-camera spatial calibration method and system based on pose joint optimization.

[0013] A microwave radar-camera spatial calibration method based on pose joint optimization, provided by the present invention, includes: Step S1: Set up a calibration scene, in which radar, cameras and calibration reference objects are arranged; Step S2: Define the radar coordinate system and then construct an optimization problem, solve the optimization problem and obtain the optimal radar pose; Step S3: Obtain an image of the calibration reference object through the camera, then obtain the pixel coordinates of the calibration reference object, and then solve the optimal pose of the camera; Step S4: Transform the coordinates of the radar and the camera, and output the final radar-camera spatial calibration result.

[0014] Preferably, in step S1, the calibration reference is arranged in the world coordinate system and includes a microwave calibration reference and a visual calibration reference; the microwave calibration reference includes a corner reflector; the visual calibration reference includes a ChArUco calibration plate.

[0015] Preferably, in step S2, the origin of the radar coordinate system is located at the radar phase center, the X-axis points directly in front of the radar along the main beam direction, and the Y-axis is along the linear array direction. The elevation angle of the radar is constrained, and then the optimal radar pose is obtained by solving an optimization problem. The elevation angle is zero, and the optimal radar pose is obtained by solving an optimization problem, expressed as:

[0016] in, This indicates the optimal radar pose. Indicates the expression When the minimum value is obtained The value of , N This indicates the total number of microwave calibration references. i Indicates the index of each microwave calibration reference. It means "to obey". and The distances and azimuths of each microwave calibration reference point are derived from the observation model. and The distance and azimuth of each microwave calibration reference obtained by radar measurement; Normalization factor; θ Indicates the pitch angle.

[0017] Preferably, in step S3, an image of the visual calibration reference is acquired by the camera, and then the pixel coordinates of the visual calibration reference are obtained. Then, the optimal pose of the camera is solved and the camera is adjusted. The pixel coordinates of the visual calibration reference are: , Let represent the set of two-dimensional real numbers; the expression for the optimal pose of the camera is:

[0018] in, This indicates the optimal pose of the camera. Indicates the expression When the minimum value is obtained The possible values ​​of ; This represents the predicted coordinates of the ChArUco corner points obtained from the camera model reprojection. This indicates a modulo operation.

[0019] Preferably, in step S4, the coordinate transformation between the radar and the camera includes: a rotation matrix and a translation vector from the camera coordinate system to the radar coordinate system, and a rotation matrix and a translation vector from the radar coordinate system to the camera coordinate system. The rotation matrix from the camera coordinate system to the radar coordinate system is expressed as:

[0020] in, This represents the rotation matrix from the camera coordinate system to the radar coordinate system; superscript. T Indicates transpose; This represents the camera's pose matrix in the world coordinate system; This represents the radar's attitude matrix in the world coordinate system; The translation vector from the camera coordinate system to the radar coordinate system is expressed as:

[0021] in, This represents the translation vector from the camera coordinate system to the radar coordinate system. Indicates the radar's position in the world coordinate system; This indicates the radar's position in the camera coordinate system; The rotation matrix from the radar coordinate system to the camera coordinate system is expressed as follows:

[0022] in, This represents the rotation matrix from the radar coordinate system to the camera coordinate system; The translation vector from the radar coordinate system to the camera coordinate system is expressed as:

[0023] in, This represents the translation vector from the radar coordinate system to the camera coordinate system.

[0024] The present invention provides a microwave radar-camera spatial calibration system based on pose joint optimization, which is used to implement the steps of the microwave radar-camera spatial calibration method based on pose joint optimization, including: a heterogeneous joint sensing module, a calibration scene construction module, a spatial relationship calibration module, and a data storage and output module; The spatial relationship calibration module is connected to the heterogeneous joint sensing module, the calibration scene construction module, and the data storage and output module, respectively.

[0025] Preferably, the heterogeneous joint sensing module includes one or more microwave radars and a visual sensor; the microwave radar is rigidly connected to the visual sensor; the visual sensor includes a camera; The calibration scene construction module includes microwave calibration reference objects and visual calibration reference objects arranged in the calibration scene.

[0026] Preferably, the spatial relationship calibration module includes: a radar pose estimation unit, a camera pose estimation unit, and a coordinate transformation relationship calculation unit; The radar pose estimation unit constructs an optimization problem based on the microwave calibration reference scanning results obtained by the microwave radar, solves the optimization problem, and then obtains the optimal radar pose. The camera pose estimation unit obtains the pixel coordinates of the visual calibration reference object based on the image data of the visual calibration reference object acquired by the camera, and then solves the optimal pose of the camera. The coordinate transformation relationship calculation unit performs matrix transformation based on the optimal radar pose and the optimal camera pose.

[0027] Preferably, the radar pose estimation unit defines a radar coordinate system and then constructs an optimization problem, solves the optimization problem, and then obtains the optimal radar pose; the origin of the radar coordinate system is located at the radar phase center, the X-axis points directly in front of the radar along the main beam direction, and the Y-axis is along the linear array direction; The radar's elevation angle is constrained to zero. The optimal radar pose is then obtained by solving an optimization problem, expressed as:

[0028] in, This indicates the optimal radar pose. Indicates the expression When the minimum value is obtained The value of , N This indicates the total number of microwave calibration references. i Indicates the index of each microwave calibration reference. It means "to obey". and The distances and azimuths of each microwave calibration reference point are derived from the observation model. and The distance and azimuth of each microwave calibration reference obtained by radar measurement; Normalization factor; θ Indicates the pitch angle; The camera pose estimation unit acquires images of visual calibration reference objects through the camera, obtains the pixel coordinates of the calibration reference objects, and then solves the optimal pose of the camera to adjust the camera. The pixel coordinates of the visual calibration reference are: , Let represent the set of two-dimensional real numbers. The expression for the optimal pose of the camera is:

[0029] in, This indicates the optimal pose of the camera. Indicates the expression When the minimum value is obtained The possible values ​​of ; This represents the predicted coordinates of the ChArUco corner points obtained from the camera model reprojection. This indicates a modulo operation.

[0030] Preferably, in the coordinate transformation relationship calculation unit, the objects of coordinate transformation include: the rotation matrix and translation vector from the camera coordinate system to the radar coordinate system, and the rotation matrix and translation vector from the radar coordinate system to the camera coordinate system. The rotation matrix from the camera coordinate system to the radar coordinate system is expressed as:

[0031] in, This represents the rotation matrix from the camera coordinate system to the radar coordinate system; superscript. T Indicates transpose; This represents the camera's pose matrix in the world coordinate system; This represents the radar's attitude matrix in the world coordinate system; The translation vector from the camera coordinate system to the radar coordinate system is expressed as:

[0032] in, This represents the translation vector from the camera coordinate system to the radar coordinate system. Indicates the radar's position in the world coordinate system; Indicates the camera's position in the world coordinate system; The rotation matrix from the radar coordinate system to the camera coordinate system is expressed as follows:

[0033] in, This represents the rotation matrix from the radar coordinate system to the camera coordinate system; The translation vector from the radar coordinate system to the camera coordinate system is expressed as:

[0034] in, This represents the translation vector from the radar coordinate system to the camera coordinate system.

[0035] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention addresses the problem that linear antenna array microwave radars can only provide range and one-dimensional angle of arrival, resulting in insufficient constraints for radar pose estimation. By introducing pitch angle constraints into radar pose estimation and constructing a solvable optimization model based on them, the radar pose estimation is transformed from a traditional ill-posed estimation problem into a stable and solvable problem, thereby significantly improving the accuracy and determinism of radar pose estimation and providing a reliable foundation for subsequent radar-camera extrinsic parameter calibration.

[0036] 2. This invention adopts a step-by-step calibration strategy of "first radar pose, then camera pose, and then calculation of transformation relationship". However, unlike the traditional step-by-step method, which has the defects of large radar pose error and easy accumulation and amplification of error in the first step, this invention can obtain high-precision radar pose in the first step, thereby effectively suppressing error propagation and amplification effect, making the final radar-camera spatial calibration result more stable and reliable.

[0037] 3. Compared with the fully joint optimization method, this invention does not need to include a large number of variables such as radar extrinsic parameters, camera extrinsic parameters and multiple sets of acquisition attitudes into the same optimization problem at the same time. Instead, it significantly reduces the number of optimization variables and the solution space by introducing reasonable constraints and solving step by step, reducing the sensitivity to the initial value of the iteration, improving the stability and computational efficiency of the solution, and effectively avoiding getting trapped in local optima in the solution process, thus taking into account both high accuracy and high robustness.

[0038] 4. This invention utilizes a mature reprojection error minimization mechanism on the camera side to obtain the camera's precise pose in the world coordinate system, and on the radar side, it utilizes the geometric constraints brought about by the three-dimensional distribution of multiple reference objects to enhance the solution conditions. Finally, based on the optimal pose of both in the same world coordinate system, the radar-camera rigid body transformation matrix is ​​directly calculated, making the calibration process clear and the engineering implementation simple, with strong feasibility and promotional value. Attached Figure Description

[0039] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the calibration method provided by the present invention; Figure 2 The following is a comparative schematic diagram of the experimental test scenarios for the implementation examples provided by the present invention. (a) is a camera pose calibration scenario, in which there is a 5*7 ChArUco calibration board in the camera's field of view, and its ChArUco corner points serve as visual calibration references; (b) is a radar pose calibration scenario, in which there are 11 corner reflectors distributed in three-dimensional space in the radar's field of view, serving as microwave calibration references, and the absorbing sponge is used to block the reflected energy of irrelevant targets such as the optical breadboard that fixes the ChArUco calibration board. Figure 3The diagram illustrates the comparison between the standard deviation of the error obtained by Monte Carlo simulation and the Cramér-RaoLower Bound (CRLB) for the implementation examples provided by this invention. (a) shows the radar pose comparison, i.e., position parameters, roll angle, and yaw angle; (b) shows the camera pose comparison, i.e., position parameters and rotation vector; (c) shows the radar-camera transformation matrix comparison, i.e., translation vector and rotation vector. The number above each parameter represents the ratio of the standard deviation obtained by Monte Carlo simulation to the theoretically calculated standard deviation of CRLB. Figure 4 The system structure block diagram provided by the present invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0041] This invention addresses the problems in spatial calibration of microwave radar and visual cameras: given that linear antenna array radar can only provide two observations, range and one-dimensional angle of arrival, by introducing constraints on the radar elevation angle, the ill-posed position estimation problem caused by insufficient radar observation information is transformed into a solvable optimization problem; based on this, a microwave radar-camera spatial calibration method based on pose joint optimization is proposed, avoiding the problems of low calibration accuracy and complex models existing in existing methods.

[0042] Specifically, this invention proposes a microwave radar-camera spatial calibration method based on pose joint optimization and the components of its system, as well as a radar pose estimation method based on pitch angle constraints.

[0043] The microwave radar-camera spatial calibration method based on pose joint optimization, such as Figure 1 As shown, it includes: First, a calibration scene is set up, and the camera and radar are rigidly connected. Then, the radar elevation angle is fixed to a known value, and an optimization problem is constructed to estimate the radar pose. Next, an optimization problem is constructed for the camera to estimate its pose. Finally, the poses of the two sensors are combined to solve the coordinate transformation relationship.

[0044] Step 1: Set up the calibration scene. Arrange a set of microwave calibration references with known 3D coordinates in the world coordinate system. In the common field of view of the radar-camera joint unit, arrange a set of visual calibration references and measure the coordinates of each visual calibration reference in the world coordinate system.

[0045] Specifically, the reference objects are distributed in three-dimensional space rather than a two-dimensional plane to ensure the effectiveness of radar pose estimation.

[0046] Specifically, a set of corner reflectors with known three-dimensional coordinates are arranged in the world coordinate system as microwave calibration references. These corner reflectors must be distributed in three-dimensional space.

[0047] Specifically, a ChArUco calibration plate of known size is set up, and the coordinates of its corner points in the world coordinate system are accurately measured as a visual calibration object.

[0048] Its beneficial effects are that, since this step arranges the microwave calibration reference and the visual calibration reference in the same world coordinate system and requires the microwave reference to be distributed in three-dimensional space, it provides more sufficient geometric constraints for radar pose estimation and avoids the solution degradation caused by coplanar or single distribution of references; at the same time, the radar and camera are rigidly connected and participate in the calibration as a whole, so that the relative geometric relationship between the two sensors is stable and reusable, reducing the workload of repeated debugging and recalibration on site, and improving the convenience of engineering implementation and long-term stability.

[0049] Step 2, Radar attitude estimation based on pitch angle constraints.

[0050] Step 2.1, define the radar coordinate system.

[0051] Specifically, the radar coordinate system is defined as follows: the origin is located at the radar phase center, the X-axis points directly in front of the radar along the main beam direction, the Y-axis is along the linear array direction, and the Z-axis is determined by the cross product of the X-axis and the Y-axis.

[0052] Step 2.2: Construct an optimization problem to solve for the optimal radar pose. Since the linear antenna array microwave radar can only provide two observations—range and angle of arrival—for each microwave calibration reference, it cannot completely constrain the three attitude angles. By constraining the radar's pitch angle to a fixed value, the pose parameters are simplified to five parameters: three translational values, roll angle, and yaw angle. The optimal radar pose is solved by minimizing the sum of squared residuals of the predicted range, predicted angle of arrival, and the range and angle of arrival measured by the radar.

[0053] Specifically, constrain the pitch angle θ = 0, simplifying the radar pose parameters to ,in This represents the radar's position in the world coordinate system. Coordinates in the world coordinate system. The roll angle is around the Z-axis. The yaw angle is around the X-axis. Represents the three-dimensional set of real numbers.

[0054] The optimal pose of the radar is obtained by optimizing the following objective function. :

[0055] in, This indicates the optimal radar pose. Indicates the expression When the minimum value is obtained The value of , N This indicates the total number of microwave calibration references. i Indicates the index of each microwave calibration reference. It means "to obey". This represents the radar's attitude matrix in the world coordinate system. and The distances and azimuths of each microwave calibration reference object are calculated by the observation model based on the predicted pose. and This refers to the distance and azimuth of each microwave calibration reference obtained from radar measurements. This is the normalization factor.

[0056] Its beneficial effect is that, since linear antenna array radar can only provide range and one-dimensional angle of arrival, directly estimating the six-degree-of-freedom pose is prone to ill-posed or unstable problems. This step constrains the radar elevation angle to a known fixed value, such as zero degrees, and reduces the dimensionality of the optimization variables accordingly, so that radar pose estimation is transformed from an unreliable solution with insufficient information into a stable optimization problem, thereby significantly improving the accuracy and determinism of radar pose estimation, and suppressing the risk of error accumulation and amplification in step-by-step calibration from the source.

[0057] Step 3: Precise camera pose calibration. After the camera captures an image, the pixel coordinates of the visual calibration reference are detected using image processing algorithms. By minimizing the reprojection error, the optimal camera pose in the world coordinate system is determined.

[0058] Specifically, camera coordinate system The origin is defined at the camera's optical center, with the X-axis pointing to the right, the Y-axis pointing downwards, and the Z-axis pointing directly in front of the camera. ChArUco corner points are detected using image processing algorithms to obtain their pixel coordinates. Let represent the set of two-dimensional real numbers. The optimal pose of the camera in the world coordinate system is solved by minimizing the reprojection error. :

[0059] in, The pixel coordinates of the visual calibration reference object detected by the image processing algorithm. This represents the predicted coordinates of the ChArUco corner points obtained from the reprojection of the camera model. This indicates a modulo operation. This indicates the camera's position in the world coordinate system.

[0060] Its beneficial effect is that, since this step solves the optimal pose of the camera in the world coordinate system by detecting the pixel coordinates of the visual calibration reference and minimizing the reprojection error, it can make full use of the mature and stable visual pose calculation mechanism to obtain high-precision camera pose results; and this camera pose will serve as the key input for subsequent coordinate transformation relationship calculation, together with the aforementioned high-precision radar pose, to form a stable common geometric benchmark, thereby improving the accuracy and consistency of the final extrinsic parameter calibration.

[0061] Step 4: Calculate the radar-camera coordinate transformation matrix. After obtaining the optimal poses of the radar and camera in the same world coordinate system, the coordinate transformation relationship between them is directly calculated, taking advantage of the rigid connection between them.

[0062] Specifically, from the camera coordinate system To radar coordinate system rotation matrix , represented as:

[0063] Among them, the upper right corner T Indicates transpose; This represents the camera's attitude matrix in the world coordinate system.

[0064] From the camera coordinate system To radar coordinate system The translation vector is expressed as:

[0065] in, Indicates from the camera coordinate system To radar coordinate system The translation vector.

[0066] From radar coordinate system To the camera coordinate system rotation matrix Represented as:

[0067] From radar coordinate system To the camera coordinate system The translation vector is expressed as:

[0068] in, Indicates from radar coordinate system To the camera coordinate system Translation vector Its beneficial effects are that, since this step directly calculates the rigid body transformation relationship between the radar and camera through matrix transformation after the radar and camera have obtained their optimal poses in the same world coordinate system, it avoids the problems of many variables, complex models, sensitivity to initial values, and easy trapping in local optima in the fully joint optimization method, making the calibration process clearer, the computational cost more controllable, and the results more stable. At the same time, the extrinsic parameter matrix obtained based on the premise of rigid connection can be directly used for subsequent radar-camera fusion applications, which has good engineering usability.

[0069] Based on the microwave radar-camera spatial calibration method based on pose joint optimization provided by this invention, Figure 2 and Figure 3 An example of experimental test results is shown. Figure 3 For a calibration scenario applying the method proposed in this invention, 11 corner reflectors were used as microwave calibration references. Eight of these were distributed in the XOY plane of the world coordinate system, forming dense position sampling points. The other three corner reflectors were distributed in a stepped pattern along the Z-axis, effectively breaking the planar symmetry and providing strong constraints for estimating roll and yaw angles. A 5×7 grid ChArUco calibration board was used as a visual calibration reference, and this calibration board was fixed to a breadboard perpendicular to the optical platform. Based on this calibration scenario setup, 10,000 independent Monte Carlo simulations were performed.

[0070] Figure 3 The results show that the empirical standard deviations of the radar and camera poses obtained from 10,000 trials are in excellent agreement with their respective Cramér-Rao Lower Bounds (CRLBs). This not only confirms the asymptotic validity of the estimators but also demonstrates the feasibility of the reflector and ChArUco marker placement scheme. Subsequently, to verify whether the estimation accuracy of the proposed coordinate transformation matrix method approaches the theoretical limit, this invention further performed 10,000 independent Monte Carlo simulations on the radar-camera coordinate transformation matrix estimation and compared it with the CRLB based on the Fisher information matrix. Figure 3 (c) presents the standard deviation results for position and attitude, i.e., the parameterization of the rotation vector. It can be seen that the standard deviation of the position error is basically consistent with CRLB in all three dimensions, with a difference of less than 5%, indicating that the translation estimation has achieved statistical effectiveness. The standard deviation of the attitude error is slightly larger than CRLB by about 2% to 30%, but still maintains the same magnitude and trend, indicating that the proposed estimation algorithm is also close to the theoretical lower bound in the attitude dimension.

[0071] Microwave radar-camera spatial calibration system based on pose joint optimization, such as Figure 4 As shown, it includes: The heterogeneous joint sensing module consists of a microwave radar with at least one linear antenna array rigidly connected to at least one visual sensor. This module participates in calibration as a whole.

[0072] The calibration scene construction module includes microwave calibration references and visual calibration references arranged in the common field of view of the radar and camera.

[0073] The spatial relationship calibration module is configured to perform the step-by-step pose estimation and coordinate transformation calculation, including a radar pose estimation unit, a camera pose estimation unit, and a coordinate transformation relationship calculation unit.

[0074] The radar pose estimation unit is used to receive distance and angle of arrival observation data from the microwave calibration reference, and has built-in pitch angle constraints to construct and solve an optimization problem that only concerns the radar position, roll angle and yaw angle, thereby outputting the optimal pose of the radar in the world coordinate system.

[0075] The camera pose estimation unit is used to receive image data from the visual calibration reference, and solve the optimal pose of the camera in the world coordinate system by detecting the pixel coordinates of the visual calibration reference and minimizing the reprojection error.

[0076] The coordinate transformation relationship calculation unit is used to receive the output results from the radar pose estimation unit and the camera pose estimation unit, and calculate the rigid body transformation matrix from the camera coordinate system to the radar coordinate system, or calculate the rigid body transformation matrix from the radar coordinate system to the camera coordinate system, based on the premise that the radar and the camera are rigidly connected.

[0077] The data storage and output module is used to store the world coordinates, intermediate pose results, and final coordinate transformation matrix of the calibration reference object, and output the final radar-camera spatial calibration results.

[0078] Specifically, according to the present invention, a microwave radar-camera spatial calibration system based on pose joint optimization is provided to implement the steps of the microwave radar-camera spatial calibration method based on pose joint optimization, including: a heterogeneous joint sensing module, a calibration scene construction module, and a spatial relationship calibration module; The spatial relationship calibration module is connected to the heterogeneous joint perception module and the calibration scene construction module, respectively.

[0079] Specifically, the heterogeneous joint sensing module includes one or more microwave radars and a visual sensor; the microwave radar is rigidly connected to the visual sensor; the visual sensor includes a camera; The calibration scene construction module includes microwave calibration reference objects and visual calibration reference objects arranged in the calibration scene.

[0080] Specifically, the spatial relationship calibration module includes: a radar pose estimation unit, a camera pose estimation unit, and a coordinate transformation relationship calculation unit; The radar pose estimation unit constructs an optimization problem based on the microwave calibration reference scanning results obtained by the microwave radar, solves the optimization problem, and then obtains the optimal radar pose. The camera pose estimation unit obtains the pixel coordinates of the visual calibration reference object based on the image data of the visual calibration reference object acquired by the camera, and then solves the optimal pose of the camera. The coordinate transformation relationship calculation unit performs matrix transformation based on the optimal radar pose and the optimal camera pose.

[0081] The beneficial effects of the system provided by this invention are as follows: The system divides the calibration process into modules such as heterogeneous joint sensing, calibration scene construction, spatial relationship calibration, and data storage and output. Furthermore, the spatial relationship calibration module is further divided into a radar pose estimation unit, a camera pose estimation unit, and a coordinate transformation relationship calculation unit, making the input and output of each functional unit clear and the implementation path well-defined. Specifically, the radar pose estimation unit incorporates pitch angle constraints and performs dimensionality reduction optimization, improving the stability and accuracy of radar pose estimation. The coordinate transformation relationship calculation unit directly calculates the extrinsic parameter matrix based on the optimal poses on both sides, reducing the computational complexity and initial value sensitivity of joint optimization. The data storage and output module provides unified management of intermediate results and final extrinsic parameters, facilitating calibration reproduction, error traceability, and engineering integration, thereby comprehensively improving the system's implementability, robustness, and application promotion value.

[0082] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0083] In the description of this application, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0084] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A microwave radar-camera spatial calibration method based on pose joint optimization, characterized in that, include: Step S1: Set up a calibration scene, in which radar, cameras and calibration reference objects are arranged; Step S2: Define the radar coordinate system and then construct an optimization problem, solve the optimization problem and then obtain the optimal radar pose; Step S3: Obtain an image of the calibration reference object through the camera, then obtain the pixel coordinates of the calibration reference object, and then solve the optimal pose of the camera; Step S4: Transform the coordinates of the radar and the camera, and output the final radar-camera spatial calibration result.

2. The microwave radar-camera spatial calibration method based on pose joint optimization according to claim 1, characterized in that, In step S1, the calibration reference is arranged in the world coordinate system and includes a microwave calibration reference and a visual calibration reference; the microwave calibration reference includes a corner reflector; the visual calibration reference includes a ChArUco calibration plate.

3. The microwave radar-camera spatial calibration method based on pose joint optimization according to claim 1, characterized in that, In step S2, the origin of the radar coordinate system is located at the radar phase center, the X-axis points directly in front of the radar along the main beam direction, and the Y-axis is along the linear array direction. The elevation angle of the radar is constrained, and then the optimal radar pose is obtained by solving an optimization problem.

4. The microwave radar-camera spatial calibration method based on pose joint optimization according to claim 2, characterized in that, In step S3, an image of the visual calibration reference is acquired by the camera, and then the pixel coordinates of the visual calibration reference are obtained, thereby solving the optimal pose of the camera.

5. The microwave radar-camera spatial calibration method based on pose joint optimization according to claim 4, characterized in that, In step S4, the coordinate transformation between the radar and the camera includes: a rotation matrix and a translation vector from the camera coordinate system to the radar coordinate system, and a rotation matrix and a translation vector from the radar coordinate system to the camera coordinate system.

6. A microwave radar-camera spatial calibration system based on pose joint optimization, used to implement the steps of the microwave radar-camera spatial calibration method based on pose joint optimization as described in any one of claims 1 to 5, characterized in that, include: Heterogeneous joint sensing module, calibration scene construction module, spatial relationship calibration module, data storage and output module; The spatial relationship calibration module is connected to the heterogeneous joint sensing module, the calibration scene construction module, and the data storage and output module, respectively.

7. The microwave radar-camera spatial calibration system based on pose joint optimization according to claim 6, characterized in that, The heterogeneous joint sensing module includes one or more microwave radars and a vision sensor; the microwave radar is rigidly connected to the vision sensor; the vision sensor includes a camera. The calibration scene construction module includes microwave calibration reference objects and visual calibration reference objects arranged in the calibration scene.

8. The microwave radar-camera spatial calibration system based on pose joint optimization according to claim 7, characterized in that, The spatial relationship calibration module includes: a radar pose estimation unit, a camera pose estimation unit, and a coordinate transformation relationship calculation unit; The radar pose estimation unit constructs an optimization problem based on the microwave calibration reference scanning results obtained by the microwave radar, solves the optimization problem, and then obtains the optimal radar pose. The camera pose estimation unit obtains the pixel coordinates of the calibration reference object based on the image data of the visual calibration reference object acquired by the camera, and then solves the optimal pose of the camera. The coordinate transformation relationship calculation unit performs matrix transformation based on the optimal radar pose and the optimal camera pose.

9. The microwave radar-camera spatial calibration system based on pose joint optimization according to claim 8, characterized in that, The radar pose estimation unit defines a radar coordinate system and then constructs an optimization problem. Solving the optimization problem yields the optimal radar pose. The origin of the radar coordinate system is located at the radar phase center, the X-axis points directly in front of the radar along the main beam direction, and the Y-axis is along the linear array direction. The camera pose estimation unit acquires images of visual calibration reference objects through the camera, obtains the pixel coordinates of the calibration reference objects, and then solves the optimal pose of the camera to adjust the camera.

10. The microwave radar-camera spatial calibration system based on pose joint optimization according to claim 9, characterized in that, In the coordinate transformation relationship calculation unit, the objects of coordinate transformation include: the rotation matrix and translation vector from the camera coordinate system to the radar coordinate system, and the rotation matrix and translation vector from the radar coordinate system to the camera coordinate system.

Images