Trajectory alignment based camera and radar extrinsic calibration method, system and medium

Abstract

The application discloses a camera and radar extrinsic calibration method and system based on trajectory alignment and a medium. The method comprises the following steps: obtaining a preset reflector center in a common detection area of a camera and a millimeter wave radar, and the camera and the millimeter wave radar correspondingly detect a pixel center trajectory and a radar center trajectory along a preset trajectory; calculating the pixel center trajectory and the radar center trajectory based on a polynomial fitting algorithm, and correspondingly obtaining a pixel center fitting trajectory and a radar center fitting trajectory; performing trajectory alignment on the pixel center fitting trajectory and the radar center fitting trajectory based on a batch optimization algorithm, and obtaining a target extrinsic matrix in an extrinsic conversion relationship between the camera and the millimeter wave radar; therefore, the calibration process is simple and efficient, and the calibration error caused by the need to find multiple sets of three-dimensional coordinates of matching points of two sensors and the need for manual measurement of the actual distance from the camera to the target is overcome.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor fusion technology, and in particular to a method, system, and medium for extrinsic parameter calibration of cameras and radar based on trajectory alignment. Background Technology

[0002] With the development of driver assistance systems and autonomous driving systems, the perception of the surrounding environment has shifted from single-sensor to multi-sensor fusion. Cameras and millimeter-wave radar are two commonly used sensors. Cameras can return the outline and color information of a target, but they are not good at obtaining the target's position information and are affected by weather. Millimeter-wave radar can return the accurate position and speed information of a target and has all-weather perception capabilities. Therefore, the effective use of these two sensors can further improve the perception capability of the entire system. Since cameras and millimeter-wave radar are installed in different locations on the vehicle, there are spatial differences between the reference coordinate systems, so spatial calibration is required.

[0003] Current common methods: The commonly used camera and millimeter-wave radar extrinsic parameter calibration method requires finding multiple sets of three-dimensional coordinates of matching points between two sensors in order to solve the spatial transformation matrix. However, the camera cannot directly obtain the actual distance to the target. This step usually requires manual measurement, which inevitably causes inconvenience and errors. Summary of the Invention

[0004] The present invention provides a method, system and medium for extrinsic parameter calibration of cameras and radar based on trajectory alignment. The calibration process is simple and efficient, overcoming the calibration errors caused by the need to find multiple sets of three-dimensional coordinates of matching points between two sensors and the need to manually measure the actual distance from the camera to the target.

[0005] Firstly, a method for extrinsic parameter calibration of cameras and radar based on trajectory alignment is provided, specifically including the following steps:

[0006] The center of a preset reflector in the common detection area of ​​the camera and millimeter-wave radar moves along a preset trajectory, and the center trajectories of the pixels detected by the camera and millimeter-wave radar are obtained respectively.

[0007] The pixel center trajectory and the radar center trajectory are calculated based on the polynomial fitting algorithm, and the pixel center fitting trajectory and the radar center fitting trajectory are obtained accordingly.

[0008] Based on the batch optimization algorithm, the fitted trajectory of the pixel center is aligned with the fitted trajectory of the radar center to obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar.

[0009] According to the first aspect, in a first possible implementation of the first aspect, the step of "calculating the pixel center trajectory and the radar center trajectory based on a polynomial fitting algorithm to obtain the pixel center fitting trajectory and the radar center fitting trajectory" specifically includes the following steps:

[0010] Let the fitting polynomial of the radar center fitted trajectory S1 be:

[0011] s1:y wr =b0+b1x wr +b2x wr 2 +b3x wr 3 +b4x wr 4 Formula (1);

[0012] Let the fitting polynomial of the pixel center fitting trajectory S2 be:

[0013] s2:u=a0+a1v+a2v 2 +a3v 3 +a4v 4 Formula (II);

[0014] Multiple sets of paired points are selected from the preset reflector center in the pixel center trajectory and the radar center trajectory:

[0015]

[0016] By performing polynomial fitting on multiple sets of paired points for equations (I) and (II), the coefficients a0, a1, a2, a3, a4, b0, b1, b2, b3, and b4 are obtained.

[0017] In the formula, a0, a1...a4 are the coefficients of the fitting polynomial u; b0, b1...b4 are the coefficients of the fitting polynomial y. wr The coefficient.

[0018] According to the first possible implementation of the first aspect, in the second possible implementation of the first aspect, the step of "aligning the pixel center fitting trajectory with the radar center fitting trajectory based on the batch optimization algorithm to obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar" specifically includes the following steps:

[0019] Construct the objective function for the transformation relationship of extrinsic parameters;

[0020] The objective function is iteratively calculated based on the batch optimization algorithm, and the loss gradient value is calculated based on the loss function. The learning rate and the loss gradient value are combined to determine the moving distance for each iteration, so as to iteratively update the external parameter conversion relationship between the camera and the millimeter-wave radar.

[0021] When the comparison result between the target function value calculated in the iterative calculation and the preset threshold is found to meet the conditions, the target extrinsic parameter matrix is ​​obtained based on the extrinsic parameter transformation relationship of the last iteration calculation.

[0022] According to the second possible implementation of the first aspect, in the third possible implementation of the first aspect, the formula of the objective function of the extrinsic parameter transformation relationship is as follows:

[0023]

[0024] in,

[0025] In the formula, These represent the trajectory points on the radar center fitting trajectory and the pixel center fitting trajectory, respectively; N is the number of sampled trajectory points i; A is the camera's intrinsic parameter matrix; R is the camera's rotation matrix; and T is the millimeter-wave radar's translation matrix.

[0026] According to the third possible implementation of the first aspect, in the fourth possible implementation of the first aspect, the formula for the loss function is as follows:

[0027] J′(R,T)=(Y i -ARX i -AT)*X i Formula (5).

[0028] According to the third possible implementation of the first aspect, in the fifth possible implementation of the first aspect, the step of "combining the learning rate and the loss gradient value to determine the moving distance calculated in each iteration, so as to iteratively update the extrinsic parameter conversion relationship between the camera and the millimeter-wave radar" specifically includes the following steps:

[0029] Based on the learning rate α, the loss gradient value (Y) i -ARX i -AT)*X i ;

[0030] The distance to be moved in each iteration is determined as: α(Y) i -ARX i -AT)*X i ;

[0031] The formula for iteratively updating the extrinsic parameter conversion relationship between the camera and the millimeter-wave radar is as follows:

[0032] (R,T)′=(R,T)-α(Y) i -ARX i -AT)*X i Formula (VI);

[0033] Where (R, T)′ represents the external parameter transformation relationship after iteration, and (R, T) represents the external parameter transformation relationship before iteration.

[0034] According to the first aspect, in the sixth possible implementation of the first aspect, before the step of "calculating the pixel center trajectory and the radar center trajectory based on the polynomial fitting algorithm to obtain the corresponding camera center fitting trajectory and radar center fitting trajectory" specifically includes the following steps:

[0035] The pixel center trajectory is synchronized with the radar center trajectory using a message filter, and the camera center trajectory is corrected based on camera distortion parameters.

[0036] Secondly, a camera and radar extrinsic parameter calibration system based on trajectory alignment is also provided, including:

[0037] The initial trajectory acquisition module is used to acquire the preset trajectory of the center of the reflector running along the preset trajectory in the common detection area of ​​the camera and the millimeter-wave radar, as well as the pixel center trajectory and radar center trajectory detected by the camera and the millimeter-wave radar respectively.

[0038] A trajectory fitting module, communicatively connected to the initial trajectory acquisition module, is used to calculate the pixel center trajectory and the radar center trajectory based on a polynomial fitting algorithm, thereby obtaining the pixel center fitted trajectory and the radar center fitted trajectory; and...

[0039] The optimization module, which is communicatively connected to the fitting trajectory module, is used to align the pixel center fitting trajectory with the radar center fitting trajectory based on the batch optimization algorithm, and obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar.

[0040] In some embodiments, a trajectory processing module communicatively connected to the fitted trajectory module is also included, which is used to synchronize the pixel center trajectory with the radar center trajectory in time based on a message filter, and to correct the camera center trajectory based on camera distortion parameters.

[0041] Thirdly, a storage medium is also provided, on which a computer program is stored, characterized in that, when the computer program is executed by a processor, it implements the camera and radar extrinsic parameter calibration method based on trajectory alignment as described above.

[0042] Compared with existing technologies, the advantages of this invention are as follows: Existing methods for joint extrinsic parameter calibration of cameras and millimeter-wave radar require finding multiple sets of three-dimensional coordinates of matching points between the two sensors and manually measuring the actual distance from the camera to the target, resulting in errors and a relatively complex calibration process. Therefore, this invention proposes a trajectory alignment-based extrinsic parameter calibration scheme for cameras and millimeter-wave radar. A reflector is placed in the common detection area of ​​the camera and millimeter-wave radar, and the reflector moves along a certain trajectory. Data is recorded, and the trajectories of the reflector center in the camera image and the millimeter-wave radar are calculated separately. The two trajectories are aligned to obtain the target extrinsic parameter matrix. Therefore, the calibration process is simple and efficient, overcoming the calibration errors caused by finding multiple sets of three-dimensional coordinates of matching points between the two sensors and manually measuring the actual distance from the camera to the target. Furthermore, the extrinsic parameter matrix obtained through calibration can be used for transformation between actual coordinate systems, providing a foundation for deep fusion of multi-sensor data. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating an embodiment of a camera and radar extrinsic parameter calibration method based on trajectory alignment according to the present invention.

[0044] Figure 2 This is a flowchart illustrating another embodiment of the camera and radar extrinsic parameter calibration method based on trajectory alignment according to the present invention.

[0045] Figure 3 This is a schematic diagram of the external parameter calibration system for cameras and radar based on trajectory alignment according to the present invention. Detailed Implementation

[0046] Referring now to specific embodiments of the invention, examples of which are illustrated in the accompanying drawings. Although the invention will be described in conjunction with specific embodiments, it will be understood that it is not intended to limit the invention to the described embodiments. Rather, it is intended to cover variations, modifications, and equivalents included within the spirit and scope of the invention as defined by the appended claims. It should be noted that the method steps described herein can be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0047] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] Note: The examples described below are merely specific examples and are not intended to limit the embodiments of the present invention to the specific steps, values, conditions, data, order, etc. Those skilled in the art can utilize the concept of the present invention to construct more embodiments not mentioned herein by reading this specification.

[0049] See Figure 1 As shown, this embodiment of the invention provides a method for extrinsic parameter calibration of a camera and radar based on trajectory alignment, specifically including the following steps:

[0050] S100: Obtain the center of the preset reflector in the common detection area of ​​the camera and the millimeter-wave radar along the preset trajectory, and the center trajectories of the pixels detected by the camera and the millimeter-wave radar respectively.

[0051] S200, calculate the pixel center trajectory and the radar center trajectory based on the polynomial fitting algorithm, and obtain the pixel center fitting trajectory and the radar center fitting trajectory accordingly.

[0052] S300, based on the batch optimization algorithm, aligns the fitted trajectory of the pixel center with the fitted trajectory of the radar center to obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar.

[0053] Specifically, in this embodiment, existing methods for joint extrinsic parameter calibration of cameras and millimeter-wave radar require finding multiple sets of three-dimensional coordinates of matching points identified by the two sensors and manually measuring the actual distance from the camera to the target, resulting in errors and a relatively complex calibration process. Therefore, this invention proposes a trajectory alignment-based extrinsic parameter calibration method for cameras and millimeter-wave radar. A reflector is placed in the common detection area of ​​the camera and millimeter-wave radar, and the reflector moves along a certain trajectory (such as a figure-eight or circle). Data is recorded, and the trajectories of the reflector center in the camera image and the millimeter-wave radar are calculated separately. The two trajectories are aligned to obtain the target extrinsic parameter matrix. Therefore, the calibration process is simple and efficient, overcoming the calibration errors caused by finding multiple sets of three-dimensional coordinates of matching points identified by the two sensors and manually measuring the actual distance from the camera to the target. Furthermore, the extrinsic parameter matrix obtained through calibration can be used for transformation between actual coordinate systems, providing a foundation for deep fusion of multi-sensor data.

[0054] It should be noted that the experimental setup and reflector should be arranged to ensure sufficient light, and the millimeter-wave radar should be adjusted so that the normal vector of the radar detection surface is parallel to the longitudinal plane of the reflector.

[0055] Preferably, in another embodiment of this application, the step "S200, calculating the pixel center trajectory and the radar center trajectory based on a polynomial fitting algorithm to obtain the corresponding pixel center fitting trajectory and radar center fitting trajectory" specifically includes the following steps:

[0056] Let the fitting polynomial of the radar center fitted trajectory S1 be:

[0057] s1:y wr =b0+b1xwr +b2x wr 2 +b3x wr 3 +b4x wr 4 Formula (1);

[0058] Let the fitting polynomial of the pixel center fitting trajectory S2 be:

[0059] s2:u=a0+a1v+a2v 2 +a3v 3 +a4v 4 Formula (II);

[0060] Multiple sets of paired points are selected from the preset reflector center in the pixel center trajectory and the radar center trajectory:

[0061]

[0062] By performing polynomial fitting on multiple sets of paired points for equations (I) and (II), the coefficients a0, a1, a2, a3, a4, b0, b1, b2, b3, and b4 are obtained.

[0063] In the formula, a0, a1...a4 are the coefficients of the fitting polynomial u; b0, b1...b4 are the coefficients of the fitting polynomial y. wr The coefficient.

[0064] Preferably, in another embodiment of this application, the step "S300, aligning the pixel center fitting trajectory with the radar center fitting trajectory based on the batch optimization algorithm to obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar" specifically includes the following steps:

[0065] S310, construct the objective function for the transformation relationship of extrinsic parameters;

[0066] S320 uses a batch optimization algorithm to iteratively calculate the objective function, calculates the loss gradient based on the loss function, and combines the learning rate and the loss gradient to determine the moving distance for each iteration, so as to iteratively update the external parameter conversion relationship between the camera and the millimeter-wave radar.

[0067] S330, when the comparison result between the target function value calculated in the iterative calculation and the preset threshold is found to meet the conditions, the target extrinsic parameter matrix is ​​obtained based on the iterative formula of the extrinsic parameter transformation relationship calculated in the last iteration.

[0068] Specifically, in this embodiment, for the extrinsic parameter matrix, multiple pairs of points are generally selected between the center of the pre-set reflector and the center trajectory of the pixel and the radar. Then, the target extrinsic parameter matrix R,T between the camera and the millimeter-wave radar is solved by the batch optimization algorithm. The batch optimization algorithm solves for the gradient direction by calculating all sampling points. The optimization algorithm is a method used to find the optimal parameters R,T of the model, and the most commonly used one is the gradient descent method.

[0069] Preferably, in another embodiment of this application, the objective function of the extrinsic parameter transformation relationship is formulated as follows:

[0070]

[0071] in,

[0072] In the formula, These represent the trajectory points on the radar center fitting trajectory and the pixel center fitting trajectory, respectively; N is the number of sampled trajectory points i; A is the camera's intrinsic parameter matrix; R is the camera's rotation matrix; and T is the millimeter-wave radar's translation matrix.

[0073] Specifically, in this embodiment, the traditional Zhang Zhengyou labeling method in OpenCV is used. The Zhang Zhengyou labeling method is a calibration method based on two-dimensional planar targets. Multiple images of planar targets, such as chessboard images, are taken by a camera at different angles. Then, a total of 45 standard chessboard images at different positions, angles, and postures are collected to obtain the camera's intrinsic parameter matrix A. The intrinsic parameter matrix is ​​determined by the parameters inside the camera.

[0074] The camera's intrinsic parameter matrix A is as follows:

[0075]

[0076] In the formula, f is the distance; dx and dy are the physical lengths of a pixel on the camera's image plane in the x and y directions, respectively, on the camera's image sensor plate; u0 and v0 are the coordinates of the center of the camera's image sensor plate in the pixel coordinate system.

[0077] Preferably, in another embodiment of this application, the formula for the loss function is as follows:

[0078] J′(R,T)=(Y i -ARX i -AT)*X i Formula (5).

[0079] Preferably, in another embodiment of this application, the step of "combining the learning rate and loss gradient value to determine the moving distance calculated in each iteration, so as to iteratively update the extrinsic parameter conversion relationship between the camera and the millimeter-wave radar" specifically includes the following steps:

[0080] Based on the learning rate α, the loss gradient value (Y) i -ARX i -AT)*X i ;

[0081] The distance to be moved in each iteration is determined as: α(Y) i -ARX i -AT)*X i ;

[0082] The formula for iteratively updating the extrinsic parameter conversion relationship between the camera and the millimeter-wave radar is as follows:

[0083] (R,T)′=(R,T)-α(Y) i -ARX i -AT)*X i Formula (VI);

[0084] Where (R,T)′ represents the external parameter transformation relationship after iteration, and (R,T) represents the external parameter transformation relationship before iteration.

[0085] Preferably, in another embodiment of this application, before the step "S200, calculating the pixel center trajectory and the radar center trajectory based on a polynomial fitting algorithm to obtain the corresponding camera center fitting trajectory and radar center fitting trajectory", the following steps are specifically included:

[0086] The pixel center trajectory is synchronized with the radar center trajectory using a message filter, and the camera center trajectory is corrected based on camera distortion parameters.

[0087] Specifically, in this embodiment, the time synchronization process is as follows:

[0088] 1) Connect all sensors and the host computer;

[0089] 2) Topics for publishing camera and millimeter-wave radar data based on the ROS system;

[0090] 3) ROS subscribes to the topics mentioned above and calls the `message_filters` function for time synchronization. `message_filters` acts like a message buffer; when a message arrives at the filter, it is not output immediately but rather at a later time when certain conditions are met. It receives camera and millimeter-wave radar data and only outputs them when they are received from each source with the same timestamp, thus achieving a synchronized message output effect.

[0091] The process of correcting the camera center trajectory based on camera distortion parameters is as follows:

[0092] The camera distortion parameters are: B = [k1 k2 k3 p1 p2];

[0093] Where k1, k2, and k3 are radial distortion parameters, and p1 and p2 are tangential distortion parameters.

[0094] The pixels at the camera's center trajectory, after distortion, have the following relationship with their ideal positions:

[0095]

[0096] Where (u dr ,v dr Let (u,v) be the distorted position of the pixel, (u,v) be the ideal position of the pixel, and r be the distance from (u,v) to the image center (0,0). In reality, the distorted pixel positions of the camera center trajectory and the camera distortion parameters are known. The ideal position of the pixel can be calculated by reverse calculation using the above formula. Therefore, the camera distortion parameters are used to correct the camera center trajectory captured by the camera, resulting in an image with relatively small distortion.

[0097] See also Figure 2 As shown in the figure, the extrinsic parameter calibration method for cameras and radar based on trajectory alignment provided by this invention specifically includes the following steps:

[0098] 1. Set up the experimental setup and reflector;

[0099] 2. Calibrate the camera's intrinsic parameters using the traditional Zhang Zhengyou marking method; have the camera acquire standard checkerboard images from different positions, angles, and postures to obtain the camera's intrinsic parameters; use the distortion coefficients in the intrinsic parameters to correct the images captured by the camera to obtain images with relatively small distortion.

[0100] 3. The moving reflector moves along a certain trajectory (such as a figure-eight or a circle) to synchronously process and collect data from the camera and millimeter-wave radar;

[0101] 4. The motion trajectory of the reflector center in the image pixel coordinate system and the millimeter-wave radar coordinate system is obtained by polynomial fitting. The two fitted trajectories are aligned by batch optimization algorithm to obtain the external parameter transformation relationship between the two sensors.

[0102] See also Figure 3 As shown, this embodiment of the invention also provides a camera and radar extrinsic parameter calibration system based on trajectory alignment, including:

[0103] The initial trajectory acquisition module is used to acquire the preset trajectory of the center of the reflector running along the preset trajectory in the common detection area of ​​the camera and the millimeter-wave radar, as well as the pixel center trajectory and radar center trajectory detected by the camera and the millimeter-wave radar respectively.

[0104] A trajectory fitting module, communicatively connected to the initial trajectory acquisition module, is used to calculate the pixel center trajectory and the radar center trajectory based on a polynomial fitting algorithm, thereby obtaining the pixel center fitted trajectory and the radar center fitted trajectory; and...

[0105] The optimization module, which is communicatively connected to the fitting trajectory module, is used to align the pixel center fitting trajectory with the radar center fitting trajectory based on the batch optimization algorithm, and obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar.

[0106] It also includes a trajectory processing module that is communicatively connected to the fitted trajectory module, used to synchronize the pixel center trajectory with the radar center trajectory in time based on a message filter, and to correct the camera center trajectory based on camera distortion parameters.

[0107] Existing methods for joint extrinsic parameter calibration of cameras and millimeter-wave radars require finding multiple sets of 3D coordinates of matching points between the two sensors and manually measuring the actual distance from the camera to the target, resulting in errors and a relatively complex calibration process. Therefore, this invention proposes a trajectory alignment-based extrinsic parameter calibration system for cameras and millimeter-wave radars. A reflector is placed in the common detection area of ​​the camera and millimeter-wave radar, and the reflector moves along a certain trajectory. Data is recorded, and the trajectories of the reflector's center in the camera image and the millimeter-wave radar are calculated separately. The two trajectories are aligned to obtain the target extrinsic parameter matrix. Therefore, the calibration process is simple and efficient, overcoming the calibration errors caused by finding multiple sets of 3D coordinates of matching points between the two sensors and manually measuring the actual distance from the camera to the target. Furthermore, the calibrated extrinsic parameter matrix can be used for transformation between actual coordinate systems, providing a foundation for deep fusion of multi-sensor data.

[0108] Specifically, this embodiment corresponds one-to-one with the above method embodiments. The functions of each module have been described in detail in the corresponding method embodiments, so they will not be repeated here.

[0109] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements all or part of the method steps of the above method.

[0110] The present invention can implement all or part of the processes in the above methods, or it can be accomplished by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0111] Based on the same inventive concept, embodiments of this application also provide an electronic device, including a memory and a processor. The memory stores a computer program that runs on the processor. When the processor executes the computer program, it implements all or part of the method steps described above.

[0112] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting all parts of the computer device through various interfaces and lines.

[0113] Memory can be used to store computer programs and / or modules. The processor performs various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system and at least one application program required for a function (e.g., sound playback, image playback, etc.); the data storage area can store data created based on the use of the mobile phone (e.g., audio data, video data, etc.). Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, SmartMedia Cards (SMC), Secure Digital (SD) cards, Flash Cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0114] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, servers, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0115] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), servers, and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for extrinsic parameter calibration of cameras and radar based on trajectory alignment, characterized in that, Specifically, the following steps are included: The center of a preset reflector in the common detection area of ​​the camera and millimeter-wave radar moves along a preset trajectory, and the center trajectories of the pixels detected by the camera and millimeter-wave radar are obtained respectively. The pixel center trajectory and the radar center trajectory are calculated based on the polynomial fitting algorithm, and the pixel center fitting trajectory and the radar center fitting trajectory are obtained accordingly. Based on the batch optimization algorithm, the fitted trajectory of the pixel center is aligned with the fitted trajectory of the radar center to obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar. The step of "aligning the pixel center fitting trajectory with the radar center fitting trajectory based on the batch optimization algorithm to obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar" specifically includes the following steps: Construct the objective function for the transformation relationship of extrinsic parameters; The objective function is iteratively calculated based on the batch optimization algorithm, and the loss gradient value is calculated based on the loss function. The learning rate and the loss gradient value are combined to determine the moving distance for each iteration, so as to iteratively update the external parameter conversion relationship between the camera and the millimeter-wave radar. When the comparison result between the target function value calculated in the iterative calculation and the preset threshold is found to meet the conditions, the target extrinsic parameter matrix is ​​obtained based on the extrinsic parameter transformation relationship of the last iterative calculation. The formula for the objective function of the external parameter transformation relationship is as follows: Formula (3); in, Formula (IV); In the formula, These represent the trajectory points on the radar center fitting trajectory and the pixel center fitting trajectory, respectively; N is the number of sampled trajectory points i; A is the camera's intrinsic parameter matrix; R is the camera's rotation matrix; and T is the millimeter-wave radar's translation matrix.

2. The camera and radar extrinsic parameter calibration method based on trajectory alignment as described in claim 1, characterized in that, The step of "calculating the pixel center trajectory and the radar center trajectory based on a polynomial fitting algorithm to obtain the corresponding pixel center fitted trajectory and radar center fitted trajectory" specifically includes the following steps: Let the radar center fit the trajectory S The fitting polynomial for 1 is: Formula (1); Let the pixel center fit the trajectory S The fitting polynomial for 2 is: Formula (II); Multiple sets of paired points are selected from the preset reflector center in the pixel center trajectory and the radar center trajectory: ； By performing polynomial fitting on multiple sets of paired points of Equation (I) and Equation (II), the coefficients are obtained. ; In the formula, To fit the polynomial The coefficient; To fit the polynomial The coefficient.

3. The camera and radar extrinsic parameter calibration method based on trajectory alignment as described in claim 1, characterized in that, The formula for the loss function is as follows: Formula (5).

4. The camera and radar extrinsic parameter calibration method based on trajectory alignment as described in claim 1, characterized in that, The step of "combining the learning rate and loss gradient value to determine the moving distance calculated in each iteration, so as to iteratively update the extrinsic parameter conversion relationship between the camera and the millimeter-wave radar" specifically includes the following steps: Based on learning rate Loss gradient value; The distance to be moved in each iteration is determined as: The formula for iteratively updating the extrinsic parameter conversion relationship between the camera and the millimeter-wave radar is as follows: Formula (VI); in, The extrinsic parameter transformation relationship after iteration. This represents the external parameter transformation relationship before iteration.

5. The camera and radar extrinsic parameter calibration method based on trajectory alignment as described in claim 1, characterized in that, Before the step of "calculating the pixel center trajectory and the radar center trajectory based on the polynomial fitting algorithm to obtain the corresponding camera center fitting trajectory and radar center fitting trajectory", the following steps are specifically included: The pixel center trajectory is synchronized with the radar center trajectory using a message filter, and the camera center trajectory is corrected based on camera distortion parameters.

6. A camera and radar extrinsic parameter calibration system based on trajectory alignment, characterized in that, include: The initial trajectory acquisition module is used to acquire the preset trajectory of the center of the reflector running along the preset trajectory in the common detection area of ​​the camera and the millimeter-wave radar, as well as the pixel center trajectory and radar center trajectory detected by the camera and the millimeter-wave radar respectively. A trajectory fitting module, communicatively connected to the initial trajectory acquisition module, is used to calculate the pixel center trajectory and the radar center trajectory based on a polynomial fitting algorithm, thereby obtaining the pixel center fitted trajectory and the radar center fitted trajectory; and... An optimization module, which is communicatively connected to the fitting trajectory module, is used to align the pixel center fitting trajectory with the radar center fitting trajectory based on the batch optimization algorithm, and obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar. The step of "aligning the pixel center fitting trajectory with the radar center fitting trajectory based on the batch optimization algorithm to obtain the target extrinsic matrix in the extrinsic parameter transformation relationship between the camera and the millimeter-wave radar" specifically includes the following steps: Construct the objective function for the transformation relationship of extrinsic parameters; The objective function is iteratively calculated based on the batch optimization algorithm, and the loss gradient value is calculated based on the loss function. The learning rate and the loss gradient value are combined to determine the moving distance for each iteration, so as to iteratively update the external parameter conversion relationship between the camera and the millimeter-wave radar. When the comparison result between the target function value calculated in the iterative calculation and the preset threshold is found to meet the conditions, the target extrinsic parameter matrix is ​​obtained based on the extrinsic parameter transformation relationship of the last iterative calculation. The formula for the objective function of the external parameter transformation relationship is as follows: Formula (3); in, Formula (IV); In the formula, These represent the trajectory points on the radar center fitting trajectory and the pixel center fitting trajectory, respectively; N is the number of sampled trajectory points i; A is the camera's intrinsic parameter matrix; R is the camera's rotation matrix; and T is the millimeter-wave radar's translation matrix.

7. The camera and radar extrinsic parameter calibration system based on trajectory alignment as described in claim 6, characterized in that, It also includes a trajectory processing module that is communicatively connected to the fitted trajectory module, used to synchronize the pixel center trajectory with the radar center trajectory in time based on a message filter, and to correct the camera center trajectory based on camera distortion parameters.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the trajectory alignment-based camera and radar extrinsic parameter calibration system as described in any one of claims 1 to 5.

Images