Radar and Camera Calibration Method, Device, Equipment, Medium and Program Product

Abstract

The present application relates to a radar and camera calibration method, device, equipment, medium and program product. The method includes: collecting a to-be-processed image of a co-visible area through a camera, and collecting radar data of the co-visible area through a radar; calculating a first calibration object plane equation in the camera coordinate system and a first line equation of a target line in the calibration object; calculating a second calibration object plane equation in the radar coordinate system and a second line equation of the target line in the calibration object; calculating an initial attitude between the radar and the camera; calculating an initial position between the radar and the camera; constructing a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; and solving the first objective function according to the initial attitude and the initial position to obtain the target attitude and the target position of the radar and the camera. Using this method can improve the calibration efficiency of the radar and the camera.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent vehicles, and in particular, to a method, device, equipment, medium and program product for calibrating a radar and a camera. Background Art

[0002] For high-speed perception tasks, a single sensor often fails to meet the requirements at present. Therefore, multi-sensor fusion has become a trend. Multi-sensor calibration is the basis of multi-sensor fusion, and the calibration accuracy often determines the effects of multi-sensor perception and positioning. At present, the calibration technology of in-vehicle lidar and camera is still immature, and most of them only utilize the plane information of the calibration board without considering edge constraints. Therefore, it is necessary to place the calibration board at the co-viewing positions of at least three lidars and cameras to calibrate the extrinsic parameters between the lidar and the camera. This not only increases the cost, but also has low calibration accuracy and is difficult to achieve the purpose of commercial mass production.

[0003] With the continuous popularization and implementation of high-speed intelligent driving technology, the efficiency of joint calibration of lidar and camera needs to be improved. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a method, device, equipment, medium and program product for calibrating a radar and a camera, which can improve the calibration efficiency of the radar and the camera.

[0005] In a first aspect, the present application provides a method for calibrating a radar and a camera, the method comprising:

[0006] Collecting a to-be-processed image of a co-viewing area through a camera, and collecting radar data of the co-viewing area through a radar, where the co-viewing area is a co-viewing area of the camera and the radar, and a calibration object is arranged at the co-viewing area;

[0007] Calculating a first calibration object plane equation in a camera coordinate system and a first line equation of a target line in the calibration object according to the to-be-processed image and the calibration object in the co-viewing area;

[0008] Calculating a second calibration object plane equation in a radar coordinate system and a second line equation of the target line in the calibration object according to the radar data;

[0009] Calculating an initial pose between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation and the second line equation;

[0010] Calculating an initial position between the radar and the camera according to the target plane constraint and target line constraint of the calibration object in the radar coordinate system and the camera coordinate system;

[0011] Construct a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system;

[0012] Take the initial attitude and the initial position as the initial values of non-linear optimization, and solve the first objective function to obtain the target attitude and the target position of the radar and the camera.

[0013] In the above radar and camera calibration method, the vehicle terminal calculates the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object according to the to-be-processed image and the calibration object in the co-visible area; calculates the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object according to the radar data; calculates the initial attitude between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation and the second line equation; calculates the initial position between the radar and the camera according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; constructs a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; takes the initial attitude and the initial position as the initial values of non-linear optimization, and solves the first objective function to obtain the target attitude and the target position of the radar and the camera. Through the constraint relationship between the calibration object plane and the target line, even if the calibration object is placed in only one co-visible area, the external parameters between the radar and the camera, that is, the target attitude and the target position of the radar and the camera, can be accurately calculated, thereby reducing operations and improving efficiency.

[0014] In one embodiment, the calculating the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object according to the to-be-processed image and the calibration object in the co-visible area includes:

[0015] Identify the corner points and boundaries of the calibration object in the to-be-processed image;

[0016] Use the homography matrix to calculate the first calibration object plane equation in the camera coordinate system according to the corner points of the calibration object in the to-be-processed image;

[0017] Obtain the plane passing through the boundary of the calibration object and the optical center of the camera, and calculate the intersection line of the plane and the calibration object in the to-be-processed image as the target line;

[0018] Calculate the first line equation of the target line in the camera coordinate system.

[0019] In the above embodiments, through the optical center of the camera, the plane where the image to be processed is located, and the calibration object entity, the first calibration object plane equation and the first line equation can be calculated, so as to obtain the plane feature and line feature in the camera coordinate system, laying a foundation for subsequent joint calibration.

[0020] In one of the embodiments, the calculating the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object according to the radar data includes:

[0021] Obtaining the point cloud data of the calibration object in the radar data, and calculating the second calibration object plane equation in the radar coordinate system according to the point cloud data of the calibration object in the radar data;

[0022] Calculating the second line equation of the target line in the radar coordinate system according to the point cloud data on the target line of the calibration object in the radar data.

[0023] In the above embodiments, through the radar data, the second calibration object plane equation and the second line equation can be calculated, so as to obtain the plane feature and line feature in the radar coordinate system, laying a foundation for subsequent joint calibration.

[0024] In one of the embodiments, the calculating the initial pose between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation includes:

[0025] Obtaining the initial pose to be calculated between the radar and the camera;

[0026] Projecting the normal vector of the second calibration object plane equation in the radar coordinate system to obtain the normal vector of the third calibration object plane equation in the camera coordinate system through the initial pose to be calculated;

[0027] Projecting the direction vector of the second line equation in the radar coordinate system to obtain the direction vector of the third line equation in the camera coordinate system through the initial pose to be calculated;

[0028] Constructing a second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation;

[0029] Solving the second objective function to obtain the solution corresponding to the initial pose to be calculated as the initial pose.

[0030] In one embodiment, constructing the second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation includes:

[0031] Constructing the following second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation:

[0032]

[0033] where N represents the number of calibration objects, where 1 ≤ i ≤ N, M is the number of target lines, where 1 ≤ j ≤ M, R CL is the initial pose to be calculated, is the calculated initial pose, is the normal vector of the second calibration object plane equation of the i-th calibration object in the radar coordinate system, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the direction vector of the second line equation of the j-th target line of the i-th calibration object in the radar coordinate system, is the direction vector of the first line equation of the j-th target line of the i-th calibration object in the camera coordinate system.

[0034] In the above embodiment, by projecting the normal vector of the calibration object plane equation in the radar coordinate system through the pose between the radar and the camera into the camera coordinate system, it should be equal to the normal vector of the calibration object plane equation in the camera coordinate system; by projecting the direction vector of the calibration object boundary equation in the radar coordinate system through the pose between the radar and the camera into the camera coordinate system, it should be equal to the direction vector of the calibration object boundary equation in the camera coordinate system. Based on this physical meaning, the second objective function is constructed, and calculating the minimum value of this second objective function is the initial pose between the radar and the camera.

[0035] In one embodiment, calculating the initial position between the radar and the camera according to the target plane constraint and target line constraint of the calibration object in the radar coordinate system and the camera coordinate system includes:

[0036] Obtaining the initial pose to be calculated and the position to be calculated between the radar and the camera;

[0037] Obtaining the point cloud data of the calibration object in the radar coordinate system;

[0038] By using the to-be-calculated attitude and to-be-calculated position between the radar and the camera, the point cloud data is converted into the camera coordinate system to obtain reference point cloud data;

[0039] Obtaining a target plane constraint equation according to the reference point cloud data and the plane equation in the camera coordinate system;

[0040] Obtaining a target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system;

[0041] An initial position between the radar and the camera is calculated according to the plane constraint and the line constraint.

[0042] In the above embodiment, after the surface features and line features are acquired, the plane constraints and boundary constraints are fully considered, and the initial position can be obtained according to the simultaneous equations of the plane constraints and the boundary constraints.

[0043] In one embodiment, the step of converting the point cloud data into the camera coordinate system to obtain reference point cloud data by using the to-be-calculated attitude and to-be-calculated position between the radar and the camera includes:

[0044] According to the following formula, the point cloud data is converted to the camera coordinate system to obtain reference point cloud data through the to-be-calculated attitude and to-be-calculated position between the radar and the camera:

[0045]

[0046] in, is the point cloud data of the i-th calibration object in the camera coordinate system, that is, the reference point cloud data, R CL is the posture to be calculated, is the point cloud data of the i-th calibration object in the radar coordinate system, t CL is the position to be calculated.

[0047] In one embodiment, obtaining the target plane constraint equation according to the reference point cloud data and the plane equation in the camera coordinate system includes:

[0048] Get the plane equation in the camera coordinate system:

[0049]

[0050] in, is the normal vector of the first calibration object plane equation in the camera coordinate system, is the direction vector of the first line equation of the i-th calibration object in the camera coordinate system;

[0051] Obtain the target plane constraint equation based on the reference point cloud data and the plane equation in the camera coordinate system:

[0052]

[0053] In one embodiment, the obtaining of the target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system includes:

[0054] Obtain the line constraint equation in the camera coordinate system:

[0055]

[0056] Wherein, The direction vector of the first line equation of the j-th target line of the i-th calibration object in the camera coordinate system, is a point on the j-th target line of the i-th calibration object in the radar coordinate system, is a point on the j-th target line of the i-th calibration object in the camera coordinate system;

[0057] Obtain the target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system:

[0058]

[0059] Where R CL is the attitude to be calculated, t CL is the position to be calculated.

[0060] In one embodiment, the constructing of the first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system includes:

[0061] Construct the following first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system:

[0062]

[0063] Where, R CL is the attitude to be calculated, t CL is the position to be calculated, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is a point on the calibration object in the radar coordinate system, is the direction vector of the first line equation in the i-th calibration object in the camera coordinate system, It is the direction vector of the first line equation of the j-th target line in the i-th calibration object in the camera coordinate system, K ij It is the number of point clouds on the j-th target line under the pose of the i-th calibration object, It is the k-th point on the j-th target line of the i-th calibration object in the radar coordinate system, The point on the j-th target line of the i-th calibration object in the camera coordinate system.

[0064] In the above embodiments, the point clouds on the calibration object in the radar coordinate system are projected into the camera coordinate system through the extrinsic parameters between the radar and the camera, and they still satisfy the plane equation of the calibration object in the camera coordinate system in the camera coordinate system; the point clouds on the boundary of the calibration object in the radar coordinate system are projected into the camera coordinate system through the extrinsic parameters between the radar and the camera, and they still satisfy the line constraint equation of the calibration object boundary in the camera coordinate system in the camera coordinate system.

[0065] In one of the embodiments, the step of using the initial attitude and the initial position as the initial values of the non-linear optimization to solve the first objective function to obtain the target attitude and target position of the radar and the camera includes:

[0066] Using the initial attitude and the initial position as the initial values of the non-linear optimization, and optimizing the first objective function through the LM gradient descent strategy to obtain the target attitude and target position between the optimized radar and the camera.

[0067] In a second aspect, the present application also provides a radar and camera calibration device, and the device includes:

[0068] An acquisition module, configured to acquire the to-be-processed image of the common view area through a camera, and acquire the radar data of the common view area through a radar, where the common view area is the common view area of the camera and the radar, and a calibration object is arranged in the common view area;

[0069] An equation generation module, configured to calculate the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object according to the to-be-processed image and the calibration object in the common view area; calculate the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object according to the radar data;

[0070] An initial pose calculation module, configured to calculate the initial attitude between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation; calculate the initial position between the radar and the camera according to the target plane constraint and target line constraint of the calibration object in the radar coordinate system and the camera coordinate system;

[0071] A first objective function construction module, configured to construct a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system;

[0072] A result calculation module, configured to use the initial attitude and the initial position as the initial values of non-linear optimization, and solve the first objective function to obtain the target attitude and the target position of the radar and the camera.

[0073] For the above radar and camera calibration device, the vehicle terminal calculates, according to the to-be-processed image and the calibration object in the co-visible area, a first calibration object plane equation in the camera coordinate system and a first line equation of the target line in the calibration object; calculates, according to the radar data, a second calibration object plane equation in the radar coordinate system and a second line equation of the target line in the calibration object; calculates, according to the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation, an initial attitude between the radar and the camera; calculates, according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system, an initial position between the radar and the camera; constructs a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; uses the initial attitude and the initial position as the initial values of non-linear optimization, and solves the first objective function to obtain the target attitude and the target position of the radar and the camera. Through the constraint relationship between the calibration object plane and the target line, even if the calibration object is placed in only one co-visible area, the external parameters between the radar and the camera, that is, the target attitude and the target position of the radar and the camera, can be accurately calculated, thereby reducing operations and improving efficiency.

[0074] In a third aspect, the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the above method are implemented.

[0075] For the above computer device, the vehicle terminal calculates the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object based on the image to be processed and the calibration object in the co-visible area; calculates the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object based on the radar data; calculates the initial attitude between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation and the second line equation; calculates the initial position between the radar and the camera according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; constructs a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; uses the initial attitude and the initial position as the initial values of non-linear optimization to solve the first objective function to obtain the target attitude and the target position of the radar and the camera. Through the constraint relationship between the calibration object plane and the target line, even if the calibration object is placed in only one co-visible area, the external parameters between the radar and the camera, that is, the target attitude and the target position of the radar and the camera, can be accurately calculated, thereby reducing operations and improving efficiency.

[0076] In a fourth aspect, the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0077] For the above storage medium, the vehicle terminal calculates the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object based on the image to be processed and the calibration object in the co-visible area; calculates the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object based on the radar data; calculates the initial attitude between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation and the second line equation; calculates the initial position between the radar and the camera according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; constructs a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; uses the initial attitude and the initial position as the initial values of non-linear optimization to solve the first objective function to obtain the target attitude and the target position of the radar and the camera. Through the constraint relationship between the calibration object plane and the target line, even if the calibration object is placed in only one co-visible area, the external parameters between the radar and the camera, that is, the target attitude and the target position of the radar and the camera, can be accurately calculated, thereby reducing operations and improving efficiency.

[0078] In a fifth aspect, the present application also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0079] For the above computer program product, the vehicle terminal calculates the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object based on the image to be processed and the calibration object in the co-visible area; calculates the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object based on the radar data; calculates the initial attitude between the radar and the camera based on the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation; calculates the initial position between the radar and the camera based on the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; constructs a first objective function based on the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; uses the initial attitude and the initial position as the initial values of the nonlinear optimization, solves the first objective function to obtain the target attitude and the target position of the radar and the camera. Through the constraint relationship between the calibration object plane and the target line, even if the calibration object is placed in only one co-visible area, the external parameters between the radar and the camera, that is, the target attitude and the target position of the radar and the camera, can be accurately calculated, thereby reducing operations and improving efficiency. Description of the Drawings

[0080] Figure 1 It is an application environment diagram of the radar and camera calibration method in an embodiment;

[0081] Figure 2 It is a schematic flowchart of the radar and camera calibration method in an embodiment;

[0082] Figure 3 It is a schematic diagram of the position of the calibration object in an embodiment;

[0083] Figure 4 It is a schematic diagram of the radar data in an embodiment;

[0084] Figure 5 It is a flowchart of the initial position calculation step in an embodiment;

[0085] Figure 6 It is a structural block diagram of the radar and camera calibration device in an embodiment;

[0086] Figure 7 It is an internal structure diagram of a computer device in an embodiment. Detailed Embodiments

[0087] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0088] The radar and camera calibration method provided by the embodiments of the present application can be applied to, for exampleFigure 1 In the application environment shown. Among them, the vehicle terminal 102 communicates with the camera 104 and the radar 106 on the vehicle. The data storage system can store the data that the vehicle terminal 102 needs to process. The data storage system can be integrated on the vehicle terminal 102, or placed on the cloud or other network servers.

[0089] Among them, the camera 104 collects the to-be-processed images in the common view area, and the radar 106 collects the radar data in the common view area. The common view area is the common view area of the camera and the radar, and a calibration object is set at the common view area. The camera 104 sends the collected to-be-processed images and the radar 106 sends the collected radar data to the vehicle terminal 102. The vehicle terminal 102 calculates the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object according to the to-be-processed images and the calibration object in the common view area; calculates the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object according to the radar data; calculates the initial attitude between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation and the second line equation; calculates the initial position between the radar and the camera according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; constructs the first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; uses the initial attitude and the initial position as the initial values of the non-linear optimization, and solves the first objective function to obtain the target attitude and the target position of the radar and the camera. In this way, by introducing the target plane constraint and the target line constraint, and only placing the calibration object in one common view area, the external parameters between the radar 106 and the camera 104, that is, the target attitude and the target position of the radar and the camera, can also be calculated.

[0090] Among them, the vehicle terminal 102 can be an intelligent vehicle-mounted device, etc. The camera 104 is a vehicle-mounted camera, such as an ordinary vehicle-mounted camera or a fish-eye camera, etc. The radar 106 is a vehicle-mounted radar, such as a vehicle-mounted lidar or a vehicle-mounted millimeter-wave radar, etc.

[0091] In one embodiment, as Figure 2 shown, a method for calibrating a radar and a camera is provided. Taking the vehicle terminal 102 in Figure 1 as an example for illustration, the method includes the following steps:

[0092] S202: Collect the to-be-processed images in the common view area through the camera, and collect the radar data in the common view area through the radar. The common view area is the common view area of the camera and the radar, and a calibration object is set at the common view area.

[0093] Specifically, the co-visible area is the co-visible area of the camera and the radar, that is, the area that can be collected by both the camera and the radar. For convenience, calibration objects are set in this co-visible area. Optionally, the number of calibration objects can be set as needed, and there is at least one calibration object.

[0094] Specifically, refer to Figure 3 as shown, where Figure 3 the position of the calibration object, the image to be processed collected, and the position of the optical center of the camera lens are given.

[0095] Among them, the camera collects an image of the co-visible area to obtain an image to be processed, so that the image of the calibration object is included in the image to be processed. The radar device collects radar data from the co-visible area to obtain radar data. Thus, the vehicle terminal can obtain the image to be processed collected by the camera and the radar data collected by the radar device. Optionally, the image to be processed carries a first timestamp, and the radar data carries a second timestamp. The corresponding image to be processed and radar data are obtained through the first timestamp and the second timestamp.

[0096] S204: According to the image to be processed and the calibration object in the co-visible area, calculate the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object.

[0097] Specifically, the first calibration object plane equation is the plane equation of the calibration object image in the image to be processed in the camera coordinate system. The first calibration object plane equation can be obtained by obtaining the positions of the target points on the calibration object in the image to be processed and then calculating using the homography matrix. The calibration object is preferably a calibration board, such as a quadrilateral calibration board, and the target points are preferably corner points, that is, the four top corners of the quadrilateral calibration board.

[0098] The first line equation can be the first line equation of the target line in the calibration object image in the image to be processed. The target line can be any line in the calibration object image. Preferably, the target line is the boundary line in the calibration object image. The acquisition method of the target line includes: acquiring the corresponding physical line in the calibration object, determining the plane where the physical line and the optical center of the camera are located, and finding the intersection line of this plane and the calibration object plane as the target line. The optical center can refer to the center of the lens.

[0099] S206: Calculate the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object according to the radar data.

[0100] Specifically, the second calibration object equation can be the plane equation of the calibration object in the radar coordinate system fitted by the RANSAC algorithm according to the radar data. Specifically, refer to Figure 4As shown, the radar data may be point cloud data, including a plurality of points, and the second calibration object plane equation of the calibration object in the radar coordinate system is fitted by the RANSAC algorithm according to the plurality of points. The second line equation may be obtained by fitting the points passing through the target line. It should be noted that the RANSAC algorithm is used as an example in this embodiment, and other algorithms may be used in other embodiments, which are not specifically limited here.

[0101] S208: Calculate an initial posture between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation.

[0102] Specifically, the initial posture refers to the rotation matrix between the radar and the camera, wherein the vehicle terminal solves the initial posture between the lidar and the camera through the normal vector of the plane equation of the calibration object in the radar coordinate system and the camera coordinate system and the direction vector of the line equation, that is, the initial posture between the lidar and the camera is solved according to the normal vector of the plane equation of the first calibration object, the normal vector of the plane equation of the second calibration object, the direction vector of the first line equation and the direction vector of the second line equation.

[0103] Among them, the normal vector of the calibration plate plane equation in the radar coordinate system is projected to the camera coordinate system through the posture between the radar and the camera, which should be equal to the normal vector of the calibration plate plane equation in the camera coordinate system; the direction vector of the calibration plate boundary equation in the radar coordinate system is projected to the camera coordinate system through the posture between the radar and the camera, which should be equal to the direction vector of the calibration plate boundary equation in the camera coordinate system. In this way, according to this transformation, the objective function for calculating the initial posture can be established, and the initial posture can be calculated according to the obtained objective function for calculating the initial posture.

[0104] S210: Calculate the initial position between the radar and the camera according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system.

[0105] Specifically, the basic principle of the target plane constraint is to project the point cloud on the calibration object in the radar coordinate system to the camera coordinate system through the external parameters between the radar and the camera, which still satisfies the plane equation of the calibration object in the camera coordinate system.

[0106] The basic principle of the target line constraint is to project the point cloud on the boundary of the calibration object in the radar coordinate system to the camera coordinate system through the external parameters between the radar and the camera, which still satisfies the line constraint equation of the boundary of the calibration object in the camera coordinate system.

[0107] In this way, the vehicle terminal can obtain the initial position between the radar and the camera by jointly constraining the target plane and the target line.

[0108] S212: Construct a first objective function according to the target plane constraint and target line constraint of the calibration object in the radar coordinate system and the camera coordinate system.

[0109] Specifically, the first objective function can be an error function, and the meaning of the error function is as follows: The point cloud on the calibration object in the radar coordinate system is projected into the camera coordinate system through the external parameters between the radar and the camera, and it still satisfies the plane equation of the calibration object in the camera coordinate system in the camera coordinate system; the point cloud on the boundary of the calibration board in the radar coordinate system is projected into the camera coordinate system through the external parameters between the radar and the camera, and it still satisfies the line constraint equation of the calibration object boundary in the camera coordinate system in the camera coordinate system.

[0110] S214: Use the initial attitude and initial position as the initial values of the nonlinear optimization, and solve the first objective function to obtain the target attitude and target position of the radar and the camera.

[0111] Specifically, the vehicle terminal uses the initial attitude and initial position as the initial values of the nonlinear optimization, constructs an error function through the calibration object plane constraint and boundary constraint relationship, and finally solves the error function to obtain the target attitude and target position of the radar and the camera.

[0112] Among them, the vehicle terminal optimizes the error function through the LM gradient descent strategy, and finally obtains the target attitude and target position between the optimized lidar and the camera. In other embodiments, the vehicle terminal can optimize the error function through other strategies, which are not specifically limited here.

[0113] In the above embodiments, the vehicle terminal calculates the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object according to the image to be processed and the calibration object in the co-visible area; calculates the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object according to the radar data; calculates the initial attitude between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation and the second line equation; calculates the initial position between the radar and the camera according to the target plane constraint and target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; constructs a first objective function according to the target plane constraint and target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; uses the initial attitude and initial position as the initial values of the nonlinear optimization, and solves the first objective function to obtain the target attitude and target position of the radar and the camera. Through the constraint relationship of the calibration object plane and the target line, even if the calibration object is placed in only one co-visible area, the external parameters between the radar and the camera, that is, the target attitude and target position of the radar and the camera, can be accurately calculated, thereby reducing operations and improving efficiency.

[0114] In one embodiment, according to the image to be processed and the calibration object in the co-visual area, the first calibration object plane equation in the camera coordinate system and the first line equation of the target line in the calibration object are calculated, including: identifying the corner points and boundaries of the calibration object in the image to be processed; using the homography matrix to calculate the first calibration object plane equation in the camera coordinate system according to the corner points of the calibration object in the image to be processed; obtaining the plane passing through the boundary of the calibration object and the optical center of the camera, and calculating the intersection line of the plane and the calibration object in the image to be processed as the target line; calculating the first line equation of the target line in the camera coordinate system.

[0115] Specifically, as shown in Figure 3 First, the vehicle terminal identifies the calibration object in the image to be processed, and then determines the corner points and boundaries of the calibration object. Then, the detected corner points of the calibration object are used to calculate the first calibration object plane equation of the calibration object in the camera coordinate system by using the homography matrix. The vehicle terminal calculates the intersection line of the plane passing through the detected boundary of the calibration object and the optical center of the camera (the center of the lens can be approximately regarded as the optical center) and the calibration object plane to obtain the first line equation of the calibration object boundary in the camera coordinate system; as shown in Figure 3 The dot represents the optical center of the camera. The plane composed of the first line and the second line is the plane passing through the boundary of the calibration object and the optical center of the camera, and the first line is the intersection line of this plane and the calibration plate plane.

[0116] In the above embodiment, through the optical center of the camera, the plane where the image to be processed is located, and the calibration object entity, the first calibration object plane equation and the first line equation can be calculated, so as to obtain the plane feature and line feature in the camera coordinate system, laying a foundation for subsequent joint calibration.

[0117] In one embodiment, according to the radar data, the second calibration object plane equation in the radar coordinate system and the second line equation of the target line in the calibration object are calculated, including: obtaining the point cloud data of the calibration object in the radar data, and calculating the second calibration object plane equation in the radar coordinate system according to the point cloud data of the calibration object in the radar data; calculating the second line equation of the target line in the radar coordinate system according to the point cloud data on the target line of the calibration object in the radar data.

[0118] Specifically, the radar data includes the point cloud data of the calibration object. Among them, the vehicle terminal fits the second calibration object plane equation of the calibration object in the lidar coordinate system through the RANSAC algorithm, and calculates the second line equation of the point cloud data corresponding to the boundary of the calibration object. As shown in the above Figure 4 The second calibration object plane equation is fitted through the point cloud, and the second line equation of the calibration object boundary is calculated through the point cloud on the boundary.

[0119] In the above embodiments, based on the radar data, the plane equation of the second calibration object and the line equation of the second calibration object can be calculated, so as to obtain the plane feature and line feature in the radar coordinate system, laying a foundation for subsequent joint calibration.

[0120] In one embodiment, based on the plane equation of the first calibration object, the plane equation of the second calibration object, the line equation of the first line, and the line equation of the second line, the initial pose between the radar and the camera is calculated, including: obtaining the initial pose to be calculated between the radar and the camera; projecting the normal vector of the plane equation of the second calibration object in the radar coordinate system to the camera coordinate system through the initial pose to be calculated to obtain the normal vector of the plane equation of the third calibration object; projecting the direction vector of the line equation of the second line in the radar coordinate system to the camera coordinate system through the initial pose to be calculated to obtain the direction vector of the line equation of the third line; constructing a second objective function based on the normal vector of the plane equation of the third calibration object, the normal vector of the plane equation of the first calibration object, the direction vector of the line equation of the third line, and the direction vector of the line equation of the first line; solving the second objective function to obtain the solution corresponding to the initial pose to be calculated, and taking it as the initial pose.

[0121] Specifically, the initial pose to be calculated can be represented by a symbol in advance, which represents the initial pose to be calculated between the radar and the camera.

[0122] Among them, the meaning of the second objective function is that the normal vector of the plane equation of the calibration object in the radar coordinate system should be equal to the normal vector of the plane equation of the calibration object in the camera coordinate system after being projected through the pose between the radar and the camera; the direction vector of the boundary equation of the calibration object in the radar coordinate system should be equal to the direction vector of the boundary equation of the calibration object in the camera coordinate system after being projected through the pose between the radar and the camera. Based on this physical meaning, the second objective function is constructed, and the minimum value of the second objective function is calculated, which is the initial pose between the radar and the camera.

[0123] Among them, optionally, constructing the second objective function based on the normal vector of the plane equation of the third calibration object, the normal vector of the plane equation of the first calibration object, the direction vector of the line equation of the third line, and the direction vector of the line equation of the first line includes: constructing the following second objective function based on the normal vector of the plane equation of the third calibration object, the normal vector of the plane equation of the first calibration object, the direction vector of the line equation of the third line, and the direction vector of the line equation of the first line:

[0124]

[0125] Among them, N represents the number of calibration objects, where 1≤i≤N, M is the number of target lines, where 1≤j≤M, R CL is the initial pose to be calculated, is the calculated initial pose, is the normal vector of the second calibration object plane equation of the i-th calibration object in the radar coordinate system, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the direction vector of the second line equation of the j-th target line of the i-th calibration object in the radar coordinate system, is the direction vector of the first line equation of the j-th target line of the i-th calibration object in the camera coordinate system.

[0126] In this way, by solving the above objective function, the initial attitude between the lidar and the camera can be obtained.

[0127] Among them, represents the difference between the normal vector of the calibration object plane equation in the radar coordinate system projected onto the camera coordinate system through the attitude between the radar and the camera and the normal vector of the calibration object plane equation in the camera coordinate system. represents the difference between the direction vector of the calibration object boundary equation in the radar coordinate system projected onto the camera coordinate system through the attitude between the radar and the camera and the direction vector of the calibration object boundary equation in the camera coordinate system. The minimum value of the sum of these two differences is the initial attitude.

[0128] In one embodiment, when there is more than one calibration object, the differences of the plane normal vectors of all calibration objects and the differences of the direction vectors of each target line of all calibration objects are solved to improve the calculation accuracy through multiple sets of data.

[0129] In the above embodiment, the normal vector of the calibration object plane equation in the radar coordinate system projected onto the camera coordinate system through the attitude between the radar and the camera should be equal to the normal vector of the calibration object plane equation in the camera coordinate system; the direction vector of the calibration object boundary equation in the radar coordinate system projected onto the camera coordinate system through the attitude between the radar and the camera should be equal to the direction vector of the calibration object boundary equation in the camera coordinate system. According to this physical meaning, a second objective function is constructed, and the minimum value of this second objective function is the initial attitude between the radar and the camera.

[0130] In one embodiment, as shown in Figure 5 shown, Figure 5 is a flowchart of the initial position calculation steps in an embodiment. In this embodiment, according to the target plane constraint and target line constraint of the calibration object in the radar coordinate system and the camera coordinate system, the initial position between the radar and the camera is calculated, including:

[0131] S502: Obtain the initial attitude and position to be calculated between the radar and the camera, and obtain the point cloud data of the calibration object in the radar coordinate system.

[0132] Among them, the initial attitude to be calculated can be pre-represented by symbols, which represents the initial attitude to be calculated between the radar and the camera. The position to be calculated can be pre-represented by symbols, which represents the initial position to be calculated between the radar and the camera.

[0133] The point cloud data is the radar data of the calibration object in the radar coordinate system, where the point cloud data can only include the point cloud data of the calibration object.

[0134] S504: Through the attitude to be calculated and the position to be calculated between the radar and the camera, the point cloud data is transformed into the camera coordinate system to obtain the reference point cloud data.

[0135] Among them, when the point cloud data is transformed into the camera coordinate system, it is transformed through the transformation matrix, and the transformation matrix is generated through the attitude to be calculated and the position to be calculated.

[0136] In one embodiment, through the attitude to be calculated and the position to be calculated between the radar and the camera, the point cloud data is transformed into the camera coordinate system to obtain the reference point cloud data, including: According to the following formula, through the attitude to be calculated and the position to be calculated between the radar and the camera, the point cloud data is transformed into the camera coordinate system to obtain the reference point cloud data:

[0137]

[0138] Among them, is the point cloud data of the i-th calibration object in the camera coordinate system, that is, the reference point cloud data, R CL is the attitude to be calculated, is the point cloud data of the i-th calibration object in the radar coordinate system, t CL is the position to be calculated.

[0139] Among them, the above formula is for coordinate transformation, which transforms the points in the radar coordinate system into the camera coordinate system.

[0140] S506: According to the reference point cloud data and the plane equation in the camera coordinate system, the target plane constraint equation is obtained.

[0141] Specifically, the points after coordinate transformation, that is, the reference point cloud data, also satisfy the plane equation in the camera coordinate system.

[0142] In one embodiment, according to the reference point cloud data and the plane equation in the camera coordinate system, the target plane constraint equation is obtained, including: Obtaining the plane equation in the camera coordinate system:

[0143]

[0144] Among them, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the direction vector of the first line equation of the i-th calibration object in the camera coordinate system; the target plane constraint equation is obtained according to the reference point cloud data and the plane equation in the camera coordinate system:

[0145]

[0146] Among them, the above formula (4) can be obtained by substituting formula (2) into formula (3).

[0147] The meaning of the above plane constraint equation is: the point cloud on the calibration object in the lidar coordinate system is projected into the camera coordinate system through the external parameters between the lidar and the camera, and it still satisfies the plane equation of the calibration object in the camera coordinate system in the camera coordinate system.

[0148] S508: Obtain the target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system.

[0149] Specifically, the points after coordinate transformation, that is, the reference point cloud data, also satisfy the line equation in the camera coordinate system.

[0150] In one of the embodiments, obtaining the target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system includes: obtaining the line constraint equation in the camera coordinate system:

[0151]

[0152] Among them, is the direction vector of the first line equation of the j-th target line of the i-th calibration object in the camera coordinate system, is the point on the j-th target line of the i-th calibration object in the lidar coordinate system, is the point on the j-th target line of the i-th calibration object in the camera coordinate system; the target line constraint equation is obtained according to the reference point cloud data and the line constraint equation in the camera coordinate system:

[0153]

[0154] Among them R CL is the attitude to be calculated, t CL is the position to be calculated.

[0155] Among them, is the point on the center of the calibration object boundary in the camera coordinate system, and the point on the calibration object boundary in the lidar coordinate system is projected through the external parameters R CL and t CL, convert to the camera coordinate system to obtain the points on the boundary of the calibration edge in the camera coordinate system That is

[0156] Then subtract the converted points from the points on the boundary of the calibration board in the camera coordinate system to obtain a vector. Theoretically, this vector satisfies the line equation of the calibration board boundary constraint, that is, the projection of this vector in the direction of the normal vector of the direction vector of the calibration board boundary is 0, that is, the above formula (5).

[0157] Among them, substituting the above formula (5) into can obtain the above formula (6).

[0158] The meaning of the above boundary constraint equation is: project the point cloud on the boundary of the calibration object in the radar coordinate system to the camera coordinate system through the external parameters between the lidar and the camera, and it still satisfies the line constraint equation of the calibration object boundary in the camera coordinate system.

[0159] S510: Calculate the initial position between the radar and the camera according to the plane constraint and the line constraint.

[0160] Specifically, in this embodiment, the vehicle terminal can solve the initial position between the lidar and the camera by simultaneously solving the plane constraint equation and the boundary constraint equation

[0161] In the above embodiment, after obtaining the plane feature and the line feature, the plane constraint and the boundary constraint are fully considered, and the initial position can be obtained by simultaneously solving the equations according to the plane constraint and the boundary constraint.

[0162] In one of the embodiments, according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system, a first objective function is constructed, including: according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system, construct the following first objective function:

[0163]

[0164]

[0165] Among them, R CL is the attitude to be calculated, t CL is the position to be calculated, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the point on the calibration object in the radar coordinate system, is the direction vector of the first line equation of the i-th calibration object in the camera coordinate system, is the direction vector of the first line equation of the j-th target line in the i-th calibration object in the camera coordinate system, K ij is the number of point clouds on the j-th target line in the pose of the i-th calibration object, is the k-th point on the j-th target line of the i-th calibration object in the radar coordinate system, Points on the j-th target line of the i-th calibration object in the camera coordinate system.

[0166] Among them, the first objective function can be an error function. The meaning of this product function is to project the point cloud on the calibration board in the lidar coordinate system to the camera coordinate system through the external parameters between the lidar and the camera, and it still satisfies the plane equation of the calibration board in the camera coordinate system in the camera coordinate system; project the point cloud on the boundary of the calibration board in the lidar coordinate system to the camera coordinate system through the external parameters between the lidar and the camera, and it still satisfies the line constraint equation of the calibration board boundary in the camera coordinate system in the camera coordinate system.

[0167] In one embodiment, taking the initial attitude and initial position as the initial values of the nonlinear optimization, solving the first objective function to obtain the target attitude and target position of the lidar and the camera includes: taking the initial attitude and initial position as the initial values of the nonlinear optimization, and optimizing the first objective function through the LM gradient descent strategy to obtain the target attitude and target position between the optimized lidar and the camera.

[0168] Among them, for a nonlinear least squares objective function, usually an initial value can be given, and the Jacobian matrix of the objective function can be calculated using numerical solution, and then the LM gradient descent strategy can be used for solution.

[0169] Among them, finally, the error function is optimized through the LM gradient descent strategy, and finally the external parameters R CL and t CL between the lidar and the camera are obtained, thus completing the joint calibration between the lidar and the camera.

[0170] In this embodiment, through the constraint relationship between the calibration board plane and the edge line, even if the calibration board is placed in only one co-visible area, the external parameters between the lidar and the camera can be accurately calculated.

[0171] It should be understood that although the steps in the flowcharts involved in the above embodiments are sequentially shown according to the indication of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same moment, but can be executed at different moments. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0172] Based on the same inventive concept, an embodiment of the present application further provides a radar and camera calibration device for implementing the radar and camera calibration method involved above. The solution provided by this device to solve the problem is similar to the solution recorded in the above method. Therefore, the specific limitations in one or more embodiments of the radar and camera calibration device provided below can refer to the limitations on the radar and camera calibration method in the above text, and will not be repeated here.

[0173] In one embodiment, as Figure 6 shown, a radar and camera calibration device is provided, including: an acquisition module 601, an equation generation module 602, an initial pose calculation module 603, a first objective function construction module 604, and a result calculation module 605, where:

[0174] The acquisition module 601 is configured to collect the to-be-processed images of the common view area through a camera, and collect the radar data of the common view area through a radar. The common view area is the common view area of the camera and the radar, and a calibration object is arranged at the common view area;

[0175] The first equation generation module 602 is configured to calculate, according to the to-be-processed images and the calibration object in the common view area, a first calibration object plane equation in the camera coordinate system and a first line equation of the target line in the calibration object; calculate, according to the radar data, a second calibration object plane equation in the radar coordinate system and a second line equation of the target line in the calibration object;

[0176] The initial pose calculation module 603 is configured to calculate, according to the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation, the initial attitude between the radar and the camera; calculate, according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system, the initial position between the radar and the camera;

[0177] The first objective function construction module 604 is configured to construct a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system;

[0178] The result calculation module 605 is configured to use the initial pose and the initial position as the initial values of the non-linear optimization, and solve the first objective function to obtain the target pose and the target position of the radar and the camera.

[0179] In one embodiment, the above equation generation module 602 includes:

[0180] The recognition unit is configured to recognize the corner points and boundaries of the calibration object in the image to be processed;

[0181] The first calibration object plane equation generation unit is configured to calculate the first calibration object plane equation in the camera coordinate system according to the corner points of the calibration object in the image to be processed by using the homography matrix;

[0182] The target line acquisition unit is configured to acquire the plane passing through the boundary of the calibration object and the optical center of the camera, and calculate the intersection line of the plane and the calibration object in the image to be processed as the target line;

[0183] The first line equation generation unit is configured to calculate the first line equation of the target line in the camera coordinate system.

[0184] In one embodiment, the above equation generation module 602 includes:

[0185] The second calibration object plane equation generation unit is configured to acquire the point cloud data of the calibration object in the radar data, and calculate the second calibration object plane equation in the radar coordinate system according to the point cloud data of the calibration object in the radar data;

[0186] The second line equation generation unit is configured to calculate the second line equation of the target line in the radar coordinate system according to the point cloud data on the target line of the calibration object in the radar data.

[0187] In one embodiment, the above initial pose calculation module 603 includes:

[0188] The initial pose to be calculated acquisition unit is configured to acquire the initial pose to be calculated between the radar and the camera;

[0189] The normal vector projection unit is configured to project the normal vector of the second calibration object plane equation in the radar coordinate system to the camera coordinate system through the initial pose to be calculated to obtain the normal vector of the third calibration object plane equation;

[0190] The direction vector projection unit is configured to project the direction vector of the second line equation in the radar coordinate system to the camera coordinate system through the initial pose to be calculated to obtain the direction vector of the third line equation;

[0191] A second objective function construction unit, configured to construct a second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation;

[0192] An initial pose calculation unit, configured to solve the second objective function to obtain a solution corresponding to the initial pose to be calculated, and use it as the initial pose.

[0193] In one embodiment, the above-mentioned second objective function construction unit is configured to construct the following second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation:

[0194]

[0195] Where N represents the number of calibration objects, where 1≤i≤N, M is the number of target lines, where 1≤j≤M, R CL is the initial pose to be calculated, is the calculated initial pose, is the normal vector of the second calibration object plane equation of the i-th calibration object in the radar coordinate system, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the direction vector of the second line equation of the j-th target line of the i-th calibration object in the radar coordinate system, is the direction vector of the first line equation of the j-th target line of the i-th calibration object in the camera coordinate system.

[0196] In one embodiment, the above-mentioned initial pose calculation module 603 includes:

[0197] A parameter to be calculated acquisition unit, configured to acquire the initial pose to be calculated and the position to be calculated between the radar and the camera

[0198] A first point cloud data acquisition unit, configured to acquire the point cloud data of the calibration object in the radar coordinate system;

[0199] A second point cloud data acquisition unit, configured to convert the point cloud data to the camera coordinate system through the pose to be calculated and the position to be calculated between the radar and the camera to obtain the reference point cloud data;

[0200] A target plane constraint equation calculation unit, configured to obtain a target plane constraint equation according to the reference point cloud data and the plane equation in the camera coordinate system;

[0201] A target line constraint equation calculation unit, configured to obtain a target line constraint equation according to reference point cloud data and a line constraint equation in a camera coordinate system;

[0202] An initial position calculation unit, configured to calculate an initial position between a radar and a camera according to a plane constraint and a line constraint.

[0203] In one embodiment, the above-mentioned second point cloud data acquisition unit is configured to convert point cloud data to the camera coordinate system to obtain reference point cloud data according to the following formula through a to-be-calculated attitude and a to-be-calculated position between the radar and the camera:

[0204]

[0205] Wherein, is the point cloud data of the i-th calibration object in the camera coordinate system, that is, the reference point cloud data, and R CL is the to-be-calculated attitude, is the point cloud data of the i-th calibration object in the radar coordinate system, and t CL is the to-be-calculated position.

[0206] In one embodiment, the above-mentioned target plane constraint equation unit includes:

[0207] A plane equation acquisition subunit, configured to acquire a plane equation in the camera coordinate system:

[0208]

[0209] Wherein, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the direction vector of the first line equation of the i-th calibration object in the camera coordinate system;

[0210] A target plane constraint equation acquisition subunit, configured to obtain a target plane constraint equation according to reference point cloud data and a plane equation in the camera coordinate system:

[0211]

[0212] In one embodiment, the above-mentioned target line constraint equation unit includes:

[0213] A line constraint equation acquisition subunit, configured to acquire a line constraint equation in the camera coordinate system:

[0214]

[0215] Wherein, is the direction vector of the first line equation of the j-th target line of the i-th calibration object in the camera coordinate system, is a point on the j-th target line of the i-th calibration object in the radar coordinate system, is a point on the j-th target line of the i-th calibration object in the camera coordinate system;

[0216] The target line constraint equation acquisition subunit is used to obtain the target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system:

[0217]

[0218] where R CL is the attitude to be calculated, and t CL is the position to be calculated.

[0219] In one embodiment, the above-mentioned first objective function construction module 606 is used to construct the following first objective function according to the target plane constraint and target line constraint of the calibration object in the radar coordinate system and the camera coordinate system:

[0220]

[0221] where, R CL is the attitude to be calculated, and t CL is the position to be calculated, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is a point on the calibration object in the radar coordinate system, is the direction vector of the first line equation in the i-th calibration object in the camera coordinate system, is the direction vector of the first line equation of the j-th target line in the i-th calibration object in the camera coordinate system, K ij is the number of point clouds on the j-th target line in the pose of the i-th calibration object, is the k-th point on the j-th target line of the i-th calibration object in the radar coordinate system, is a point on the j-th target line of the i-th calibration object in the camera coordinate system.

[0222] In one embodiment, the above-mentioned result calculation module 605 is used to take the initial attitude and initial position as the initial values of non-linear optimization, and optimize the first objective function through the LM gradient descent strategy to obtain the target attitude and target position between the optimized radar and camera.

[0223] Each module in the above radar and camera calibration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in the processor of a computer device in hardware form or independent of the processor, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0224] In one embodiment, a computer device is provided. The computer device can be a terminal, and its internal structural diagram can be as Figure 7 shown. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a radar and camera calibration method. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the housing of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0225] Those skilled in the art can understand that Figure 7 the structure shown in

[0226] is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0227] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, it implements the steps in each of the above method embodiments.

[0227] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by the processor, it implements the steps in each of the above method embodiments.

[0228] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by the processor, it implements the steps in each of the above method embodiments.

[0229] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., and are not limited thereto. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., and are not limited thereto.

[0230] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0231] The above-described embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A radar and camera calibration method, characterized in that, The method includes: Collecting an image to be processed in the common view area through a camera, and collecting radar data of the common view area through a radar, where the common view area is the common view area of the camera and the radar, and a calibration object is arranged in the common view area; Calculating a first calibration object plane equation in the camera coordinate system and a first line equation of a target line in the calibration object according to the image to be processed and the calibration object in the common view area; Calculating a second calibration object plane equation in the radar coordinate system and a second line equation of the target line in the calibration object according to the radar data; Calculating an initial pose between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation; Calculating an initial position between the radar and the camera according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; Constructing a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; Using the initial pose and the initial position as the initial values of nonlinear optimization, and solving the first objective function to obtain the target pose and the target position of the radar and the camera; The calculating the initial pose between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation includes: Obtaining an initial pose to be calculated between the radar and the camera; Projecting the normal vector of the second calibration object plane equation in the radar coordinate system to obtain the normal vector of a third calibration object plane equation in the camera coordinate system through the initial pose to be calculated; Projecting the direction vector of the second line equation in the radar coordinate system to obtain the direction vector of a third line equation in the camera coordinate system through the initial pose to be calculated; Constructing a second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation; Solving the second objective function to obtain the solution corresponding to the initial pose to be calculated as the initial pose; The calculating the initial position between the radar and the camera according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system includes: Obtaining an initial pose to be calculated and an initial position to be calculated between the radar and the camera; Obtaining the point cloud data of the calibration object in the radar coordinate system; Converting the point cloud data to the camera coordinate system through the initial pose to be calculated and the initial position to be calculated between the radar and the camera to obtain reference point cloud data; Obtaining a target plane constraint equation according to the reference point cloud data and the plane equation in the camera coordinate system; Obtaining a target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system; Calculate the initial position between the radar and the camera according to the plane constraint and the line constraint; Construct a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system, including, Construct the following first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system: Among them, R CL is the pose to be calculated, t CL is the position to be calculated, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the point on the calibration object in the radar coordinate system, is the direction vector of the first line equation in the i-th calibration object in the camera coordinate system, is the direction vector of the first line equation of the j-th target line in the i-th calibration object in the camera coordinate system, K ij is the number of point clouds on the j-th target line in the pose of the i-th calibration object, is the k-th point on the j-th target line of the i-th calibration object in the radar coordinate system, The point on the j-th target line of the i-th calibration object in the camera coordinate system, N is the number of calibration objects, N i is the number of target lines on the i-th calibration object.

2. The method according to claim 1, characterized in that Calculate a first calibration object plane equation in the camera coordinate system and a first line equation of the target line in the calibration object according to the image to be processed and the calibration object in the co-visibility area, including: Identify the corner points and boundaries of the calibration object in the image to be processed; Calculate a first calibration object plane equation in the camera coordinate system using the homography matrix according to the corner points of the calibration object in the image to be processed; Obtain a plane passing through the boundary of the calibration object and the optical center of the camera, and calculate the intersection line of the plane and the calibration object in the image to be processed as the target line; Calculate a first line equation of the target line in the camera coordinate system.

3. The method according to claim 1, wherein Calculate a second calibration object plane equation in the radar coordinate system and a second line equation of the target line in the calibration object according to the radar data, including: Obtain the point cloud data of the calibration object in the radar data, and calculate a second calibration object plane equation in the radar coordinate system according to the point cloud data of the calibration object in the radar data; Calculate a second line equation of the target line in the radar coordinate system according to the point cloud data on the target line of the calibration object in the radar data.

4. The method according to claim 1, wherein Construct a second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation, including, Construct the following second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation: Among them, N represents the number of calibration objects, where 1 ≤ i ≤ N, M is the number of target lines, where 1 ≤ j ≤ M, R CL is the initial pose to be calculated, is the calculated initial pose, is the normal vector of the second calibration object plane equation of the i-th calibration object in the radar coordinate system, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the direction vector of the second line equation of the j-th target line of the i-th calibration object in the radar coordinate system, is the direction vector of the first line equation of the j-th target line of the i-th calibration object in the camera coordinate system.

5. The method according to claim 1, wherein Convert the point cloud data to the camera coordinate system to obtain reference point cloud data through the pose to be calculated and the position to be calculated between the radar and the camera, including, According to the following formula, convert the point cloud data to the camera coordinate system to obtain reference point cloud data through the pose to be calculated and the position to be calculated between the radar and the camera: Among them, is the point cloud data of the i-th calibration object in the camera coordinate system, that is, the reference point cloud data, R CL is the attitude to be calculated, is the point cloud data of the i-th calibration object in the radar coordinate system, t CL is the position to be calculated.

6. The method according to claim 5, wherein Obtain a target plane constraint equation according to the reference point cloud data and the plane equation in the camera coordinate system, including: Obtain the plane equation in the camera coordinate system: wherein, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the direction vector of the first line equation of the i-th calibration object in the camera coordinate system; Obtain a target plane constraint equation according to the reference point cloud data and the plane equation in the camera coordinate system:

7. The method according to claim 5, characterized in that, Obtain a target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system, including, Obtain the line constraint equation in the camera coordinate system: Among them, The direction vector of the first line equation of the j-th target line of the i-th calibration object in the camera coordinate system is a point on the j-th target line of the i-th calibration object in the radar coordinate system is a point on the j-th target line of the i-th calibration object in the camera coordinate system; Obtain a target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system: Among them R CL is the attitude to be calculated, and t CL is the position to be calculated. is the point on the j-th target line of the i-th calibration object in the radar coordinate system, where and are the extrinsic parameters between the radar and i.

8. The method according to claim 1, wherein Using the initial pose and the initial position as the initial values of non-linear optimization, solving the first objective function to obtain the target pose and the target position of the radar and the camera includes: Using the initial pose and the initial position as the initial values of non-linear optimization, optimizing the first objective function through the LM gradient descent strategy to obtain the target pose and the target position between the optimized radar and the camera.

9. A radar and camera calibration device, characterized in that, The device includes: An acquisition module, configured to acquire an image to be processed in the common view area through a camera, and acquire radar data of the common view area through a radar, where the common view area is the common view area of the camera and the radar, and a calibration object is arranged in the common view area; An equation generation module, configured to calculate a first calibration object plane equation in the camera coordinate system and a first line equation of a target line in the calibration object according to the image to be processed and the calibration object in the common view area; calculate a second calibration object plane equation in the radar coordinate system and a second line equation of the target line in the calibration object according to the radar data; An initial pose calculation module, configured to calculate an initial pose between the radar and the camera according to the first calibration object plane equation, the second calibration object plane equation, the first line equation, and the second line equation; calculate an initial position between the radar and the camera according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; A first objective function construction module, configured to construct a first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system; A result calculation module, configured to use the initial pose and the initial position as the initial values of non-linear optimization, and solve the first objective function to obtain the target pose and the target position of the radar and the camera; The initial pose calculation module includes: A to-be-calculated initial pose acquisition unit, configured to acquire a to-be-calculated initial pose between the radar and the camera; A normal vector projection unit, configured to project the normal vector of the second calibration object plane equation in the radar coordinate system to obtain the normal vector of the third calibration object plane equation in the camera coordinate system through the to-be-calculated initial pose; A direction vector projection unit, configured to project the direction vector of the second line equation in the radar coordinate system to obtain the direction vector of the third line equation in the camera coordinate system through the to-be-calculated initial pose; A second objective function construction unit, configured to construct a second objective function according to the normal vector of the third calibration object plane equation, the normal vector of the first calibration object plane equation, the direction vector of the third line equation, and the direction vector of the first line equation; An initial pose calculation unit, configured to solve the second objective function to obtain the solution corresponding to the to-be-calculated initial pose as the initial pose; A to-be-calculated parameter acquisition unit, configured to acquire a to-be-calculated initial pose and a to-be-calculated position between the radar and the camera; A first point cloud data acquisition unit, configured to acquire point cloud data of the calibration object in the radar coordinate system; The second point cloud data acquisition unit is configured to convert the point cloud data to the camera coordinate system to obtain reference point cloud data through the to-be-calculated pose and to-be-calculated position between the radar and the camera; The target plane constraint equation calculation unit is configured to obtain a target plane constraint equation according to the reference point cloud data and the plane equation in the camera coordinate system; The target line constraint equation calculation unit is configured to obtain a target line constraint equation according to the reference point cloud data and the line constraint equation in the camera coordinate system; The initial position calculation unit is configured to calculate the initial position between the radar and the camera according to the plane constraint and the line constraint; The first objective function construction module is configured to construct the following first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system, including, Construct the following first objective function according to the target plane constraint and the target line constraint of the calibration object in the radar coordinate system and the camera coordinate system: Among them, R CL is the attitude to be calculated, t CL is the position to be calculated, is the normal vector of the first calibration object plane equation of the i-th calibration object in the camera coordinate system, is the point on the calibration object in the radar coordinate system, is the direction vector of the first line equation in the i-th calibration object in the camera coordinate system, is the direction vector of the first line equation of the j-th target line in the i-th calibration object in the camera coordinate system, K ij is the number of point clouds on the j-th target line in the pose of the i-th calibration object, is the k-th point on the j-th target line of the i-th calibration object in the radar coordinate system, The point on the j-th target line of the i-th calibration object in the camera coordinate system.

10. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images