Laser radar and camera joint calibration method and device, server, and computer-readable storage medium

Abstract

The present application relates to a laser radar and camera joint calibration method and device, a server, and a computer-readable storage medium, including: acquiring images and laser point cloud data, and mapping the point cloud corresponding to the laser point cloud data to the image to obtain a mapped point cloud according to the camera's intrinsic parameter matrix and the laser radar's initial joint calibration parameters relative to the camera. Calculate the overlapping area between the point cloud target box and the image recognition target box of the same target on the image, the point cloud target box is a target box drawn on the image based on the mapped point cloud, and the image recognition target box is a target box obtained by annotating the image. Adjust the initial joint calibration parameters based on the overlapping area, and output the adjusted joint calibration parameters as the target joint calibration parameters of the laser radar relative to the camera. By performing self-calibration with the laser radar and the camera, there is no need to use calibration objects, thereby avoiding waste of manpower and material resources. Because no calibration objects are required, the reusability of the joint calibration method is also improved.

Description

Technical Field

[0001] The present application relates to the field of camera calibration technology, and in particular to a laser radar and camera joint calibration method and device, a server, and a computer-readable storage medium. Background Art

[0002] With the continuous development of autonomous driving or unmanned driving technology, autonomous driving or unmanned driving will gradually enter people's daily lives and bring convenience to people's lives. Among them, the core problem faced by autonomous driving or unmanned driving technology is that the vehicle's visual perception ability is insufficient and it cannot form a comprehensive perception of the overall road traffic environment, making the vehicle's driving safety not fully guaranteed.

[0003] In recent years, there have been solutions to improve the visual perception capabilities of vehicles by using lidar and cameras together. However, when using traditional methods to calibrate lidar and cameras, they need to be calibrated based on calibration objects, which wastes manpower and material resources and has low reusability. Summary of the invention

[0004] The embodiments of the present application provide a laser radar and camera joint calibration method, device, server, and computer-readable storage medium, which can avoid wasting manpower and material resources and improve the reusability of the calibration process.

[0005] A laser radar and camera joint calibration method, comprising:

[0006] Acquire images and laser point cloud data collected at the same time and in the same scene;

[0007] Mapping the point cloud corresponding to the laser point cloud data onto the image to obtain a mapped point cloud according to the intrinsic parameter matrix of the camera and the initial joint calibration parameters of the laser radar relative to the camera;

[0008] Calculate the overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image, wherein the point cloud target frame is a target frame drawn on the image based on the mapped point cloud, and the image recognition target frame is a target frame obtained by annotating the image;

[0009] The initial joint calibration parameters are adjusted based on the overlapping area until the overlapping area is greater than a preset threshold, and the adjusted joint calibration parameters are output as target joint calibration parameters of the laser radar relative to the camera.

[0010] In one embodiment, adjusting the initial joint calibration parameters based on the overlapping area until the overlapping area is greater than a preset threshold, and outputting the adjusted joint calibration parameters as target joint calibration parameters of the laser radar relative to the camera, includes:

[0011] The initial joint calibration parameters are adjusted based on the overlapping area and the category similarity between the point cloud target frame and the image recognition target frame until the overlapping area is greater than a preset threshold and the category similarity is greater than a preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera.

[0012] In one embodiment, the method further comprises:

[0013] Calculate the category similarity between the point cloud target box and the image recognition target box of the same target on the image.

[0014] In one embodiment, calculating the category similarity between the point cloud target box and the image recognition target box of the same target on the image includes:

[0015] Annotating the laser point cloud data, and acquiring an original point cloud target frame and a first target category corresponding to the original point cloud target frame from the laser point cloud data;

[0016] Annotating the image, and obtaining an image recognition target frame and a second target category corresponding to the image recognition target frame from the image;

[0017] The category similarity between the first target category and the second target category corresponding to the same target is calculated, and the category similarity is used as the category similarity between the point cloud target frame of the target on the image and the image recognition target frame.

[0018] In one embodiment, mapping the point cloud corresponding to the laser point cloud data onto the image to obtain the mapped point cloud according to the intrinsic parameter matrix of the camera and the initial joint calibration parameters of the laser radar relative to the camera includes:

[0019] Calculating the intrinsic parameter matrix of the camera and the distortion coefficient of the camera;

[0020] Obtaining the coordinates of the original point cloud target frame in the world coordinate system;

[0021] Input the intrinsic parameter matrix of the camera, the distortion coefficient of the camera, the initial joint calibration parameters of the laser radar relative to the camera, and the coordinates of the original point cloud target frame in the world coordinate system into the transmission transformation matrix, and calculate the coordinates of the original point cloud target frame in the pixel coordinate system;

[0022] Pixel points at the coordinates of the original point cloud target frame in the pixel coordinate system are extracted from the image as a mapping point cloud.

[0023] In one of the embodiments, before calculating the overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image, the method further includes:

[0024] For each target in the image, obtaining a bounding box of the mapped point cloud on the image as a point cloud target box;

[0025] Obtain a point cloud target frame and an image recognition target frame of the same target on the image.

[0026] In one embodiment, the process of calculating the initial joint calibration parameters of the laser radar relative to the camera includes:

[0027] Get the coordinates of the laser radar in the world coordinate system;

[0028] Convert the coordinates of the camera in the camera coordinate system to the coordinates of the camera in the world coordinate system;

[0029] According to the coordinates of the laser radar in the world coordinate system and the coordinates of the camera in the world coordinate system, the initial joint calibration parameters of the laser radar relative to the camera are calculated.

[0030] In one embodiment, adjusting the initial joint calibration parameters based on the overlapping area until the overlapping area is greater than a preset threshold, and outputting the adjusted joint calibration parameters as target joint calibration parameters of the laser radar relative to the camera, includes:

[0031] A first objective function is constructed based on the overlapping area, the initial joint calibration parameters are adjusted by the first objective function until the value of the first objective function is greater than a preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera.

[0032] A laser radar and camera joint calibration device, comprising:

[0033] Data acquisition module, used to obtain images and laser point cloud data collected at the same time and in the same scene;

[0034] A point cloud mapping module, used for mapping the point cloud corresponding to the laser point cloud data onto the image to obtain a mapping point cloud according to the intrinsic parameter matrix of the camera and the initial joint calibration parameters of the laser radar relative to the camera;

[0035] An overlapping area calculation module, used to calculate the overlapping area between a point cloud target frame and an image recognition target frame of the same target on the image, wherein the point cloud target frame is a target frame drawn on the image based on the mapped point cloud, and the image recognition target frame is a target frame obtained by annotating the image;

[0036] The target joint calibration parameter output module is used to adjust the initial joint calibration parameters based on the overlapping area until the overlapping area is greater than a preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the laser radar relative to the camera.

[0037] A server comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0038] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0039] The above-mentioned laser radar and camera joint calibration method, device, server, and computer-readable storage medium obtain images and laser point cloud data collected at the same time and in the same scene, and map the point cloud corresponding to the laser point cloud data to the image to obtain a mapped point cloud according to the camera's intrinsic parameter matrix and the initial joint calibration parameters of the laser radar relative to the camera. Calculate the overlapping area between the point cloud target box and the image recognition target box of the same target on the image. The point cloud target box is a target box drawn on the image based on the mapped point cloud, and the image recognition target box is a target box obtained by annotating the image. Adjust the initial joint calibration parameters based on the overlapping area until the overlapping area is greater than a preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the laser radar relative to the camera. By self-calibrating the laser radar and the camera, there is no need to use calibration objects, which avoids wasting manpower and material resources. Because no calibration objects are required, the reusability of the joint calibration method is also improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 This is a diagram of an application scenario of a laser radar and camera joint calibration method in one embodiment;

[0042] Figure 2 is a flow chart of a laser radar and camera joint calibration method in one embodiment;

[0043] Figure 3 is a flowchart of a laser radar and camera joint calibration method in another embodiment;

[0044] Figure 4 A flowchart of a method for calculating the category similarity between a point cloud target frame and an image recognition target frame of the same target on an image in one embodiment;

[0045] Figure 5 A flowchart of a method for obtaining a mapping point cloud in one embodiment;

[0046] Figure 6 It is a flowchart of a laser radar and camera joint calibration method in a specific embodiment;

[0047] Figure 7 is a structural block diagram of a laser radar and camera joint calibration device in one embodiment;

[0048] Figure 8 It is a structural block diagram of a laser radar and camera joint calibration device in another embodiment;

[0049] Fig. 9 FIG. 1 is a schematic diagram of the internal structure of a server in an embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] It can be understood that the terms "first", "second", etc. used in the present application can be used in this article to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.

[0052] Figure 1 FIG. 1 is an application scenario diagram of a laser radar and camera joint calibration method in one embodiment. Figure 1As shown, the application environment includes a laser radar 120, a camera 140 and a server 160. The image is acquired by the camera, and the laser point cloud data is acquired by the laser radar at the same time and in the same scene. The server obtains the image and laser point cloud data acquired at the same time and in the same scene; according to the intrinsic parameter matrix of the camera and the initial joint calibration parameters of the laser radar relative to the camera, the point cloud corresponding to the laser point cloud data is mapped to the image to obtain the mapping point cloud; the overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image is calculated, the point cloud target frame is the target frame drawn on the image based on the mapping point cloud, and the image recognition target frame is the target frame obtained by annotating the image; based on the overlapping area, the initial joint calibration parameters are adjusted until the overlapping area is greater than the preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera. Among them, the camera has a great advantage in extracting the characteristics of the target, but cannot accurately obtain the speed and position of the target. The laser radar has a short detection distance and high measurement accuracy, and can accurately obtain the three-dimensional information and speed information of the object, but it is expensive, lacks RGB information, and cannot work normally in bad weather, such as rainy days and foggy days. Therefore, the integration of LiDAR and camera can bring out their respective advantages. Based on the integration, information exchange can be carried out to obtain the surrounding environment information to the maximum extent.

[0053] like Figure 2 As shown, a laser radar and camera joint calibration method is provided, which is applied to a server and includes steps 220 to 280.

[0054] Step 220, acquiring images and laser point cloud data collected at the same time and in the same scene.

[0055] When jointly calibrating the laser radar and the camera, first, adjust the laser radar and the camera to the preset angle, and fix the relative position between the laser radar and the camera. Then, the image is captured by the camera, and the laser point cloud data is collected by the laser radar at the same time and in the same scene. Finally, the server obtains the image corresponding to the same moment and the laser point cloud data corresponding to the image from the image collected by the camera and the laser point cloud data collected by the laser radar to form a pair of data. When jointly calibrating the laser radar and the camera, multiple pairs of such data can be obtained. Among them, the same target corresponds to multiple laser point cloud data.

[0056] Specifically, the system timestamps of the laser radar and the camera are obtained respectively, and the system time difference between the laser radar and the industrial computer, and the system time difference between the camera and the industrial computer are calculated respectively. Then, the laser point cloud data and the data corresponding to the image are collected simultaneously by the same industrial computer. Based on the system time difference between the laser radar and the industrial computer, and the system time difference between the camera and the industrial computer, the system timestamps of the laser point cloud data and the data corresponding to the image are converted to the time axis of the industrial computer, so that the laser radar data and the data corresponding to the image with time synchronization (at the same moment) can be obtained.

[0057] Step 240 , based on the camera's intrinsic parameter matrix and the initial joint calibration parameters of the laser radar relative to the camera, the point cloud corresponding to the laser point cloud data is mapped onto the image to obtain a mapped point cloud.

[0058] Specifically, the camera's intrinsic parameter matrix Among them, (c x ,c y ) represents the coordinates of the center point of the image, f x , f y is the focal length in pixels.

[0059] In addition, the laser radar is originally located in the world coordinate system, while the camera is located in the camera coordinate system, and the image captured by the camera is located in the pixel coordinate system. The camera coordinate system can be converted to and from the world coordinate system, and the camera coordinate system can be converted to and from the pixel coordinate system. Therefore, when the server calculates the initial joint calibration parameters of the laser radar relative to the camera, it can use the world coordinate system as a bridge to calculate the initial joint calibration parameters of the laser radar relative to the camera based on the coordinates of the laser radar in the world coordinate system and the coordinates of the camera in the world coordinate system. Among them, the joint calibration parameters can also be called the external parameter matrix.

[0060] Then, according to the camera's intrinsic parameter matrix, the camera's distortion coefficient, and the initial extrinsic parameter matrix of the lidar relative to the camera, the point cloud corresponding to the laser point cloud data is mapped to the image through the transmission transformation matrix to obtain the mapping point cloud, thus establishing the corresponding relationship between the laser point cloud and the pixels on the image.

[0061] Step 260, calculating the overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image, where the point cloud target frame is a target frame drawn on the image based on the mapped point cloud, and the image recognition target frame is a target frame obtained by performing image recognition on the image.

[0062] After the server maps the point cloud corresponding to the laser point cloud data onto the image to obtain the mapped point cloud, it obtains the point cloud target frame and the image recognition target frame of the same target on the image. The overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image is calculated, and the accuracy of the initial external parameter matrix of the laser radar relative to the camera can be verified based on the overlapping area. Because the larger the overlapping area, the higher the accuracy of the spatial alignment of the point cloud corresponding to the laser point cloud data (hereinafter referred to as the laser point cloud) and the image taken by the camera.

[0063] Step 280, adjusting the initial joint calibration parameters based on the overlapping area until the overlapping area meets a preset threshold, and outputting the adjusted joint calibration parameters as target joint calibration parameters of the laser radar relative to the camera.

[0064] Because the larger the overlapping area, the higher the accuracy of spatial alignment between the laser point cloud data and the image. Therefore, the server can adjust the initial extrinsic parameter matrix based on the overlapping area until the overlapping area meets the preset threshold. The preset threshold can be set in advance according to the actual situation. After the overlapping area meets the preset threshold, the adjusted extrinsic parameter matrix is ​​output as the target extrinsic parameter matrix of the laser radar relative to the camera, completing the joint calibration process of the laser radar and the camera.

[0065] In an embodiment of the present application, the server maps the point cloud corresponding to the laser point cloud data onto the image to obtain a mapped point cloud. Then calculate the overlapping area between the point cloud target box and the image recognition target box of the same target on the image, adjust the initial joint calibration parameters based on the overlapping area until the overlapping area meets the preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the laser radar relative to the camera. Because the larger the overlapping area, the higher the accuracy of the spatial alignment of the laser point cloud data and the image. Therefore, the server can adjust the initial external parameter matrix based on the overlapping area until the overlapping area meets the preset threshold. Thereby, self-calibration is achieved through the laser radar and the camera, without the need to use calibration objects, avoiding waste of manpower and material resources, and because the lack of the need to use calibration objects also improves the reusability of the joint calibration method.

[0066] In one embodiment, Figure 3 As shown, a laser radar and camera joint calibration method is provided, which adjusts the initial joint calibration parameters based on the overlapping area until the overlapping area is greater than a preset threshold, and outputs the adjusted joint calibration parameters as target joint calibration parameters of the laser radar relative to the camera, including:

[0067] Step 290, adjust the initial joint calibration parameters based on the overlapping area and the category similarity between the point cloud target frame and the image recognition target frame until the overlapping area is greater than a preset threshold and the category similarity is greater than a preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the lidar relative to the camera.

[0068] Specifically, first, after calculating the overlapping area, the ratio of the overlapping area to the sum of the areas of the point cloud target frame and the image recognition target frame is calculated, and the first objective function is constructed based on the ratios of each target. L The target category C of the target box with image recognition C The category similarity between them is used to construct the second objective function. Among them, the constructed first objective function L R As shown below:

[0069]

[0070] The second objective function L constructed C As shown below:

[0071]

[0072] in,

[0073] In formula (1-3), if the ratio Rj is 0, or the ratio Rj is greater than 0 and the target category C of the point cloud target box L The target category C of the target box with image recognition C The similarity of C is less than or equal to the preset threshold. L With C C Not the same (C L ≠C C ), then lc=1. If the ratio Rj is greater than 0, and the target category C of the point cloud target box L The target category C of the target box with image recognition C The similarity of C is greater than the preset threshold, which means that L With C C Same (C L =C C ), then lc=0.

[0074] The final objective function calculation formula is: L = L C +L R (1-4)

[0075] Finally, the initial extrinsic parameter matrix is ​​continuously adjusted through the objective function L until the value of the objective function L meets the preset threshold, and the adjusted extrinsic parameter matrix is ​​output as the target extrinsic parameter matrix of the lidar relative to the camera. That is, the initial extrinsic parameter matrix is ​​continuously adjusted through the objective function L until the overlapping area is greater than the preset threshold and the category similarity is greater than the preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the lidar relative to the camera.

[0076] In the embodiment of the present application, because there is a strong correlation between the point cloud target frame and the image recognition target frame of the same target on the image, the objective function is constructed by the two dimensions of the overlapping area and category similarity between the point cloud target frame and the image recognition target frame of the same target on the image. Then, the initial external parameter matrix is ​​continuously adjusted by the objective function L until the overlapping area is greater than the preset threshold and the category similarity is greater than the preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera. In this way, the accuracy of the joint calibration parameters of the laser radar and the camera obtained according to the objective function is high.

[0077] In one embodiment, a laser radar and camera joint calibration method is provided, further comprising:

[0078] Calculate the category similarity between the point cloud target box and the image recognition target box of the same target on the image.

[0079] Specifically, first, the laser point cloud data and the image are annotated respectively, and the original point cloud target frame and the first target category corresponding to the original point cloud target frame are obtained from the laser point cloud data, and the image recognition target frame and the second target category corresponding to the image recognition target frame are obtained from the image. Then, the category similarity between the first target category and the second target category corresponding to the same target can be calculated, and the category similarity is used as the category similarity between the point cloud target frame and the image recognition target frame of the target on the image.

[0080] In an embodiment of the present application, the category similarity between the point cloud target box and the image recognition target box of the same target on the image is calculated to facilitate the subsequent adjustment of the joint calibration parameters from the dimension of category and improve the accuracy of the joint calibration.

[0081] In one embodiment, Figure 4 As shown, the category similarity between the point cloud target box and the image recognition target box of the same target on the image is calculated, including:

[0082] Step 420, annotating the laser point cloud data, obtaining an original point cloud target frame and a first target category corresponding to the original point cloud target frame from the laser point cloud data;

[0083] Step 440, annotating the image, obtaining an image recognition target frame and a second target category corresponding to the image recognition target frame from the image;

[0084] Step 460, calculating the category similarity between the first target category and the second target category corresponding to the same target, and using the category similarity as the category similarity between the point cloud target frame of the target on the image and the image recognition target frame.

[0085] Specifically, when the laser point cloud data is annotated, the laser point cloud data can be annotated manually or by computer, and the original point cloud target frame corresponding to the target is annotated from the laser point cloud data. Then, the target category in the original point cloud target frame is annotated to obtain the first target category. The original point cloud target frame annotated here is a three-dimensional target frame, and the labeling of the target frame can generally be completed by obtaining the coordinates of each vertex (8 vertices) of the target frame.

[0086] When annotating an image, an image recognition target frame can be obtained from the image through an image recognition algorithm, and the target category in the image recognition target frame can be annotated to obtain a second target category. For example, the target categories include vehicles, pedestrians, curbs, green plants, etc., of course, this application is not limited to this. Here, the image recognition target frame is a two-dimensional target frame, and the labeling of the target frame can generally be completed by obtaining the coordinates of each vertex (4 vertices) of the target frame.

[0087] Then, the category similarity between the first target category and the second target category corresponding to the same target is calculated, and the category similarity is used as the category similarity between the point cloud target box of the target on the image and the image recognition target box.

[0088] In the embodiment of the present application, the laser point cloud data and the image are annotated respectively, the original point cloud target frame and the first target category corresponding to the original point cloud target frame are obtained from the laser point cloud data, and the image recognition target frame and the second target category corresponding to the image recognition target frame are obtained from the image. Because the amount of laser point cloud data is too large, useful data is extracted from a large amount of laser point cloud data by obtaining the original point cloud target frame, which reduces the amount of data and improves the calculation efficiency. At the same time, the first target category corresponding to the original point cloud target frame and the second target category corresponding to the image recognition target frame are obtained, and the category similarity between the first target category and the second target category corresponding to the same target is calculated, and the category similarity is used as the category similarity between the point cloud target frame and the image recognition target frame of the target on the image. It is convenient to adjust the joint calibration parameters from the category dimension in the future to improve the accuracy of the joint calibration.

[0089] In one embodiment, Figure 5 As shown, step 240, according to the intrinsic parameter matrix of the camera and the initial joint calibration parameters of the laser radar relative to the camera, maps the point cloud corresponding to the laser point cloud data onto the image to obtain a mapped point cloud, including:

[0090] Step 242, calculating the camera's intrinsic parameter matrix and the camera's distortion coefficient.

[0091] The camera’s intrinsic parameter matrix and camera distortion coefficients are obtained by camera calibration. Among them, (cx ,c y ) represents the coordinates of the center point of the image, f x , f y is the focal length in pixels.

[0092] Among them, the distortion coefficient of the camera can be divided into radial distortion and tangential distortion.

[0093] Step 244, obtaining the coordinates of the original point cloud target frame in the world coordinate system.

[0094] For LiDAR, since LiDAR originally uses the world coordinate system, the coordinates of the original point cloud target frame in the world coordinate system are directly obtained. The world coordinate system consists of Xw, Yw, and Zw.

[0095] Step 246, the camera's intrinsic parameter matrix, the camera's distortion coefficient, the initial joint calibration parameters of the lidar relative to the camera, and the coordinates of the original point cloud target frame in the world coordinate system are input into the transmission transformation matrix to calculate the coordinates of the original point cloud target frame in the pixel coordinate system.

[0096] After the above calculations, we can obtain the camera's intrinsic parameter matrix A, the camera's distortion coefficient K, the initial extrinsic parameter matrix [RT] of the laser radar relative to the camera, and the coordinates of each vertex of the original point cloud target frame in the world coordinate system (X W ,Y W ,Z W ). The joint calibration parameters can also be called the external parameter matrix. Among them, the transmission transformation matrix is:

[0097]

[0098] in,

[0099] The camera's intrinsic parameter matrix A, the camera's distortion coefficient K, the initial extrinsic parameter matrix [RT] of the laser radar relative to the camera, and the coordinates of each vertex of the original point cloud target frame in the world coordinate system (X W ,Y W ,Z W ) is input into the transmission transformation matrix, and the coordinates (μ, ν) of each vertex of the original point cloud target box in the pixel coordinate system can be calculated.

[0100] The process of calculating the coordinates (μ, ν) of each vertex of the original point cloud target box in the pixel coordinate system is as follows:

[0101]

[0102] x′=x / zy′=y / z (1-8)

[0103]

[0104]

[0105] Among them, r 2 = x′ 2 +y′ 2

[0106] u=f x *x″+c x v=f y *y″+c y

[0107] (1-11)

[0108] Among them, k1, k2, k3, k4, k5, k6 represent radial distortion coefficients, and p1, p2 represent tangential distortion coefficients.

[0109] Step 248 , extracting pixel points of the original point cloud target frame at the coordinates in the pixel coordinate system from the image as the mapping point cloud.

[0110] After calculating the coordinates (μ, ν) of each vertex of the original point cloud target box in the pixel coordinate system, the vertices of the original point cloud target box are mapped to the image to obtain the mapping point cloud. Specifically, the pixel points at the coordinates (μ, ν) of the original point cloud target box in the pixel coordinate system are extracted from the image as the mapping point cloud.

[0111] In an embodiment of the present application, first, the coordinates of the original point cloud target frame in the world coordinate system are obtained. Then, the camera's intrinsic parameter matrix, the camera's distortion coefficient, the initial joint calibration parameters of the laser radar relative to the camera, and the coordinates of the original point cloud target frame in the world coordinate system are input into the transmission transformation matrix to calculate the coordinates of the original point cloud target frame in the pixel coordinate system. The pixel points at the coordinates of the original point cloud target frame in the pixel coordinate system are extracted from the image as a mapping point cloud. The three-dimensional original point cloud target frame is converted to a mapping point cloud on a two-dimensional image, and then the laser point cloud is spatially matched with the image taken by the camera, thereby facilitating the subsequent joint calibration of the laser radar and the camera. Moreover, the transmission transformation matrix can be used to calculate the image data after distortion correction, and can also be used to calculate the image data that has not been corrected for distortion, which has a wider applicability.

[0112] In one embodiment, before calculating the overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image, the method further includes:

[0113] For each target in the image, obtain the bounding box of the mapped point cloud on the image as the point cloud target box;

[0114] Get the point cloud target box and image recognition target box of the same target on the image.

[0115] Specifically, the three-dimensional original point cloud target box has 8 vertices, so by mapping the 8 vertices of the original point cloud target box to the pixel coordinate system, 8 mapping point clouds are obtained. The original point cloud target box of each target is mapped to the image to obtain the mapping point cloud of the target. Then, for each target in the image, the bounding box of the mapping point cloud is obtained on the image as a two-dimensional point cloud target box. The bounding box here can be a rectangle, a circle, etc., which is not limited in this application. For example, for the 8 mapping point clouds of target A, calculate the minimum bounding rectangle of these 8 mapping point clouds.

[0116] The image is annotated in advance, and the image recognition target frame is obtained from the image. Similarly, the shape of the image recognition target frame here can be a circular frame or a rectangular frame, and this application does not limit the shape of the target frame. Exemplarily, the point cloud target frame and the image recognition target frame are preferably target frames of the same shape for easy comparison.

[0117] Because the image captured by the camera contains multiple targets, it is necessary to obtain the point cloud target frame and image recognition target frame corresponding to the same target from the image. That is, the point cloud target frame and image recognition target frame on the image are paired to obtain the point cloud target frame and image recognition target frame corresponding to the same target.

[0118] In the embodiment of the present application, for each target in the image, the bounding box of the mapped point cloud is obtained on the image as the point cloud target box. The point cloud target box and the image recognition target box of the same target on the image are obtained. The two-dimensional point cloud target box corresponding to the laser point cloud of the same target is spatially matched with the image recognition target box of the same target on the image taken by the camera, so as to facilitate the subsequent joint calibration of the laser radar and the camera according to the degree of target box matching.

[0119] In one embodiment, the calculation process of the initial joint calibration parameters of the laser radar relative to the camera includes:

[0120] Get the coordinates of the laser radar in the world coordinate system;

[0121] Convert the coordinates of the camera in the camera coordinate system to the coordinates of the camera in the world coordinate system;

[0122] According to the coordinates of the lidar in the world coordinate system and the coordinates of the camera in the world coordinate system, the initial joint calibration parameters of the lidar relative to the camera are calculated.

[0123] Specifically, for laser radar, because laser radar originally uses the world coordinate system, the coordinates of the laser point cloud in the world coordinate system are directly obtained. Among them, the world coordinate system is composed of Xw, Yw, and Zw.

[0124] The camera's own coordinate system is the camera coordinate system. The camera's coordinates in the camera coordinate system are converted to the camera's coordinates in the world coordinate system through the conversion formula between the camera coordinate system and the world coordinate system. The camera coordinate system can be expressed as (x, y, z).

[0125] After the calculations in the first two steps above, the coordinates of the LiDAR in the world coordinate system and the coordinates of the camera in the world coordinate system are obtained. Then, the initial joint calibration parameters (initial external parameter matrix) of the LiDAR relative to the camera can be calculated based on the world coordinate system as the intermediate coordinate system. The joint calibration parameters can also be called the external parameter matrix. [RT] includes the rotation matrix R 3×3 , translation vector T 3×1 , specifically:

[0126] In the embodiment of the present application, first, the coordinates of the laser radar in the world coordinate system are obtained; second, the coordinates of the camera in the camera coordinate system are converted to the coordinates of the camera in the world coordinate system; finally, the initial extrinsic parameter matrix of the laser radar relative to the camera can be calculated based on the world coordinate system as the intermediate coordinate system. With the help of the world coordinate system as the intermediate coordinate system, the initial extrinsic parameter matrix of the laser radar relative to the camera can be accurately calculated.

[0127] In one embodiment, adjusting the initial joint calibration parameters based on the overlap area until the overlap area is greater than a preset threshold, and outputting the adjusted joint calibration parameters as target joint calibration parameters of the laser radar relative to the camera, includes:

[0128] A first objective function is constructed based on the overlapping area, and the initial joint calibration parameters are adjusted by the first objective function until the value of the first objective function is greater than a preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the lidar relative to the camera.

[0129] Specifically, the objective function is also called a loss function or a cost function. After mapping each vertex of the original point cloud target frame to the image to obtain the mapped point cloud, the server obtains the point cloud target frame and the image recognition target frame of the same target on the image. The overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image is calculated, and then the objective function is constructed based on the overlapping area. The initial extrinsic parameter matrix is ​​continuously adjusted through the objective function until the value of the objective function meets the preset threshold, and the adjusted extrinsic parameter matrix is ​​output as the target extrinsic parameter matrix of the lidar relative to the camera.

[0130] In the embodiment of the present application, the larger the overlapping area, the higher the accuracy of the spatial alignment of the original point cloud target frame and the image recognition target frame on the image. The objective function constructed by the overlapping area can verify the accuracy of the initial joint calibration parameters of the laser radar relative to the camera, and the initial joint calibration parameters of the laser radar relative to the camera are continuously optimized until the value of the objective function meets the preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera. Self-calibration is achieved through the laser radar and the camera, without the need for calibration objects, thus avoiding waste of manpower and material resources. Ultimately, the efficiency of the calibration process and the reusability of the joint calibration method are improved.

[0131] In one embodiment, constructing an objective function based on the overlap area includes:

[0132] Calculate the ratio of the overlapping area to the sum of the areas of the point cloud target box and the image recognition target box;

[0133] The objective function is constructed based on the ratio of the overlapping area corresponding to each target to the sum of the areas of the image recognition target box.

[0134] Specifically, for the same target, the point cloud target box S is calculated on the image L And the image recognition target box S C The overlapping area S OA , then, calculate the ratio R of the overlapping area to the sum of the areas of the point cloud target frame and the image recognition target frame corresponding to the target. For example, assuming that 10 targets are selected from the image, then for each target, calculate a ratio R of the overlapping area to the sum of the areas of the point cloud target frame and the image recognition target frame corresponding to the target (including R1-R 10 The formula for calculating the ratio R is as follows:

[0135]

[0136] After the ratio R is calculated, the objective function can be constructed based on the ratio R of each target. For example, the first objective function constructed is as follows:

[0137]

[0138] In the embodiment of the present application, for each frame of the image, the ratio of the overlapping area corresponding to each target on the image to the sum of the areas of the point cloud target frame and the image recognition target frame is calculated, and then the first objective function is constructed based on the ratio of each target. In this way, the sample data for constructing the first objective function is relatively rich, which improves the accuracy of the constructed first objective function to a certain extent.

[0139] In one embodiment, adjusting the initial joint calibration parameters by the objective function until the value of the objective function meets a preset threshold, and outputting the adjusted joint calibration parameters as target joint calibration parameters of the laser radar relative to the camera, includes:

[0140] The gradient descent algorithm is used to calculate the value of the objective function;

[0141] The initial joint calibration parameters are adjusted according to the value of the objective function until the value of the objective function meets a preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the lidar relative to the camera.

[0142] Specifically, the final objective function is shown in the above formula (1-13), and the gradient descent algorithm can be used to calculate the value of the objective function. Among them, the gradient descent algorithm can be applied to linear regression or nonlinear regression problems, and the gradient descent algorithm is used to find the minimum value of the objective function through iteration, or converge to the minimum value.

[0143] Gradient descent is a type of iterative method that can be used to solve least squares problems. Gradient descent is one of the most commonly used methods for solving the parameters of a neural network. When solving the minimum value of the loss function, the gradient descent method can be used to iterate step by step to obtain the minimized loss function and the values ​​of each parameter, that is, the loss function is optimized as recorded in step 501 until the value of the loss function is less than the preset threshold. Adam is a first-order optimization algorithm that can replace the traditional stochastic gradient descent process. It can iteratively update the weights of the neural network based on training data. Adam can design independent adaptive learning rates for different parameters. The specific formula of Adam is as follows:

[0144]

[0145]

[0146] Among them, t represents the number of times, is m t Correction, v t correction, β1 and β2 are constants that control the exponential decay, m t is the exponential moving mean of the gradient, through the gradient G k The first moment of v is obtained. t is the squared gradient, through the gradient G k The second-order moment of is obtained, and α and ∈ are coefficients.

[0147] Specifically, after the value of the objective function is calculated using the gradient descent algorithm, because a preset threshold is set for the value of the objective function in advance, the initial extrinsic parameter matrix is ​​adjusted according to the value of the objective function until the value of the objective function meets the preset threshold, and the adjusted extrinsic parameter matrix is ​​output as the target extrinsic parameter matrix of the laser radar relative to the camera. Among them, the joint calibration parameters can also be referred to as the extrinsic parameter matrix. Of course, in this application, any one of the gradient descent algorithm and the adaptive learning rate algorithm can also be used to calculate the value of the objective function, or other optimization algorithms can be used to calculate the value of the objective function, and this application does not limit this.

[0148] In the embodiment of the present application, because there is a strong correlation between the point cloud target frame and the image recognition target frame of the same target on the image, the objective function is constructed through the two dimensions of the overlapping area and category similarity between the point cloud target frame and the image recognition target frame of the same target on the image. Then the initial external parameter matrix is ​​continuously adjusted by the objective function L until the overlapping area is greater than the preset threshold and the category similarity is greater than the preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera. In this way, the accuracy of the joint calibration parameters of the laser radar and the camera obtained according to the objective function is high. At the same time, self-calibration is achieved through the laser radar and the camera, without the need to use calibration objects, thus avoiding waste of manpower and material resources.

[0149] In one embodiment, a laser radar and camera joint calibration method is provided, which also includes: adjusting the laser radar and the camera to a preset angle, and fixing the relative position between the laser radar and the camera.

[0150] In an embodiment of the present application, before calibrating the laser radar and the camera, the laser radar and the camera are adjusted to a preset angle, and the relative position between the laser radar and the camera is fixed. This facilitates the subsequent conversion between the spherical coordinate system of the laser radar and the camera coordinate system where the camera is located through the world coordinate system. And the preset angle refers to the angle corresponding to when the laser radar and the camera cover a large number of overlapping areas together. For example, the laser radar is installed horizontally, and the camera is adjusted to a downward tilt angle of 5°-15° relative to the vertical direction. Of course, this application is not limited to this.

[0151] In a specific embodiment, Figure 6 As shown, a laser radar and camera joint calibration method is provided, comprising:

[0152] Step 602, adjusting the laser radar and the camera to a suitable angle, and assembling a fusion kit of the laser radar and the camera;

[0153] Step 604, calculating the initial extrinsic parameter matrix of the laser radar relative to the camera;

[0154] Step 606, acquiring images collected at the same time and in the same scene and laser point clouds corresponding to the images;

[0155] Step 608, annotating the laser point cloud, obtaining an original point cloud target frame and a first target category corresponding to the original point cloud target frame from the laser point cloud;

[0156] Step 610, annotating the image, obtaining an image recognition target frame and a second target category corresponding to the image recognition target frame from the image;

[0157] Step 612, mapping the original point cloud target frame onto the image to obtain a mapped point cloud according to the transmission transformation matrix;

[0158] Step 614, obtaining the bounding box of the mapped point cloud on the image as a point cloud target box;

[0159] Step 616, calculating the overlapping area and category similarity between the point cloud target frame and the image recognition target frame of the same target on the image;

[0160] Step 618, constructing a first objective function based on the overlap area;

[0161] Step 620, constructing a second objective function based on category similarity;

[0162] Step 622, combining the first objective function with the second objective function to obtain a final objective function;

[0163] Step 624, using a gradient descent algorithm to optimize the final objective function, outputting the optimal joint calibration parameters as the target joint calibration parameters of the laser radar relative to the camera.

[0164] In the embodiment of the present application, because there is a strong correlation between the point cloud target frame and the image recognition target frame of the same target on the image, the objective function is constructed by the two dimensions of the overlapping area and category similarity between the point cloud target frame and the image recognition target frame of the same target on the image. Then, the initial external parameter matrix is ​​continuously adjusted by the objective function until the overlapping area is greater than the preset threshold and the category similarity is greater than the preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera. In this way, the accuracy of the joint calibration parameters of the laser radar and the camera obtained according to the objective function is high. At the same time, self-calibration is achieved through the laser radar and the camera, without the need to use calibration objects, thus avoiding waste of manpower and material resources.

[0165] In one embodiment, Figure 7 As shown, a laser radar and camera joint calibration device 700 is provided, comprising:

[0166] The data acquisition module 720 is used to acquire images and laser point cloud data collected at the same time and in the same scene;

[0167] The point cloud mapping module 740 is used to map the point cloud corresponding to the laser point cloud data onto the image to obtain a mapped point cloud according to the camera's intrinsic parameter matrix and the initial joint calibration parameters of the laser radar relative to the camera;

[0168] The overlapping area calculation module 760 is used to calculate the overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image, where the point cloud target frame is a target frame drawn on the image based on the mapped point cloud, and the image recognition target frame is a target frame obtained by annotating the image;

[0169] The target joint calibration parameter output module 780 is used to adjust the initial joint calibration parameters based on the overlapping area until the overlapping area is greater than a preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the laser radar relative to the camera.

[0170] In one embodiment, the target joint calibration parameter output module 780 is also used to adjust the initial joint calibration parameters based on the overlapping area and the category similarity between the point cloud target box and the image recognition target box until the overlapping area is greater than a preset threshold and the category similarity is greater than a preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the lidar relative to the camera.

[0171] In one embodiment, Figure 8 As shown, a laser radar and camera joint calibration device 700 is provided, which also includes:

[0172] The category similarity calculation module 790 is used to calculate the category similarity between the point cloud target frame and the image recognition target frame of the same target on the image.

[0173] In one embodiment, the category similarity calculation module 790 includes:

[0174] A first target category labeling unit is used to label the laser point cloud data, and obtain an original point cloud target frame and a first target category corresponding to the original point cloud target frame from the laser point cloud data;

[0175] A second target category labeling unit, used to label the image, obtain an image recognition target frame and a second target category corresponding to the image recognition target frame from the image;

[0176] The category similarity generating unit is used to calculate the category similarity between the first target category and the second target category corresponding to the same target, and use the category similarity as the category similarity between the point cloud target frame of the target on the image and the image recognition target frame.

[0177] In one embodiment, the point cloud mapping module 740 includes:

[0178] The camera's intrinsic parameter matrix and distortion coefficient calculation unit is used to calculate the camera's intrinsic parameter matrix and the camera's distortion coefficient;

[0179] A coordinate acquisition unit, used to acquire the coordinates of the original point cloud target frame in the world coordinate system;

[0180] A transmission transformation unit is used to input the camera's intrinsic parameter matrix, the camera's distortion coefficient, the initial joint calibration parameters of the laser radar relative to the camera, and the coordinates of the original point cloud target frame in the world coordinate system into the transmission transformation matrix, and calculate the coordinates of the original point cloud target frame in the pixel coordinate system;

[0181] The mapping point cloud acquisition unit is used to extract pixel points at the coordinates of the original point cloud target frame in the pixel coordinate system from the image as the mapping point cloud.

[0182] In one embodiment, a laser radar and camera joint calibration device 700 is provided, which also includes: a target frame determination module, which is used to obtain an external frame of the mapped point cloud on the image as a point cloud target frame for each target in the image; and obtain a point cloud target frame and an image recognition target frame of the same target on the image.

[0183] In one embodiment, an initial joint calibration parameter calculation module is used to obtain the coordinates of the laser radar in the world coordinate system; convert the coordinates of the camera in the camera coordinate system into the coordinates of the camera in the world coordinate system; and calculate the initial joint calibration parameters of the laser radar relative to the camera based on the coordinates of the laser radar in the world coordinate system and the coordinates of the camera in the world coordinate system.

[0184] In one embodiment, the target joint calibration parameter output module 780 is also used to construct a first objective function based on the overlapping area, adjust the initial joint calibration parameters through the first objective function until the value of the first objective function is greater than a preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the lidar relative to the camera.

[0185] In one embodiment, the target joint calibration parameter output module 780 is also used to calculate the value of the objective function using a gradient descent algorithm; adjust the initial joint calibration parameters according to the value of the objective function until the value of the objective function meets a preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the lidar relative to the camera.

[0186] In one embodiment, a laser radar and camera joint calibration device 700 is provided, which also includes: a laser radar and a camera fixing module, which is used to adjust the laser radar and the camera to a preset angle and fix the relative position between the laser radar and the camera.

[0187] The division of each module in the above-mentioned laser radar and camera joint calibration device is only for illustration. In other embodiments, the laser radar and camera joint calibration device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned laser radar and camera joint calibration device.

[0188] Fig. 9 FIG. 1 is a schematic diagram of the internal structure of a server in an embodiment. Fig. 9 As shown, the server includes a processor and a memory connected via a system bus. The processor is used to provide computing and control capabilities to support the operation of the entire server. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement a laser radar and camera joint calibration method provided in each of the following embodiments. The internal memory provides a cache operating environment for the operating system computer program in the non-volatile storage medium. The server can be a mobile phone, a tablet computer, a personal digital assistant, a wearable device, etc.

[0189] The implementation of each module in the laser radar and camera joint calibration device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on a terminal or a server. The program modules constituted by the computer program can be stored in the memory of the terminal or the server. When the computer program is executed by the processor, the steps of the method described in the embodiment of the present application are implemented.

[0190] The embodiment of the present application also provides a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the laser radar and camera joint calibration method.

[0191] A computer program product comprising instructions, when running on a computer, enables the computer to perform a laser radar and camera joint calibration method.

[0192] Any reference to memory, storage, database or other media used in the embodiments of the present application may include non-volatile and / or volatile memory. Suitable non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0193] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A laser radar and camera joint calibration method, characterized in that: include: Acquire images and laser point cloud data collected at the same time and in the same scene; Mapping the point cloud corresponding to the laser point cloud data onto the image to obtain a mapped point cloud according to the intrinsic parameter matrix of the camera and the initial joint calibration parameters of the laser radar relative to the camera; Calculate the overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image, wherein the point cloud target frame is a target frame drawn on the image based on the mapped point cloud, and the image recognition target frame is a target frame obtained by annotating the image; The initial joint calibration parameters are adjusted based on the overlapping area and the category similarity between the point cloud target frame and the image recognition target frame until the overlapping area is greater than a preset threshold and the category similarity is greater than a preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera.

2. The method according to claim 1, characterized in that The method further comprises: Calculate the category similarity between the point cloud target box and the image recognition target box of the same target on the image.

3. The method according to claim 2, characterized in that The calculating the category similarity between the point cloud target frame and the image recognition target frame of the same target on the image includes: Annotating the laser point cloud data, and acquiring an original point cloud target frame and a first target category corresponding to the original point cloud target frame from the laser point cloud data; Annotating the image, and obtaining an image recognition target frame and a second target category corresponding to the image recognition target frame from the image; The category similarity between the first target category and the second target category corresponding to the same target is calculated, and the category similarity is used as the category similarity between the point cloud target frame of the target on the image and the image recognition target frame.

4. The method according to claim 3, characterized in that The step of mapping the point cloud corresponding to the laser point cloud data onto the image to obtain a mapped point cloud according to the intrinsic parameter matrix of the camera and the initial joint calibration parameters of the laser radar relative to the camera includes: Calculating the intrinsic parameter matrix of the camera and the distortion coefficient of the camera; Obtaining the coordinates of the original point cloud target frame in the world coordinate system; Input the intrinsic parameter matrix of the camera, the distortion coefficient of the camera, the initial joint calibration parameters of the laser radar relative to the camera, and the coordinates of the original point cloud target frame in the world coordinate system into the transmission transformation matrix, and calculate the coordinates of the original point cloud target frame in the pixel coordinate system; Pixel points at the coordinates of the original point cloud target frame in the pixel coordinate system are extracted from the image as a mapping point cloud.

5. The method according to claim 4, characterized in that Before calculating the overlapping area between the point cloud target frame and the image recognition target frame of the same target on the image, the method further includes: For each target in the image, obtaining a bounding box of the mapped point cloud on the image as a point cloud target box; Obtain a point cloud target frame and an image recognition target frame of the same target on the image.

6. The method according to any one of claims 1 to 5, characterized in that The calculation process of the initial joint calibration parameters of the laser radar relative to the camera includes: Get the coordinates of the laser radar in the world coordinate system; Convert the coordinates of the camera in the camera coordinate system to the coordinates of the camera in the world coordinate system; According to the coordinates of the laser radar in the world coordinate system and the coordinates of the camera in the world coordinate system, the initial joint calibration parameters of the laser radar relative to the camera are calculated.

7. The method according to claim 6, characterized in that The adjusting the initial joint calibration parameters based on the overlapping area until the overlapping area is greater than a preset threshold, and outputting the adjusted joint calibration parameters as target joint calibration parameters of the laser radar relative to the camera, comprises: A first objective function is constructed based on the overlapping area, the initial joint calibration parameters are adjusted by the first objective function until the value of the first objective function is greater than a preset threshold, and the adjusted joint calibration parameters are output as the target joint calibration parameters of the laser radar relative to the camera.

8. A laser radar and camera joint calibration device, characterized in that: include: Data acquisition module, used to obtain images and laser point cloud data collected at the same time and in the same scene; A point cloud mapping module, used for mapping the point cloud corresponding to the laser point cloud data onto the image to obtain a mapping point cloud according to the intrinsic parameter matrix of the camera and the initial joint calibration parameters of the laser radar relative to the camera; An overlapping area calculation module, used to calculate the overlapping area between a point cloud target frame and an image recognition target frame of the same target on the image, wherein the point cloud target frame is a target frame drawn on the image based on the mapped point cloud, and the image recognition target frame is a target frame obtained by annotating the image; The target joint calibration parameter output module is used to adjust the initial joint calibration parameters based on the overlapping area and the category similarity between the point cloud target frame and the image recognition target frame until the overlapping area is greater than a preset threshold and the category similarity is greater than a preset threshold, and output the adjusted joint calibration parameters as the target joint calibration parameters of the laser radar relative to the camera.

9. A server comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the computer program is executed by the processor, the processor executes the steps of the laser radar and camera joint calibration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the laser radar and camera joint calibration method as described in any one of claims 1 to 7 are implemented.

Images