A dynamic calibration method, device and storage medium for joint calibration of three-dimensional laser radar and camera

Abstract

The present invention discloses a dynamic calibration method, device and storage medium for joint calibration of three-dimensional laser radar and camera; the method comprises: using three-dimensional laser radar and camera to synchronously scan external scenes, respectively generating three-dimensional point cloud and two-dimensional image data; using initial installation matrix parameters, converting the coordinates of three-dimensional point cloud data from the laser radar coordinate system to the camera coordinate system; projecting the point cloud to the phase plane coordinate system to obtain depth map data; adding calibration parameter perturbation data, training and testing a joint calibration deep neural network model to obtain prediction correction parameters; correcting the initial installation matrix parameters to obtain final correction parameters. The method proposed by the present invention does not rely on special targets such as targets in the scene, and only realizes dynamic joint calibration based on the synchronous acquisition of three-dimensional point cloud data and two-dimensional image data and a deep neural network model, and has the advantages of low cost, high speed and no need for manual intervention.

Description

Technical Field

[0001] The present invention belongs to the technical field of sensor equipment calibration, and in particular, designs a dynamic calibration method, device and storage medium for joint calibration of a three-dimensional laser radar and a camera. Background Art

[0002] The fusion of two types of sensors, 3D lidar and camera, to collect data can overcome the problem of insufficient 3D environment detection capability of a single sensor. It has broad application prospects in the field of 3D environmental perception of airborne or vehicle-mounted equipment, and can solve environmental perception problems such as obstacle avoidance, positioning, and identification and tracking.

[0003] The prerequisite for using two types of sensors, 3D LiDAR and camera, at the same time is to accurately calibrate the external parameters of the two types of sensors and obtain the relative position and attitude parameters of the 3D LiDAR and camera. At present, most joint calibration methods require the use of calibration targets and manual point selection, which makes them unusable in dynamic application scenarios. In addition, due to bad weather conditions, aging of sensor hardware, and sensor offset due to vibration, the relative position and attitude of the original joint calibration of 3D LiDAR and camera will change after a period of use, and they need to be calibrated again to ensure high-precision and high-performance heterogeneous data fusion.

[0004] Therefore, there is an urgent need to design a dynamic calibration method for the joint calibration of three-dimensional lidar and camera, so that operators can perform joint calibration of the two types of sensors in real time based on dynamically collected data during the movement of the aircraft or automobile carrier, which is of great significance for the correction of calibration parameters. Summary of the invention

[0005] In view of the problems existing in the above-mentioned prior art, the present invention proposes a dynamic calibration method, device and storage medium for the joint calibration of three-dimensional laser radar and camera, so as to realize the optimization correction of the joint calibration parameters by calculating the correction parameters of the calibration parameters based on the synchronously collected three-dimensional point cloud data and two-dimensional image data using a deep neural network model without the need for the sensor to statically collect data and rely on an external calibration plate.

[0006] In order to achieve the above technical objectives, the present invention provides the following technical solutions:

[0007] A dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera, which specifically includes the following steps:

[0008] S1. Point cloud and image data acquisition: Use 3D laser radar and camera to synchronously collect data to obtain 3D point cloud data and visible light 2D image data of the common observation area;

[0009] S2, according to the initial installation matrix parameters of the 3D laser radar and camera , convert the coordinates of the 3D point cloud data obtained by 3D laser radar scanning from the laser radar coordinate system to the camera coordinate system;

[0010] The initial installation matrix parameters Provided by the design drawing of the pre-processed tooling, its parameters include: the relative position and posture parameters from the laser radar coordinate system to the camera coordinate system, that is, the translation vector between the two coordinate systems and the rotation matrix ;

[0011] S3, projecting the point cloud to the phase plane coordinate system according to the intrinsic parameters of the camera and the coordinates of the three-dimensional point cloud data in the camera coordinate system to obtain depth map data;

[0012] S4, adding calibration parameter disturbance data to the depth map data obtained in step S3, and forming a data set with the visible light two-dimensional image data obtained by synchronous acquisition in step S1; dividing the data set into a training set and a test set, and the training set is used to train the joint calibration deep neural network model;

[0013] S5. After the joint calibration deep neural network model is trained in step S4, the data in the test set is input into the model, and the prediction correction parameters of the relative position and posture of the three-dimensional laser radar and the camera are obtained through the model; the prediction correction parameters include the rotation parameter correction vector and the translation parameter correction vector ;

[0014] S6. Use the rotation parameter correction vector and the translation parameter correction vector to correct the initial installation matrix parameters. Perform correction to obtain the final correction parameters of the relative position and posture of the 3D laser radar and the camera, that is, the predicted transformation matrix obtained during the test process .

[0015] Furthermore, the synchronous data collection using the three-dimensional laser radar and the camera in step S1 is specifically as follows:

[0016] The three-dimensional laser radar and the camera are mounted and fixed on a pre-processed tooling; the tooling is designed according to the design drawing to meet the needs of the two types of sensors to collect data on the same observation area, thereby obtaining three-dimensional point cloud data and two-dimensional image data with overlapping observation areas.

[0017] Furthermore, step S3 specifically includes:

[0018] S31, according to the initial installation matrix parameters Get the translation vector from the lidar coordinate system to the camera coordinate system and the rotation matrix ;

[0019] S32, taking the three-dimensional point cloud data as input, converting the coordinates of each point in the three-dimensional point cloud data in the laser radar coordinate system through the coordinate conversion calculation formula , converted to the coordinates in the camera coordinate system ; The coordinate transformation calculation formula is:

[0020] ;

[0021] S33, multiplying the converted coordinates by the camera intrinsic parameter matrix to calculate the projection pixel coordinates in the depth map data; the camera intrinsic parameter matrix includes the focal length of the camera , pixel coordinates of the principal point ; The specific calculation process is expressed as follows:

[0022] ;

[0023] in, Represents the depth distance of the 3D point cloud from the camera.

[0024] Furthermore, step S4 specifically includes:

[0025] S41, using sample library data that has undergone high-precision joint calibration, which contains accurate calibration parameter original data; adding noise to the calibration parameter original data, i.e., certain rotation and translation disturbances, to construct calibration parameter disturbance data, and retaining the calibration parameter original data and the calibration parameter disturbance data respectively; the calibration parameter original data is used as the calibration parameter true value of the network model, and the calibration parameter disturbance data is used as the input of the network model;

[0026] S42, adding calibration parameter disturbance data to the depth map data obtained in step S3, forming a data set with the synchronously collected visible light two-dimensional image data, and dividing the data set into a training set and a test set, and using the training set as the input of the joint calibration deep neural network model to achieve multimodal information fusion;

[0027] S43, using the Adam optimizer, the network parameters are continuously optimized iteratively through the reverse gradient algorithm, the loss function is minimized, and the accuracy and robustness of the model are improved;

[0028] S44. During the training process, the loss function value is continuously monitored, and when it reaches the minimum value, the corresponding network parameters are automatically saved to ensure the effectiveness and stability of the model training and finally complete the entire training process.

[0029] Furthermore, the joint calibration deep neural network model in step S42 includes a feature extraction layer, a connection layer and a global aggregation layer, specifically:

[0030] The feature extraction layer consists of two asymmetric branches, including an RGB image branch and a depth image branch, which are used to process the data collected by the 3D lidar and the camera to obtain its features; for the RGB image branch, in order to deepen the network to extract deeper information of the features, the convolution layer of the ResNet-18 network is combined with four additional convolution layers to achieve an output channel of 1024 dimensions; for the depth image branch, a network structure similar to that of the RGB branch is used, but the number of filters at each level is half of that of the previous level;

[0031] The connection layer merges the output channel numbers of the RGB image branch and the depth image branch to integrate the RGB image and the depth image information;

[0032] The global aggregation layer includes a series of additional convolutional layers and fully connected layers to achieve global feature aggregation; each convolutional module in the entire network is followed by a BatchNorm layer to speed up convergence; finally, the global aggregation layer decouples parameters in six dimensions, including output branches of rotation and translation.

[0033] Furthermore, the loss function in step S43 is composed of pixel loss , point cloud distance loss , Directly monitor losses It consists of three parts, which are:

[0034] The pixel loss Refers to: Using the prediction transformation matrix obtained during training Project the point cloud data to get the predicted depth map, compare it with the real depth map projected by the original data of the calibration parameters, and get the error; the formula is expressed as:

[0035] ;

[0036] in, is the real depth map, is the predicted depth map with calibration error, K is the camera intrinsic parameter matrix, is the prediction transformation matrix, It is the mapping of depth map to point cloud;

[0037] The point cloud distance loss Refers to: By comparing the point cloud data without disturbance and with disturbance, and using the Chamfer Distance as the distance metric of the point cloud distance loss; two sets of point clouds and The Chamfer Distance between two points is defined as the sum of the squares of the distances between the closest points in the two sets of point clouds. The formula is:

[0038] ;

[0039] Among them, x, y are a pair of closest points between two sets of point clouds;

[0040] The direct supervision loss It means that by using the training method of self-supervised learning, the calibration parameter perturbation data packet used for training contains the rotation and translation perturbation error vectors added to the calibration parameter original data, and the output of the joint calibration deep neural network is the rotation parameter correction vector and the translation parameter correction vector; by comparing the correction vector with the perturbation error vector, the formula expression can be obtained as follows:

[0041] ;

[0042] in, and It represents the predicted rotation parameter correction vector and translation parameter correction vector obtained during network training. and Represents the artificially added rotation parameter perturbation vector and translation parameter perturbation vector.

[0043] Furthermore, step S6 is specifically as follows:

[0044] The rotation parameter correction vector obtained by the network model testing process in step S5 is used and the translation parameter correction vector , for the initial installation matrix parameters Correction is performed, and the rotation parameter correction vector and the translation parameter correction vector Transformed into correction matrix parameters through SE(3) layer ; The process of SE (3) layer conversion is: rotation parameter correction vector Convert to rotation matrix by exponential mapping method , combined with the translation parameter correction vector , get the correction matrix parameters , the formula is as follows:

[0045] ;

[0046] The correction matrix parameters With the initial installation matrix parameters Multiply them together to obtain the final correction parameters of the relative position and posture of the 3D laser radar and the camera, that is, the predicted transformation matrix .

[0047] The present invention also provides a device, comprising a memory and a processor, wherein:

[0048] A memory for storing computer programs that can be run on the processor;

[0049] The processor is used to execute the dynamic calibration method for joint calibration of three-dimensional laser radar and camera as described above when running the computer program.

[0050] In addition, according to the above-mentioned device, the present invention also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the processor to implement the dynamic calibration method of joint calibration of three-dimensional laser radar and camera as mentioned above when executed.

[0051] Based on the above technical solution, the present invention has the following beneficial effects:

[0052] Compared with the prior art, the present invention chooses to perform joint calibration based on a joint calibration deep neural network, which has the characteristics of low cost, high speed, and no need for human intervention. At the same time, the designed method is different from the traditional joint calibration method and does not rely on special goals such as targets in the scene, thereby realizing true dynamic joint calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 This is an overall flow chart of a dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera proposed in the present invention;

[0055] Figure 2 This is a comparison chart of the results before and after correction;

[0056] Figure 3 This is a visualization diagram of the final joint calibration results. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0058] Although the steps in the present invention are arranged with numbers, they are not used to limit the order of the steps. Unless the order of the steps is clearly stated or the execution of a certain step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" used in this article involves and covers any and all possible combinations of one or more of the associated listed items.

[0059] like Figure 1 As shown, the present invention proposes a dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera, which specifically includes the following steps:

[0060] S1. Point cloud and image data acquisition: Use 3D laser radar and camera to synchronously collect data to obtain 3D point cloud data and visible light 2D image data of the common observation area;

[0061] In this embodiment, Velodyne LiDAR laser radar is used; the specific internal parameters of the camera used are:

[0062] ;

[0063] The use of 3D laser radar and camera to synchronously collect data is specifically as follows:

[0064] The three-dimensional laser radar and the camera are mounted and fixed on a pre-processed tooling; the tooling is designed according to the design drawing to meet the needs of the two types of sensors to collect data on the same observation area, thereby obtaining three-dimensional point cloud data and two-dimensional image data with overlapping observation areas.

[0065] S2, according to the initial installation matrix parameters of the 3D laser radar and camera , convert the coordinates of the 3D point cloud data obtained by 3D laser radar scanning from the laser radar coordinate system to the camera coordinate system;

[0066] The initial installation matrix parameters Provided by the design drawing of the pre-processed tooling, its parameters include: relative position and posture parameters from the laser radar coordinate system to the camera coordinate system, that is, the translation vector between the two coordinate systems and the rotation matrix ;

[0067] In this embodiment, the initial installation matrix is ​​composed of a 3×3 rotation matrix and a 3×1 translation vector. The parameters of the selected initial installation matrix are specifically:

[0068] ;

[0069] S3, projecting the point cloud to the phase plane coordinate system according to the intrinsic parameters of the camera and the coordinates of the three-dimensional point cloud data in the camera coordinate system to obtain depth map data;

[0070] As a preferred implementation, step S3 specifically includes:

[0071] S31, according to the initial installation matrix parameters Get the translation vector from the lidar coordinate system to the camera coordinate system and the rotation matrix ;

[0072] S32, taking the three-dimensional point cloud data as input, converting the coordinates of each point in the three-dimensional point cloud data in the laser radar coordinate system through the coordinate conversion calculation formula , converted to the coordinates in the camera coordinate system ; The coordinate transformation calculation formula is:

[0073] ;

[0074] S33, multiplying the converted coordinates by the camera intrinsic parameter matrix to calculate the projection pixel coordinates in the depth map data; the camera intrinsic parameter matrix includes the focal length of the camera , pixel coordinates of the principal point ; The specific calculation process is expressed as follows:

[0075] ;

[0076] in, Represents the depth distance of the 3D point cloud from the camera.

[0077] S4, adding calibration parameter disturbance data to the depth map data obtained in step S3, and forming a data set with the visible light two-dimensional image data obtained by synchronous acquisition in step S1; dividing the data set into a training set and a test set, and the training set is used to train the joint calibration deep neural network model;

[0078] As a preferred implementation, step S4 specifically includes:

[0079] S41, using the sample library data that has undergone high-precision joint calibration, adding noise to the calibration parameter original data, constructing the calibration parameter perturbation data, and retaining the calibration parameter original data and the calibration parameter perturbation data respectively; the calibration parameter original data provides an output true value for the network model, and the calibration parameter perturbation data serves as the input of the network model;

[0080] S42, adding calibration parameter disturbance data to the depth map data obtained in step S3, forming a data set with the synchronously collected visible light two-dimensional image data, and dividing the data set into a training set and a test set, and using the training set as the input of the joint calibration deep neural network model to achieve multimodal information fusion;

[0081] More specifically, the joint calibration deep neural network model in step S42 includes a feature extraction layer, a connection layer and a global aggregation layer, specifically:

[0082] The feature extraction layer consists of two asymmetric branches, including an RGB image branch and a depth image branch, which are used to process the data collected by the 3D lidar and the camera to obtain its features; for the RGB image branch, in order to deepen the network to extract deeper information of the features, the convolution layer of the ResNet-18 network is combined with four additional convolution layers to achieve an output channel of 1024 dimensions; for the depth image branch, a network structure similar to that of the RGB branch is used, but the number of filters at each level is half of that of the previous level;

[0083] The connection layer merges the output channel numbers of the RGB image branch and the depth image branch to integrate the RGB image and the depth image information;

[0084] The global aggregation layer includes a series of additional convolutional layers and fully connected layers to achieve global feature aggregation; each convolutional module in the entire network is followed by a BatchNorm layer to speed up convergence; finally, the global aggregation layer decouples parameters in six dimensions, including output branches of rotation and translation.

[0085] S43, using the Adam optimizer, the network parameters are continuously optimized iteratively through the reverse gradient algorithm, the loss function is minimized, and the accuracy and robustness of the model are improved;

[0086] More specifically, the loss function in step S43 is composed of pixel loss , point cloud distance loss , Directly monitor losses It consists of three parts, which are:

[0087] The pixel loss Refers to: Using the prediction transformation matrix obtained during training Project the point cloud data to get the predicted depth map, compare it with the real depth map projected by the original data of the calibration parameters, and get the error; the formula is expressed as:

[0088] ;

[0089] in, is the real depth map, is the predicted depth map with calibration error, K is the camera intrinsic parameter matrix, is the prediction transformation matrix, It is the mapping of depth map to point cloud;

[0090] The point cloud distance loss Refers to: By comparing the point cloud data without disturbance and with disturbance, and using the Chamfer Distance as the distance metric of the point cloud distance loss; two sets of point clouds and The Chamfer Distance between two points is defined as the sum of the squares of the distances between the closest points in the two sets of point clouds. The formula is:

[0091] ;

[0092] Among them, x, y are a pair of closest points between two sets of point clouds;

[0093] The direct supervision loss It means that by using the training method of self-supervised learning, the calibration parameter perturbation data packet used for training contains the rotation and translation perturbation error vectors added to the calibration parameter original data, and the output of the joint calibration deep neural network is the rotation parameter correction vector and the translation parameter correction vector; by comparing the correction vector with the perturbation error vector, the formula expression can be obtained as follows:

[0094] ;

[0095] in, and It represents the predicted rotation parameter correction vector and translation parameter correction vector obtained during network training. and Represents the artificially added rotation parameter perturbation vector and translation parameter perturbation vector.

[0096] S44. During the training process, the loss function value is continuously monitored, and when it reaches the minimum value, the corresponding network parameters are automatically saved to ensure the effectiveness and stability of the model training and finally complete the entire training process.

[0097] In this embodiment, the model configuration is first optimized through a training process, and then a test set is input for testing to obtain the final prediction conversion matrix T; in essence, the process of obtaining the prediction conversion matrix through the training process and the testing process is the same.

[0098] S5. After the joint calibration deep neural network model is trained in step S4, the data in the test set is input into the model, and the prediction correction parameters of the relative position and posture of the three-dimensional laser radar and the camera are obtained through the model; the prediction correction parameters include the rotation parameter correction vector and the translation parameter correction vector ;

[0099] S6. Use the rotation parameter correction vector and the translation parameter correction vector to correct the initial installation matrix parameters. Perform calibration to obtain the final calibration parameters of the relative position and posture of the 3D laser radar and the camera;

[0100] More specifically, in this embodiment, step S6 specifically uses the rotation parameter correction vector obtained by the network model testing process in step S5 and the translation parameter correction vector , for the initial installation matrix parameters Correction is performed, and the rotation parameter correction vector and the translation parameter correction vector Transformed into correction matrix parameters through SE(3) layer ; The process of SE(3) layer conversion is: rotation parameter correction vector Convert to rotation matrix by exponential mapping method , combined with the translation parameter correction vector , and obtain the correction matrix , the formula is as follows:

[0101] ;

[0102] The correction matrix parameters With the initial installation matrix parameters Multiply them together to obtain the final correction parameters of the relative position and posture of the 3D laser radar and the camera, that is, the predicted transformation matrix .

[0103] This embodiment further provides a device, including a memory and a processor, wherein:

[0104] A memory for storing computer programs that can be run on the processor;

[0105] The processor is used to execute the dynamic calibration method for joint calibration of three-dimensional laser radar and camera as described above when running the computer program.

[0106] In addition, according to the above-mentioned device, the present invention also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to implement the dynamic calibration method of joint calibration of three-dimensional laser radar and camera as described above when the processor executes them.

[0107] Through the above method, we can obtain Figure 2 , Figure 3 The effect shown:

[0108] The quantitative effect of combined calibration is as follows Figure 2 As shown in Figure 2, the final joint calibration result visualization effect is as follows: Figure 3 As shown. Figure 2(a) shows the average rotation error before and after calibration. In the case of the original error calibration with ±10° random error, after correction by the network, the results on the test set show that the average absolute error of the rotation angle is: (X: 0.1382°, Y: 0.7224°, Z: 0.1901°), and the average rotation error is only 0.35°. Figure 2 (b) shows the average translation error before and after calibration. Under the original incorrect joint calibration with a random error of ±0.2m, after correction by the network, the average absolute error value of the translation component is: (X: 4.35cm, Y: 1.69cm, Z: 3.68cm). Figure 2 It can be seen that after using the dynamic calibration method for joint calibration of three-dimensional lidar and camera designed in this article, both rotation and translation errors have been greatly improved, and accurate joint calibration can be achieved.

[0109] In order to prove the robustness of the algorithm, joint calibration experiments of LiDAR and camera were conducted in different scenarios, including alleys, roads, and streets. Figure 3 (a) is the input RGB image. Under the condition of joint external reference true, perturbed and predicted, we can get the following in turn: Figure 3 The real joint external parameter point cloud projection effect diagram shown in (b) is as follows: Figure 3 The disturbed joint external parameter point cloud projection effect diagram shown in (c) is as follows: Figure 3 The predicted joint external parameter point cloud projection effect diagram shown in (d) in the figure; the joint external parameter refers to the external parameters between the lidar and the camera, and the predicted transformation matrix obtained by jointly calibrating the deep neural network It is also a joint external parameter. As can be seen from the figure, it is difficult to calibrate the car under disturbance, that is, when the initial error is very high; while the method proposed in the present invention can obtain a point cloud projection that is basically consistent with the actual situation, that is, the network model successfully aligns the incorrectly calibrated depth map to the RGB image, completing the joint calibration work.

[0110] In summary, the method proposed in the present invention can achieve the optimization correction of the joint calibration parameters by calculating the correction parameters of the calibration parameters based on the synchronously collected three-dimensional point cloud data and two-dimensional image data using a deep neural network model without the need for the sensor to collect data statically and rely on an external calibration plate; at the same time, it has lower cost and faster calibration speed.

[0111] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0112] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera, characterized in that: The specific steps include: S1. Point cloud and image data acquisition: Use 3D laser radar and camera to synchronously collect data to obtain 3D point cloud data and visible light 2D image data of the common observation area; S2, according to the initial installation matrix parameters of the 3D laser radar and camera , convert the coordinates of the 3D point cloud data obtained by 3D laser radar scanning from the laser radar coordinate system to the camera coordinate system; The initial installation matrix parameters Provided by the design drawing of the pre-processed tooling, its parameters include: the relative position and posture parameters from the laser radar coordinate system to the camera coordinate system, that is, the translation vector between the two coordinate systems and the rotation matrix ; S3, projecting the point cloud to the phase plane coordinate system according to the intrinsic parameters of the camera and the coordinates of the three-dimensional point cloud data in the camera coordinate system to obtain depth map data; S4, adding calibration parameter disturbance data to the depth map data obtained in step S3, and forming a data set with the visible light two-dimensional image data obtained by synchronous acquisition in step S1; dividing the data set into a training set and a test set, and the training set is used to train the joint calibration deep neural network model; S5. After the joint calibration deep neural network model is trained in step S4, the data in the test set is input into the model, and the prediction correction parameters of the relative position and posture of the three-dimensional laser radar and the camera are obtained through the model; the prediction correction parameters include the rotation parameter correction vector and the translation parameter correction vector ; S6. Use the rotation parameter correction vector and the translation parameter correction vector to correct the initial installation matrix parameters. Perform correction to obtain the final correction parameters of the relative position and posture of the 3D laser radar and the camera, that is, the predicted transformation matrix obtained during the test process .

2. The dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera according to claim 1, characterized in that: The synchronous data collection using the three-dimensional laser radar and the camera described in step S1 is specifically as follows: The three-dimensional laser radar and the camera are mounted and fixed on a pre-processed tooling; the tooling is designed according to the design drawing to meet the needs of the two types of sensors to collect data on the same observation area, thereby obtaining three-dimensional point cloud data and two-dimensional image data with overlapping observation areas.

3. The dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera according to claim 1, characterized in that: Step S3 specifically includes: S31, according to the initial installation matrix parameters Get the translation vector from the lidar coordinate system to the camera coordinate system and the rotation matrix ; S32, taking the three-dimensional point cloud data as input, converting the coordinates of each point in the three-dimensional point cloud data in the laser radar coordinate system through the coordinate conversion calculation formula , converted to the coordinates in the camera coordinate system ; The coordinate transformation calculation formula is: ； S33, multiplying the converted coordinates by the camera intrinsic parameter matrix to calculate the projection pixel coordinates in the depth map data; the camera intrinsic parameter matrix includes the focal length of the camera , pixel coordinates of the principal point ; The specific calculation process is expressed as follows: ； in, Represents the depth distance of the 3D point cloud from the camera.

4. The dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera according to claim 1, characterized in that: Step S4 specifically includes: S41, using sample library data that has undergone high-precision joint calibration, which contains accurate calibration parameter original data; adding noise to the calibration parameter original data, i.e., certain rotation and translation disturbances, to construct calibration parameter disturbance data, and retaining the calibration parameter original data and the calibration parameter disturbance data respectively; the calibration parameter original data is used as the calibration parameter true value of the network model, and the calibration parameter disturbance data is used as the input of the network model; S42, adding calibration parameter disturbance data to the depth map data obtained in step S3, forming a data set with the synchronously collected visible light two-dimensional image data, and dividing the data set into a training set and a test set, and using the training set as the input of the joint calibration deep neural network model to achieve multimodal information fusion; S43, using the Adam optimizer, the network parameters are continuously optimized iteratively through the reverse gradient algorithm, the loss function is minimized, and the accuracy and robustness of the model are improved; S44. During the training process, the loss function value is continuously monitored, and when it reaches the minimum value, the corresponding network parameters are automatically saved to ensure the effectiveness and stability of the model training and finally complete the entire training process.

5. The dynamic calibration method for joint calibration of three-dimensional laser radar and camera according to claim 4, characterized in that: The joint calibration deep neural network model in step S42 includes a feature extraction layer, a connection layer and a global aggregation layer, specifically: The feature extraction layer consists of two asymmetric branches, including an RGB image branch and a depth image branch, which are used to process the data collected by the 3D laser radar and the camera to obtain its features; For the RGB image branch, in order to deepen the network to extract deeper information of features, the convolutional layer of the ResNet-18 network is combined with four additional convolutional layers to achieve an output channel of 1024 dimensions; for the depth map branch, a similar network structure is used as the RGB branch, but the number of filters at each level is half of that of the previous level; The connection layer merges the output channel numbers of the RGB image branch and the depth image branch to integrate the RGB image and the depth image information; The global aggregation layer includes a series of additional convolutional layers and fully connected layers to achieve global feature aggregation; each convolutional module in the entire network is followed by a BatchNorm layer to speed up convergence; finally, the global aggregation layer decouples parameters in six dimensions, including output branches of rotation and translation.

6. The dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera according to claim 4, characterized in that: The loss function in step S43 is composed of pixel loss , point cloud distance loss , direct supervision of losses It consists of three parts, which are: The pixel loss Refers to: Using the prediction transformation matrix obtained during training Project the point cloud data to get the predicted depth map, compare it with the real depth map projected by the original data of the calibration parameters, and get the error; the formula is expressed as: ； in, is the real depth map, is the predicted depth map with calibration error, K is the camera intrinsic parameter matrix, is the prediction transformation matrix, It is the mapping of depth map to point cloud; The point cloud distance loss Refers to: By comparing the point cloud data without disturbance and with disturbance, and using ChamferDistance as the distance metric of point cloud distance loss; two sets of point clouds and The ChamferDistance between two points is defined as the sum of the squares of the distances between the closest points in the two sets of point clouds. The formula is: ； Among them, x, y are a pair of closest points between two sets of point clouds; The direct supervision loss It means that by using the training method of self-supervised learning, the calibration parameter perturbation data packet used for training contains the rotation and translation perturbation error vectors added to the calibration parameter original data, and the output of the joint calibration deep neural network is the rotation parameter correction vector and the translation parameter correction vector; by comparing the correction vector with the perturbation error vector, the formula expression can be obtained as follows: ； in, and It represents the predicted rotation parameter correction vector and translation parameter correction vector obtained during network training. and Represents the artificially added rotation parameter perturbation vector and translation parameter perturbation vector.

7. The dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera according to claim 1, characterized in that: Step S6 is specifically as follows: The rotation parameter correction vector obtained by the network model testing process in step S5 is used and the translation parameter correction vector , for the initial installation matrix parameters Correction is performed, and the rotation parameter correction vector and the translation parameter correction vector Transformed into correction matrix parameters through SE(3) layer ; The process of SE (3) layer conversion is: rotation parameter correction vector Convert to rotation matrix by exponential mapping method , combined with the translation parameter correction vector , get the correction matrix parameters , the formula is as follows: ； The correction matrix parameters With the initial installation matrix parameters Multiply them together to obtain the final correction parameters of the relative position and posture of the 3D laser radar and the camera, that is, the predicted transformation matrix .

8. A device, characterized in that: The device comprises a memory and a processor, wherein: A memory for storing computer programs that can be run on the processor; A processor is used to execute the dynamic calibration method for joint calibration of three-dimensional laser radar and camera as described in any one of claims 1 to 7 when running the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the dynamic calibration method for joint calibration of a three-dimensional laser radar and a camera according to any one of claims 1 to 7 when executed.

Images