Multi-sensor calibration method, device and system

Abstract

The invention relates to the technical field of multi-sensor calibration, and particularly discloses a multi-sensor calibration method, device and system, and the method comprises the steps: obtaining image data and point cloud data; preprocessing the image data and the point cloud data to enable the point cloud data to be projected to an image coordinate system where the image data is located; performing iterative optimization processing on a projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix; repeating the previous steps according to the image data and the point cloud data at the current sampling moment to obtain an optimized coordinate transformation matrix at the current sampling moment; and repeating the step of obtaining the optimized coordinate transformation matrix at the current sampling moment, maintaining the sample number of the optimized coordinate transformation matrix at a preset threshold value, and obtaining an expected coordinate transformation matrix. According to the multi-sensor calibration method provided by the invention, a stable and accurate calibration result can be obtained without a calibration plate, and the method can adapt to environmental changes.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor calibration, and particularly to a multi-sensor calibration method, a multi-sensor calibration device, and a multi-sensor calibration system. Background Art

[0002] With the continuous progress of autonomous driving technology and the innovation of sensor technology, multi-sensor fusion is increasingly widely used in the field of autonomous driving. In heterogeneous operating scenarios, such as urban streets, highways, mining areas, etc., autonomous driving systems need to maintain high perception stability and accuracy in various complex environments. Traditional multi-sensor calibration methods rely on calibration boards or other fixed targets, which have limitations in dynamic changing environments, especially in adverse weather conditions or when the vehicle has large vibrations and noises. Currently, multi-sensor calibration technologies are mainly designed for specific scenarios or specific types of vehicles, such as small cars or intelligent robots. These methods generally include calibration based on markers and calibration based on environmental features. However, these methods have some challenges in practical applications. For example, in complex weather conditions such as rain, fog, and snow, the visibility of the calibration board is reduced, or when the vehicle has large vibrations and noises, the stability of the sensors is affected, resulting in a decline in perception performance.

[0003] Therefore, how to provide a calibration method that can adapt to environmental changes to improve perception performance has become an urgent technical problem for those skilled in the art. Summary of the Invention

[0004] The present invention provides a multi-sensor calibration method, a multi-sensor calibration device, and a multi-sensor calibration system, which solve the problem in the related art that the calibration method with a calibration board is easily affected by the environment, thereby causing a decline in perception performance.

[0005] As the first aspect of the present invention, there is provided a multi-sensor calibration method, which includes:

[0006] Obtain image data and point cloud data;

[0007] Preprocess both the image data and the point cloud data so that the point cloud data is projected onto the image coordinate system where the image data is located;

[0008] Perform iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix;

[0009] Repeat the foregoing steps according to the image data and the point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment;

[0010] Repeat the step of obtaining the optimized coordinate transformation matrix at the current sampling moment, and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix.

[0011] Further, preprocess both the image data and the point cloud data so that the point cloud data is projected onto the image coordinate system where the image data is located, including:

[0012] Perform image semantic segmentation on the image data to obtain multiple image segmentation regions, and perform point cloud feature extraction on the point cloud data to obtain multiple point cloud features;

[0013] Convert the point cloud features from the lidar coordinate system where the point cloud data is located to the image coordinate system where the image data is located according to a preset coordinate transformation matrix, so that each image segmentation region obtains multiple point cloud feature projection points;

[0014] Calculate the point cloud attribute consistency of the point cloud feature projection points in each image segmentation region to obtain the projection result of the point cloud data in the image coordinate system.

[0015] Further, calculating the point cloud attribute consistency of the point cloud feature projection points in each image segmentation region includes:

[0016] Calculate the consistency scores of the intensity of the point cloud, the normal vector of the point cloud, and the segmentation category of the point cloud in each image segmentation region respectively;

[0017] Determine the consistency score of the point cloud feature projection points in the image segmentation region according to the consistency score of the intensity of the point cloud, the consistency score of the normal vector of the point cloud, and the consistency score of the segmentation category of the point cloud, and use the consistency score of the point cloud feature projection points in the image segmentation region as the projection result in the image coordinate system.

[0018] Further, perform iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix, including:

[0019] Determine the loss function of the iterative optimization algorithm according to the projection result of the point cloud data in the image coordinate system;

[0020] Optimize the loss function according to the iterative optimization algorithm until a preset iterative stop condition is met;

[0021] Determine the corresponding optimized coordinate transformation matrix according to the loss function when the preset iterative stop condition is met.

[0022] Further, optimizing the loss function according to the iterative optimization algorithm until a preset iterative stop condition is met includes:

[0023] Update the external parameter matrix according to the gradient descent algorithm and along the opposite direction of the gradient to minimize the loss function;

[0024] When the preset iteration stop condition is satisfied, stop the update and obtain the loss function at the time of stopping the update, where the preset iteration stop condition includes any one of the norm of the gradient descent algorithm being less than a preset norm threshold, the change in the loss function being less than a preset change threshold, and the number of iterations reaching a preset number of iterations.

[0025] Further, repeat the foregoing steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment, including:

[0026] Project the point cloud data at the current sampling moment into the image coordinate system where the image data at the current sampling moment is located according to the optimized coordinate transformation matrix at the previous sampling moment;

[0027] Determine the corresponding current loss function according to the projection result at the current sampling moment;

[0028] Iteratively optimize the current loss function according to the pre-trained neural network model until the preset iteration stop condition is satisfied to obtain the optimized coordinate transformation matrix at the current sampling moment.

[0029] Further, obtaining the pre-trained neural network model includes:

[0030] Determine the training data set, where the training data set includes feature vectors and corresponding external parameter errors, and the feature vectors include vehicle operating state information, environmental data information, and sensor data information;

[0031] Determine the neural network model prediction function according to the training data set;

[0032] Train according to the training data set and the neural network model prediction function to minimize the loss function and obtain the pre-trained neural network model.

[0033] Further, repeat the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix, including:

[0034] Form a sample set of the optimized coordinate transformation matrix from the optimized coordinate transformation matrices at multiple sampling moments;

[0035] Judge whether the number of samples in the sample set of the optimized coordinate transformation matrix reaches the preset threshold;

[0036] If it does not reach the preset threshold, directly add the newly obtained optimized coordinate transformation matrix at the current sampling moment to the sample set of the optimized coordinate transformation matrix;

[0037] If a preset threshold is reached, delete the optimized coordinate transformation matrix at the earliest sampling time, and then add the newly obtained optimized coordinate transformation matrix at the current sampling time to the sample set of the optimized coordinate transformation matrix;

[0038] Retrain the neural network model according to the sample set of the optimized coordinate transformation matrix to obtain a neural network model matching the current sampling time;

[0039] Obtain the expected coordinate transformation matrix according to the neural network model matching the current sampling time.

[0040] As another aspect of the present invention, there is provided a multi-sensor calibration device for implementing the multi-sensor calibration method described above, which includes:

[0041] An acquisition module for acquiring image data and point cloud data;

[0042] A preprocessing module for preprocessing both the image data and the point cloud data so that the point cloud data is projected onto the image coordinate system where the image data is located;

[0043] An iterative optimization module for performing iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix;

[0044] A module for repeatedly obtaining the optimized coordinate transformation matrix, which is used to repeatedly perform the foregoing steps according to the image data and the point cloud data at the current sampling time to obtain the optimized coordinate transformation matrix at the current sampling time;

[0045] A real-time optimization module for repeatedly performing the step of obtaining the optimized coordinate transformation matrix at the current sampling time and maintaining the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the expected coordinate transformation matrix.

[0046] As another aspect of the present invention, there is provided a multi-sensor calibration system, which includes: an image sensor, a lidar sensor, and the multi-sensor calibration device described above, and both the image sensor and the lidar sensor are communicatively connected to the multi-sensor calibration device,

[0047] The image sensor is used to collect environmental image information to form image data,

[0048] The lidar sensor is used to detect environmental information through lidar signals to form point cloud data,

[0049] The multi-sensor calibration device is used to automatically calibrate according to the image data and the point cloud data to obtain a coordinate transformation matrix.

[0050] The multi-sensor calibration method provided by the present invention obtains the projection result of point cloud data projected onto the image coordinate system through the preprocessing of image data and point cloud data, and optimizes to obtain an optimized coordinate transformation matrix based on this projection result. Finally, the obtained multiple optimized coordinate transformation matrices are optimized and adjusted to obtain a stable and accurate coordinate transformation matrix. This multi-sensor calibration method of the present invention does not rely on an external calibration board. On the basis of initial calibration, the transformation matrix between the camera and the radar sensor is obtained through iterative optimization processing of the initial calibration. This multi-sensor calibration method of the present invention can adjust the external parameters of the sensor in real time according to the dynamic changes of the vehicle in different operating scenarios to adapt to environmental changes, thereby improving the stability and accuracy of the perception system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation to the present invention.

[0052] Figure 1 It is a flowchart of the multi-sensor calibration method provided by the present invention.

[0053] Figure 2 It is a flowchart of preprocessing the image data and the point cloud data provided by the present invention.

[0054] Figure 3 It is a flowchart of calculating the point cloud attribute consistency in the image segmentation region provided by the present invention.

[0055] Figure 4 It is a schematic diagram of the consistency score in the image segmentation region provided by the present invention.

[0056] Figure 5 It is a flowchart of performing iterative optimization processing on the projection result provided by the present invention.

[0057] Figure 6 It is a flowchart of optimizing the loss function provided by the present invention.

[0058] Figure 7 It is a flowchart of repeatedly obtaining the optimized coordinate transformation matrix at the current sampling moment provided by the present invention.

[0059] Figure 8 It is a flowchart of obtaining the desired coordinate transformation matrix provided by the present invention.

[0060] Figure 9 It is a schematic diagram of the legend of multi-sensor calibration provided by the present invention.

[0061] Figure 10 It is a structural block diagram of the multi-sensor calibration device provided by the present invention.

[0062] Figure 11 It is a structural block diagram of the multi - sensor calibration system provided by the present invention. Detailed implementation manners

[0063] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0064] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in combination with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above - mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such data used in an appropriate case can be interchanged so as to describe the embodiments of the present invention here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non - exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0066] In this embodiment, a multi - sensor calibration method is provided. Figure 1 It is a flowchart of the multi - sensor calibration method provided by the embodiment of the present invention, as Figure 1 shown, and includes:

[0067] S100. Obtain image data and point cloud data;

[0068] In the embodiment of the present invention, specifically, the image information of the vehicle surrounding environment can be collected by an image sensor to form image data, and the vehicle surrounding environment can be detected by a lidar sensor to form point cloud data.

[0069] S200. Perform pre - processing on both the image data and the point cloud data so that the point cloud data is projected onto the image coordinate system where the image data is located.

[0070] In the embodiments of the present invention, the image data and the point cloud data are preprocessed respectively, so that the point cloud data can be projected in the image coordinate system where the image data is located. It should be understood that when the point cloud data is initially projected in the image coordinate system, it can be specifically realized by a preset coordinate transformation matrix.

[0071] S300. Perform iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix;

[0072] It should be understood that a projection consistency score calculation can be performed on the projection of the point cloud data in the image coordinate system, that is, the matching situation between the point cloud data and the image data is evaluated after the point cloud data is projected into the image coordinate system, and iterative optimization processing is performed according to the evaluation result to narrow the gap between the point cloud data and the image data in the image coordinate system.

[0073] S400. Repeat the foregoing steps according to the image data and the point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment;

[0074] Specifically, the optimized coordinate transformation matrix at the current sampling moment can be obtained by repeating the foregoing steps. It should be understood that the optimized coordinate transformation matrix at the current sampling moment is specifically optimized on the basis of the optimized coordinate transformation matrix at the previous sampling moment.

[0075] It should be noted here that the preset coordinate transformation matrix is used for the initial projection, and then at the second sampling moment, the preset coordinate transformation matrix is optimized to obtain the optimized coordinate transformation matrix. By analogy, the optimized coordinate transformation matrices at subsequent sampling moments are all obtained after optimizing the optimized coordinate transformation matrix at the previous sampling moment.

[0076] S500. Repeat the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment, and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix.

[0077] It should be understood that the optimized coordinate transformation matrices at multiple sampling moments can be obtained by repeating the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment. The optimized coordinate transformation matrices at multiple sampling moments are adjusted in real time to maintain the number of samples at a preset threshold, and finally an accurate and stable coordinate transformation matrix is obtained.

[0078] In summary, the multi-sensor calibration method provided by the present invention obtains the projection result of the point cloud data projected onto the image coordinate system through the preprocessing of the image data and the point cloud data, and optimizes the coordinate transformation matrix based on the projection result. Finally, the obtained multiple optimized coordinate transformation matrices are optimized and adjusted to obtain a stable and accurate coordinate transformation matrix. This multi-sensor calibration method of the present invention does not rely on an external calibration board. On the basis of the initial calibration, the transformation matrix between the camera and the radar sensor is obtained through iterative optimization processing of the initial calibration. This multi-sensor calibration method of the present invention can adjust the external parameters of the sensor in real time according to the dynamic changes of the vehicle in different operating scenarios to adapt to the changes in the environment, thereby improving the stability and accuracy of the perception system.

[0079] In the embodiment of the present invention, both the image data and the point cloud data are preprocessed so that the point cloud data is projected onto the image coordinate system where the image data is located, as Figure 2 shown, including:

[0080] S210. Perform image semantic segmentation on the image data to obtain multiple image segmentation regions, and perform point cloud feature extraction on the point cloud data to obtain multiple point cloud features;

[0081] It should be understood that when performing image semantic segmentation on the image data, the specific number of the image segmentation regions after segmentation is ; the preprocessing of the point cloud data may specifically include normal estimation, intensity normalization, and segmentation category.

[0082] S220. According to a preset coordinate transformation matrix, convert the point cloud features from the lidar coordinate system where the point cloud data is located to the image coordinate system where the image data is located, so that each image segmentation region obtains multiple point cloud feature projection points;

[0083] Specifically, each point in the point cloud is converted from the lidar coordinate system to the image coordinate system, and the consistency of the point cloud attributes within each segmentation region is calculated, including the intensity, normal vector, and segmentation category of the point cloud. The alignment between the image and the point cloud is evaluated by calculating the consistency score.

[0084] In the embodiment of the present invention, the preset coordinate transformation matrix when the point cloud data is projected onto the image coordinate system is expressed as:

[0085] ,

[0086] where represents a scale factor for converting homogeneous coordinates to regular two-dimensional coordinates, represents the internal parameter matrix of the camera, represents the empirically estimated extrinsic parameter matrix from the lidar to the camera, including rotation and translation. , represents the position of point p in the lidar coordinate system, represents the position of point p in the image coordinate system.

[0087] S230. Calculate the point cloud attribute consistency of the projected points of the point cloud features in each image segmentation region to obtain the projection result of the point cloud data in the image coordinate system.

[0088] In the embodiments of the present invention, specifically as Figure 3 shown, calculating the point cloud attribute consistency of the projected points of the point cloud features in each image segmentation region includes:

[0089] S231. Calculate the consistency scores of the intensity of the point cloud, the normal vector of the point cloud, and the segmentation category of the point cloud in each image segmentation region respectively;

[0090] S232. Determine the consistency score of the projected points of the point cloud features in the image segmentation region according to the consistency score of the intensity of the point cloud, the consistency score of the normal vector of the point cloud, and the consistency score of the segmentation category of the point cloud, and use the consistency score of the projected points of the point cloud features in the image segmentation region as the projection result in the image coordinate system.

[0091] Specifically, for each image segmentation region , a set of points falling on it can be obtained :

[0092] ,

[0093] Here, represents a binary function, and when point p falls within the image segmentation region , the function value is 1.

[0094] In the embodiments of the present invention, the consistency calculation formula for reflectivity (i.e., the intensity of the point cloud) is:

[0095] ,

[0096] Among them, represents the intensity values of all points in the point set;

[0097] The consistency calculation formula for the normal vector is:

[0098] ,

[0099] Among them, represents the normal vector of the jth point in the point set , Represents the average direction of all normal vectors. Represents the number of points in the point cloud.

[0100] For each segmentation category in the point cloud, first calculate the number of points in each category, and then sort these numbers of points from largest to smallest. The sorted numbers of points are denoted as , , ..., therefore, the calculation formula for the segmentation category consistency is:

[0101] ,

[0102] ,

[0103] Among them, Represents the number of points in the i-th category, Represents the total sum of the number of points in the point cloud of all categories, Represents the weight factor, which is used to reduce its contribution to the total consistency score as the number of points in the category decreases.

[0104] In summary, for the measured point set The consistency score formula is:

[0105] ,

[0106] Among them, Represents the weight coefficients, corresponding to the reflectivity consistency, normal vector consistency, and segmentation category consistency respectively; Represents the point set The reflectivity consistency score of, Represents the point set The normal vector consistency score of, Represents the point set The segmentation category consistency score of.

[0107] Finally, the calculation formula for the consistency score s is:

[0108] ,

[0109] Among them, Represents the weight of the th image segmentation region, and the consistency score Is the consistency score of the th image segmentation region, specifically as Figure 4 Shown. The above calculation of the consistency score comprehensively considers the intensity, normal vector, and segmentation category consistency of the point cloud to evaluate the alignment between the image and the point cloud and is used to optimize the external parameters.

[0110] It should be understood that the consistency score is mainly used to indicate the consistency difference between the point cloud features projected into the image coordinate system and the image data in the image coordinate system. Since the projection is implemented based on the coordinate transformation matrix, this consistency difference can characterize the difference between the coordinate transformation matrix used in the projection and the expected coordinate transformation matrix. Furthermore, by optimizing the consistency difference, the current coordinate transformation matrix can be optimized to obtain the expected coordinate transformation matrix.

[0111] In the embodiment of the present invention, iterative optimization processing is performed on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix, as Figure 5 shown, including:

[0112] S310. Determine the loss function of the iterative optimization algorithm according to the projection result of the point cloud data in the image coordinate system;

[0113] It should be understood that in the embodiment of the present invention, the gradient descent algorithm is used to optimize the external parameter matrix, realizing zero-training external parameter initial calibration without a calibration board, and providing an accurate optimized coordinate transformation relationship for the multi-sensor system. Specifically, the loss function can be determined for the projection result, that is, the consistency score situation described above. Since the consistency score exactly indicates the consistency difference between the point cloud features projected into the image coordinate system and the image data in the image coordinate system, a loss function matching this consistency difference is determined based on it, and iterative optimization processing is performed on this loss function.

[0114] In the embodiment of the present invention, using the gradient descent algorithm to approximate the optimal transformation matrix (external parameter matrix) is an iterative optimization process. Define the loss function as the sum of the consistency scores of all image segmentation regions:

[0115] ,

[0116] ,

[0117] where, represents the elements of the external parameter matrix .

[0118] S320. Optimize the loss function according to the iterative optimization algorithm until a preset iterative stop condition is met;

[0119] In the embodiment of the present invention, as Figure 6 shown, optimizing the loss function according to the iterative optimization algorithm until a preset iterative stop condition is met includes:

[0120] S321. Update the external parameter matrix along the opposite direction of the gradient according to the gradient descent algorithm to minimize the loss function;

[0121] S322. Stop the update when the preset iteration stop condition is satisfied, and obtain the loss function at the time of stopping the update, where the preset iteration stop condition includes any one of the norm of the gradient descent algorithm being less than a preset norm threshold, the change in the loss function being less than a preset change threshold, and the number of iterations reaching a preset number of iterations.

[0122] Specifically, the gradient descent algorithm minimizes the loss function by updating the parameters along the opposite direction of the gradient. The update rule is as follows:

[0123] ,

[0124] where, represents the learning rate, a positive scalar used to control the step size of each iteration. Repeat the process of calculating the gradient and updating the parameters until the stop condition is met. The iteration stop condition can specifically include the norm of the gradient being less than a certain threshold, or the change in the loss function being less than a certain threshold, or reaching a preset number of iterations. Finally, when the gradient descent algorithm converges, the obtained external parameter matrix is the transformation matrix from the lidar to the camera coordinate system.

[0125] S330. Determine the corresponding optimized coordinate transformation matrix according to the loss function at the preset iteration stop condition.

[0126] It should be understood that the coordinate transformation matrix corresponding to the loss function obtained by iteratively updating the loss function until the iteration stop condition is satisfied can be understood as the optimized coordinate transformation matrix expected above. That is, when the loss function reaches the minimum, the consistency score also reaches the minimum, which further indicates that the difference between the point cloud features projected into the image coordinate system and the image features is minimized. At this moment, the corresponding coordinate transformation matrix is the expected coordinate transformation matrix.

[0127] In the embodiment of the present invention, repeat the foregoing steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment, as Figure 7 shown, including:

[0128] S410. Project the point cloud data at the current sampling moment into the image coordinate system where the image data at the current sampling moment is located according to the optimized coordinate transformation matrix at the previous sampling moment;

[0129] S420. Determine the corresponding current loss function according to the projection result at the current sampling moment;

[0130] S430. Perform iterative optimization processing on the current loss function according to the pre-trained neural network model until the preset iteration stop condition is satisfied to obtain the optimized coordinate transformation matrix at the current sampling moment.

[0131] Specifically, obtaining a pre-trained neural network model includes:

[0132] 1) Determine a training data set, where the training data set includes feature vectors and corresponding extrinsic error, and the feature vectors include vehicle running state information, environmental data information, and sensor data information;

[0133] 2) Determine a neural network model prediction function according to the training data set;

[0134] 3) Train according to the training data set and the neural network model prediction function to minimize the loss function, and obtain a pre-trained neural network model.

[0135] It should be understood that for the pre-trained neural network model, it is necessary to complete the data fusion of IMU and GPS sensors, vehicle state estimation, and parameter update.

[0136] Specifically, the vehicle running saves data samples. Now assume that the initial data sample is , and each sample contains a feature vector and the corresponding extrinsic error :

[0137] ,

[0138] The feature vector is constructed in the following way:

[0139] .

[0140] The sensor data is the received reflectivity consistency score, normal vector consistency score, and segmentation category consistency score; the vehicle state is the statistical features of the three-axis angular velocity and the statistical features of the three-axis acceleration, and the environmental data is highways, urban roads, and unstructured roads.

[0141] Select a linear regression model. The training data set and the model prediction function can be expressed as:

[0142] ,

[0143] ,

[0144] The training process is realized by minimizing the loss function and solved by the gradient descent optimization algorithm :

[0145] ,

[0146] .

[0147] In the embodiments of the present invention, the growing extrinsic parameter continuous fine-tuning algorithm implemented by a neural network can be specifically understood as an adaptive machine learning strategy, which optimizes the transformation matrix between sensors by integrating the real-time pose information continuously accumulated during the operation of the vehicle.

[0148] In the embodiments of the present invention, the step of repeatedly obtaining the optimized coordinate transformation matrix at the current sampling moment is performed, and the number of samples of the optimized coordinate transformation matrix is maintained at a preset threshold to obtain the desired coordinate transformation matrix, as Figure 8 shown, including:

[0149] S510. Form a sample set of the optimized coordinate transformation matrix from the optimized coordinate transformation matrices at multiple sampling moments;

[0150] S520. Determine whether the number of samples in the sample set of the optimized coordinate transformation matrix reaches the preset threshold;

[0151] S530. If the preset threshold is not reached, directly add the newly obtained optimized coordinate transformation matrix at the current sampling moment to the sample set of the optimized coordinate transformation matrix;

[0152] S540. If the preset threshold is reached, delete the optimized coordinate transformation matrix at the earliest sampling moment and then add the newly obtained optimized coordinate transformation matrix at the current sampling moment to the sample set of the optimized coordinate transformation matrix;

[0153] It should be understood that, for example, if the preset threshold of the number of samples is 1000, after obtaining 1000 optimized coordinate transformation matrices at 1000 sampling moments, since sampling is still in progress, the 1001st optimized coordinate transformation matrix obtained after sampling at the 1001st moment will replace the 1st optimized coordinate transformation matrix and be added to the sample set. That is, for each newly obtained optimized coordinate transformation matrix, the optimized coordinate transformation matrix obtained at the earliest moment in the sample set will be replaced. Thus, with the continuous dynamic update of the sample set, the optimized coordinate transformation matrix at the latest moment will always be obtained. Therefore, the coordinate transformation matrix obtained after training this sample set will be more matched with the current sensor.

[0154] S550. Retrain the neural network model according to the sample set of the optimized coordinate transformation matrix to obtain a neural network model matching the current sampling moment;

[0155] S560. Obtain the desired coordinate transformation matrix according to the neural network model matching the current sampling moment.

[0156] It should be understood that as the vehicle continues to travel, the pose data captured by sensors such as GPS and IMU are used as training samples. The accumulation of these samples provides rich learning materials for the neural network. The neural network model can learn from these growing data and gradually adjust its weights and biases to more accurately predict and compensate for the extrinsic parameter errors caused by vehicle motion and environmental changes. As the vehicle continues to operate, new samples are continuously added to the sample set, and the sample set gradually increases. The model can be retrained regularly or in real time using the new sample set to adapt to vehicle motion and environmental changes.

[0157] Specifically, as Figure 9 shown, it can be assumed that at vehicle operation time , the sample set is , and at time , the new sample is added to the data, and the sample set becomes . The model is at parameter , and is updated to at time . The update rule is expressed as:

[0158] ,

[0159] ,

[0160] where represents the learning rate, which is used to control the step size of each iteration. The characteristic of this growing algorithm lies in its self-optimization ability, that is, as time goes by and the number of samples increases, the prediction accuracy and robustness of the model are significantly improved, thus achieving higher-precision sensor calibration. This method can not only adapt to the dynamic changes of the vehicle under different operating conditions, but also continuously improve its performance through continuous learning, ensuring the long-term stability and reliability of the sensor fusion system.

[0161] In summary, the multi-sensor calibration method provided by the present invention optimizes the alignment consistency feature of the image and the point cloud to achieve zero-training extrinsic parameter initial calibration and determine the initial coordinate transformation relationship between sensors; combines the real-time pose information of the road vehicle, designs a growing extrinsic parameter continuous fine-tuning method through a neural network, and dynamically adjusts the extrinsic parameters of the sensors to adapt to environmental changes; optimizes the initial calibration result through the real-time pose information and sensor data to improve the calibration accuracy and the robustness of the system; in complex weather conditions and vehicle vibration and noise environments, reduces the decline of the multi-sensor perception state and improves the perception stability through the self-growing calibration method. This multi-sensor calibration method can significantly improve the perception performance of the autonomous driving system in various environments, especially in harsh weather such as rain, fog, and snow, as well as in situations with large vehicle vibration and noise, ensuring the perception stability and accuracy of multi-type road vehicle task orientation. Through this method, autonomous driving vehicles can better adapt to different operating scenarios and improve their autonomous driving ability in complex environments such as mining areas.

[0162] As another embodiment of the present invention, a multi-sensor calibration device 100 is provided for implementing the multi-sensor calibration method described above. Among them, as Figure 10 shown, it includes:

[0163] An acquisition module 110, configured to acquire image data and point cloud data;

[0164] A preprocessing module 120, configured to preprocess both the image data and the point cloud data so that the point cloud data is projected onto the image coordinate system where the image data is located;

[0165] An iterative optimization module 130, configured to perform iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix;

[0166] A repeated acquisition of optimized coordinate transformation matrix module 140, configured to repeat the foregoing steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment;

[0167] A real-time optimization module 150, configured to repeat the step of obtaining the optimized coordinate transformation matrix at the current sampling moment and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain a desired coordinate transformation matrix.

[0168] The multi-sensor calibration device provided by the present invention utilizes the projection of point clouds on the image coordinate system to obtain the coincidence consistency score within the image segmentation region, so as to evaluate the alignment of the two. By optimizing the consistency score through gradients, zero-training external parameter initial calibration is achieved. This calibration device does not rely on an external calibration board, but optimizes the alignment situation through gradients, thereby obtaining the transformation matrix between the camera and the radar sensor. Additionally, based on the initial calibration and combined with the real-time pose information of the road vehicle, a growing external parameter continuous fine-tuning method is designed, which can adjust the external parameters of the sensor in real time according to the dynamic changes of the vehicle in different operating scenarios to adapt to environmental changes, thereby improving the stability and accuracy of the perception system.

[0169] For the description of the specific working process of the multi-sensor calibration device of the present invention, reference can be made to the description of the multi-sensor calibration method in the previous text, and details will not be elaborated here.

[0170] As another embodiment of the present invention, a multi-sensor calibration system 10 is provided, wherein, as Figure 11 shown, it includes: an image sensor 200, a lidar sensor 300, and the multi-sensor calibration device 100 described above. Both the image sensor 200 and the lidar sensor 300 are communicatively connected to the multi-sensor calibration device.

[0171] The image sensor 200 is used to collect environmental image information to form image data.

[0172] The lidar sensor 300 is used to detect environmental information through lidar signals to form point cloud data.

[0173] The multi-sensor calibration device 100 is used to achieve automatic calibration based on the image data and the point cloud data to obtain a coordinate transformation matrix.

[0174] The multi-sensor calibration system provided by the present invention adopts the multi-sensor calibration device described above, which can adjust the external parameters of the sensor in real time according to the dynamic changes of the vehicle in different operating scenarios to adapt to environmental changes, thereby improving the stability and accuracy of the perception system.

[0175] Specifically, the image sensor may specifically be a camera.

[0176] For the description of the specific working process of the multi-sensor calibration system of the present invention, reference can be made to the description of the multi-sensor calibration method in the previous text, and details will not be elaborated here.

[0177] It is understandable that the above embodiments are merely exemplary embodiments adopted to illustrate the principles of the present invention. However, the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also regarded as the protection scope of the present invention.

Claims

1. A multi-sensor calibration method, characterized in that, Including: Obtain image data and point cloud data; Preprocess both the image data and the point cloud data so that the point cloud data is projected onto the image coordinate system where the image data is located; Perform iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix; Repeat the foregoing steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment; Repeat the step of obtaining the optimized coordinate transformation matrix at the current sampling moment and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix.

2. The multi-sensor calibration method according to claim 1, wherein Preprocess both the image data and the point cloud data so that the point cloud data is projected onto the image coordinate system where the image data is located, including: Perform image semantic segmentation on the image data to obtain multiple image segmentation regions, and perform point cloud feature extraction on the point cloud data to obtain multiple point cloud features; Convert the point cloud features from the lidar coordinate system where the point cloud data is located to the image coordinate system where the image data is located according to a preset coordinate transformation matrix, so that each image segmentation region obtains multiple point cloud feature projection points; Calculate the point cloud attribute consistency of the point cloud feature projection points in each image segmentation region to obtain the projection result of the point cloud data in the image coordinate system.

3. The multi-sensor calibration method according to claim 2, wherein Calculating the point cloud attribute consistency of the point cloud feature projection points in each image segmentation region includes: Calculate the consistency scores of the intensity of the point cloud, the normal vector of the point cloud, and the segmentation category of the point cloud in each image segmentation region respectively; Determine the consistency score of the point cloud feature projection points in the image segmentation region according to the consistency score of the intensity of the point cloud, the consistency score of the normal vector of the point cloud, and the consistency score of the segmentation category of the point cloud, and use the consistency score of the point cloud feature projection points in the image segmentation region as the projection result in the image coordinate system.

4. The multi-sensor calibration method according to claim 1, wherein, Performing iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix includes: Determine the loss function of the iterative optimization algorithm according to the projection result of the point cloud data in the image coordinate system; Optimize the loss function according to the iterative optimization algorithm until a preset iterative stop condition is satisfied; Determine the corresponding optimized coordinate transformation matrix according to the loss function when the preset iterative stop condition is satisfied.

5. The multi-sensor calibration method according to claim 4, wherein, Optimizing the loss function according to the iterative optimization algorithm until a preset iterative stop condition is satisfied, including: Update the external parameter matrix along the opposite direction of the gradient according to the gradient descent algorithm to minimize the loss function; Stop updating when the preset iterative stop condition is satisfied and obtain the loss function at the time of stopping the update, where the preset iterative stop condition includes any one of the norm of the gradient descent algorithm being less than a preset norm threshold, the change of the loss function being less than a preset change threshold, and the number of iterations reaching a preset number of iterations.

6. The multi-sensor calibration method according to claim 1, wherein Repeating the foregoing steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment includes: Project the point cloud data at the current sampling moment into the image coordinate system where the image data at the current sampling moment is located according to the optimized coordinate transformation matrix at the previous sampling moment; Determine the current loss function that matches according to the projection result at the current sampling moment; Iteratively optimize the current loss function according to the pre-trained neural network model until the preset iteration stop condition is met to obtain the optimized coordinate transformation matrix at the current sampling moment.

7. The multi-sensor calibration method according to claim 6, wherein, Obtaining the pre-trained neural network model includes: Determine the training data set, where the training data set includes feature vectors and corresponding extrinsic parameter errors, and the feature vectors include vehicle operating state information, environmental data information, and sensor data information; Determine the neural network model prediction function according to the training data set; Train according to the training data set and the neural network model prediction function to minimize the loss function to obtain the pre-trained neural network model.

8. The multi-sensor calibration method according to claim 1, wherein Repeat the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment, and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix, including: Form a sample set of the optimized coordinate transformation matrix with the optimized coordinate transformation matrices at multiple sampling moments; Judge whether the number of samples in the sample set of the optimized coordinate transformation matrix reaches the preset threshold; If the preset threshold is not reached, directly add the newly obtained optimized coordinate transformation matrix at the current sampling moment to the sample set of the optimized coordinate transformation matrix; If the preset threshold is reached, delete the optimized coordinate transformation matrix at the earliest sampling moment and then add the newly obtained optimized coordinate transformation matrix at the current sampling moment to the sample set of the optimized coordinate transformation matrix; Retrain the neural network model according to the sample set of the optimized coordinate transformation matrix to obtain the neural network model that matches the current sampling moment; Obtain the desired coordinate transformation matrix according to the neural network model that matches the current sampling moment.

9. A multi-sensor calibration device for implementing the multi-sensor calibration method according to any one of claims 1 to 8, characterized in that, Includes: An acquisition module for acquiring image data and point cloud data; A preprocessing module for preprocessing both the image data and the point cloud data so that the point cloud data is projected into the image coordinate system where the image data is located; An iterative optimization module for iteratively optimizing the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix; A module for repeatedly obtaining the optimized coordinate transformation matrix, which is used to repeatedly obtain the optimized coordinate transformation matrix at the current sampling moment according to the image data and the point cloud data at the current sampling moment; A real-time optimization module for repeating the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment, and maintaining the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix.

10. A multi-sensor calibration system, characterized in that, Includes: An image sensor, a lidar sensor, and the multi-sensor calibration device according to claim 9, where the image sensor and the lidar sensor are both communicatively connected to the multi-sensor calibration device, The image sensor is used to collect environmental image information to form image data, The lidar sensor is used to detect environmental information through lidar signals to form point cloud data, The multi-sensor calibration device is used to automatically calibrate according to the image data and the point cloud data to obtain a coordinate transformation matrix.

Images