Multi-sensor calibration method, device and system

Abstract

The present invention relates to the field of multi-sensor calibration technology, and specifically discloses a multi-sensor calibration method, device, and system, comprising: acquiring image data and point cloud data; preprocessing the image data and point cloud data so that the point cloud data is projected into the image coordinate system where the image data is located; iteratively optimizing the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix; repeating the aforementioned steps based on the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment; repeating the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment, and maintaining the number of samples of the optimized coordinate transformation matrix within a preset threshold to obtain a desired coordinate transformation matrix. The multi-sensor calibration method provided by the present invention can obtain stable and accurate calibration results without the need for a calibration plate, and can adapt to environmental changes.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-sensor calibration, and in particular to a multi-sensor calibration method, a multi-sensor calibration device and a multi-sensor calibration system. Background Art

[0002] With the continuous advancement of autonomous driving technology and innovations in sensor technology, multi-sensor fusion is increasingly being used in the autonomous driving field. In heterogeneous operating scenarios, such as urban streets, highways, and mining areas, autonomous driving systems must maintain high perception stability and accuracy in a variety of complex environments. Traditional multi-sensor calibration methods rely on calibration plates or other fixed targets, which have limitations in dynamically changing environments, especially in adverse weather conditions or when vehicles are subject to high vibration and noise. Currently, multi-sensor calibration technologies are primarily designed for specific scenarios or specific vehicle types, such as small cars or intelligent robots. These methods typically include landmark-based calibration and environmental feature-based calibration. However, these methods face challenges in practical application. For example, in complex weather conditions such as rain, fog, and snow, the visibility of the calibration plate is reduced, and in the presence of high vehicle vibration and noise, sensor stability is compromised, resulting in degraded perception performance.

[0003] Therefore, how to provide a calibration method that adapts to environmental changes to improve perception performance has become a technical problem that needs to be urgently solved by those skilled in the art. Summary of the Invention

[0004] The present invention provides a multi-sensor calibration method, a multi-sensor calibration device and a multi-sensor calibration system, which solve the problem in the related art that the calibration method with a calibration plate is easily affected by the environment and thus leads to a decrease in perception performance.

[0005] As a first aspect of the present invention, a multi-sensor calibration method is provided, which includes:

[0006] Acquire image data and point cloud data;

[0007] Preprocessing the image data and the point cloud data so that the point cloud data is projected into an image coordinate system where the image data is located;

[0008] Iteratively optimizing the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix;

[0009] Repeat the above steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment;

[0010] Repeat the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment, and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix.

[0011] Furthermore, the image data and the point cloud data are both preprocessed so that the point cloud data is projected into the image coordinate system where the image data is located, including:

[0012] Performing image semantic segmentation on the image data to obtain a plurality of image segmentation regions, and performing point cloud feature extraction on the point cloud data to obtain a plurality of point cloud features;

[0013] The point cloud features are converted from the laser radar coordinate system where the point cloud data is located to the image coordinate system where the image data is located according to a preset coordinate conversion matrix, so that each image segmentation area obtains a plurality of point cloud feature projection points;

[0014] The point cloud attribute consistency of the point cloud feature projection points in each image segmentation area is calculated to obtain the projection result of the point cloud data in the image coordinate system.

[0015] Furthermore, the point cloud attribute consistency of the point cloud feature projection points in each image segmentation area is calculated, including:

[0016] Calculate the consistency score of the point cloud intensity, point cloud normal vector and point cloud segmentation category in each image segmentation area respectively;

[0017] The consistency score of the point cloud feature projection points in the image segmentation area is determined based on the consistency score of the point cloud intensity, the consistency score of the point cloud normal vector and the consistency score of the point cloud segmentation category, and the consistency score of the point cloud feature projection points in the image segmentation area is used as the projection result in the image coordinate system.

[0018] Furthermore, iterative optimization processing is performed on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix, including:

[0019] Determining a loss function of an iterative optimization algorithm based on a projection result of the point cloud data in the image coordinate system;

[0020] Optimizing the loss function according to the iterative optimization algorithm until a preset iteration stop condition is met;

[0021] The corresponding optimized coordinate transformation matrix is ​​determined according to the loss function at the preset iteration stopping condition.

[0022] Furthermore, the loss function is optimized according to the iterative optimization algorithm until a preset iteration stop condition is met, including:

[0023] According to the gradient descent algorithm, the external parameter matrix is ​​updated in the reverse direction of the gradient to minimize the loss function;

[0024] When the preset iterative stopping condition is met, the update is stopped and the loss function when the update is stopped is obtained, wherein the preset iterative stopping condition includes any one of the norm of the gradient descent algorithm being less than a preset norm threshold, the change of the loss function being less than a preset change threshold, and the number of iterations reaching a preset number of iterations.

[0025] Furthermore, the above steps are repeated according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment, including:

[0026] Projecting the point cloud data at the current sampling moment into the image coordinate system where the image data at the current sampling moment is located according to the optimized coordinate transformation matrix at the previous sampling moment;

[0027] Determine the current loss function that matches the projection result at the current sampling moment;

[0028] The current loss function is iteratively optimized according to the pre-trained neural network model until the preset iteration stop condition is met to obtain the optimized coordinate transformation matrix at the current sampling moment.

[0029] Furthermore, obtaining a pre-trained neural network model includes:

[0030] Determine a training data set, the training data set including a feature vector and a corresponding extrinsic parameter error, the feature vector including vehicle operating state information, environmental data information, and sensor data information;

[0031] Determine the neural network model prediction function based on the training data set;

[0032] Training is performed based on the training data set and the neural network model prediction function to minimize the loss function and obtain a pre-trained neural network model.

[0033] Furthermore, the step of obtaining the optimized coordinate transformation matrix at the current sampling moment is repeated, and the number of samples of the optimized coordinate transformation matrix is ​​maintained at a preset threshold to obtain the desired coordinate transformation matrix, including:

[0034] The optimized coordinate transformation matrices at multiple sampling moments are used to form a sample set of optimized coordinate transformation matrices;

[0035] Determining whether the number of samples in the sample set of the optimized coordinate transformation matrix reaches a preset threshold;

[0036] If the preset threshold is not reached, the newly obtained optimized coordinate transformation matrix of the current sampling moment is directly added to the sample set of the optimized coordinate transformation matrix;

[0037] If the preset threshold is reached, the optimized coordinate transformation matrix of the earliest sampling moment is deleted and then the newly obtained optimized coordinate transformation matrix of the current sampling moment is added to the sample set of the optimized coordinate transformation matrix;

[0038] Retraining the neural network model based on the sample set of the optimized coordinate transformation matrix to obtain a neural network model that matches the current sampling moment;

[0039] The expected coordinate transformation matrix is ​​obtained according to the neural network model matching the current sampling moment.

[0040] As another aspect of the present invention, a multi-sensor calibration device is provided for implementing the multi-sensor calibration method described above, comprising:

[0041] Acquisition module, used to acquire image data and point cloud data;

[0042] a preprocessing module, configured to preprocess both the image data and the point cloud data so as to project the point cloud data into an image coordinate system where the image data is located;

[0043] an iterative optimization module, configured to perform iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix;

[0044] Repeatedly obtain the optimized coordinate transformation matrix module, used to repeat the above steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment;

[0045] The real-time optimization module is used to repeatedly obtain the steps of optimizing the coordinate transformation matrix at the current sampling moment, and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix.

[0046] As another aspect of the present invention, a multi-sensor calibration system is provided, comprising: an image sensor, a lidar sensor, and the multi-sensor calibration device described above, wherein the image sensor and the lidar sensor are both communicatively connected to the multi-sensor calibration device.

[0047] The image sensor is used to collect environmental image information to form image data.

[0048] The laser radar sensor is used to detect environmental information through laser radar signals to form point cloud data.

[0049] The multi-sensor calibration device is used to automatically calibrate the image data and point cloud data to obtain a coordinate transformation matrix.

[0050] The multi-sensor calibration method provided by the present invention obtains the projection result of the point cloud data onto the image coordinate system by preprocessing the image data and the point cloud data, and optimizes the projection result to obtain the optimized coordinate conversion matrix, and finally optimizes and adjusts the obtained multiple optimized coordinate conversion matrices to obtain a stable and accurate coordinate conversion matrix. This multi-sensor calibration method of the present invention does not need to rely on an external calibration plate. On the basis of the initial calibration, the conversion matrix between the camera and the radar sensor is obtained by iterative optimization processing of the initial calibration. This multi-sensor calibration method of the present invention can adjust the external parameters of the sensor in real time according to the dynamic changes of the vehicle in different operating scenarios to adapt to changes in the environment, thereby improving the stability and accuracy of the perception system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0052] Figure 1 This is a flow chart of the multi-sensor calibration method provided by the present invention.

[0053] Figure 2 This is a flowchart for preprocessing image data and point cloud data provided by the present invention.

[0054] Figure 3 A flowchart of calculating the consistency of point cloud attributes in an image segmentation area provided by the present invention.

[0055] Figure 4 A schematic diagram of the consistency scores of image segmentation regions provided by the present invention.

[0056] Figure 5 This is a flowchart of the iterative optimization process for projection results provided by the present invention.

[0057] Figure 6 This is a flowchart of the optimization process of the loss function provided by the present invention.

[0058] Figure 7 The present invention provides a flow chart for repeatedly obtaining the optimized coordinate transformation matrix at the current sampling moment.

[0059] Figure 8 The present invention provides a flow chart for obtaining the desired coordinate transformation matrix.

[0060] Figure 9 This is a schematic diagram of the multi-sensor calibration provided by the present invention.

[0061] Figure 10 This is a structural block diagram of the multi-sensor calibration device provided by the present invention.

[0062] Figure 11 This is a structural block diagram of the multi-sensor calibration system provided by the present invention. DETAILED DESCRIPTION

[0063] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0064] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0065] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0066] In this embodiment, a multi-sensor calibration method is provided. Figure 1 is a flow chart of a multi-sensor calibration method according to an embodiment of the present invention. Figure 1 Shown, including:

[0067] S100, acquiring image data and point cloud data;

[0068] In an embodiment of the present invention, the image information of the vehicle's surrounding environment can be collected by an image sensor to form image data, and the vehicle's surrounding environment can be detected by a lidar sensor to form point cloud data.

[0069] S200, preprocessing the image data and the point cloud data so that the point cloud data is projected into an image coordinate system where the image data is located;

[0070] In an embodiment of the present invention, the image data and the point cloud data are preprocessed separately so that the point cloud data can be projected in the image coordinate system in which the image data resides. It should be understood that the initial projection of the point cloud data in the image coordinate system can be achieved by presetting a coordinate transformation matrix.

[0071] S300, performing iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix;

[0072] It should be understood that a projection consistency score calculation can be performed for the projection of the point cloud data in the image coordinate system, that is, the matching between the point cloud data and the image data is evaluated after the point cloud data is projected into the image coordinate system, and iterative optimization processing is performed based on the evaluation results to narrow the gap between the point cloud data and the image data in the image coordinate system.

[0073] S400, repeating the above steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment;

[0074] Specifically, the optimized coordinate transformation matrix at the current sampling moment can be obtained by repeating the aforementioned steps. It should be understood that the optimized coordinate transformation matrix at the current sampling moment is obtained by optimizing the optimized coordinate transformation matrix at the previous sampling moment.

[0075] It should be noted here that for the initial projection, the preset coordinate transformation matrix is ​​used, and then the preset coordinate transformation matrix is ​​optimized at the second sampling moment to obtain the optimized coordinate transformation matrix. By analogy, the optimized coordinate transformation matrices at subsequent sampling moments are all obtained after optimizing the optimized coordinate transformation matrix at the previous sampling moment.

[0076] S500 , repeating the step of obtaining the optimized coordinate transformation matrix at the current sampling moment, and maintaining the number of samples of the optimized coordinate transformation matrix at a preset threshold, to obtain the desired coordinate transformation matrix.

[0077] It should be understood that by repeatedly obtaining the optimized coordinate transformation matrix at the current sampling moment, the optimized coordinate transformation matrices at multiple sampling moments can be obtained, and the optimized coordinate transformation matrices at multiple sampling moments are adjusted in real time to maintain the number of samples within a preset threshold, and finally an accurate and stable coordinate transformation matrix is ​​obtained.

[0078] In summary, the multi-sensor calibration method provided by the present invention obtains the projection result of the point cloud data onto the image coordinate system by preprocessing the image data and the point cloud data, and optimizes the projection result to obtain the optimized coordinate transformation matrix, and finally optimizes and adjusts the obtained multiple optimized coordinate transformation matrices to obtain a stable and accurate coordinate transformation matrix. This multi-sensor calibration method of the present invention does not need to rely on an external calibration plate. On the basis of the initial calibration, the conversion matrix between the camera and the radar sensor is obtained by iterative optimization processing of the initial calibration. This multi-sensor calibration method of the present invention can adjust the external parameters of the sensor in real time according to the dynamic changes of the vehicle in different operating scenarios to adapt to changes in the environment, thereby improving the stability and accuracy of the perception system.

[0079] In an embodiment of the present invention, the image data and the point cloud data are both pre-processed so that the point cloud data is projected to the image coordinate system where the image data is located, such as Figure 2 Shown, including:

[0080] S210, performing image semantic segmentation on the image data to obtain a plurality of image segmentation regions, and performing point cloud feature extraction on the point cloud data to obtain a plurality of point cloud features;

[0081] It should be understood that the image data is segmented into image semantics, and the segmented image area is specifically The preprocessing of point cloud data can specifically include normal estimation, intensity normalization and segmentation categories.

[0082] S220, converting the point cloud features from the laser radar coordinate system where the point cloud data is located to the image coordinate system where the image data is located according to a preset coordinate conversion matrix, so that each image segmentation area obtains a plurality of point cloud feature projection points;

[0083] Specifically, each point in the point cloud is transformed from the LiDAR coordinate system to the image coordinate system, and the consistency of the point cloud attributes within each segmented region is calculated, including the point cloud's intensity, normal vector, and segmentation category. A consistency score is calculated to evaluate the alignment between the image and point cloud.

[0084] In an embodiment of the present invention, the preset coordinate transformation matrix when the point cloud data is projected into the image coordinate system is expressed as:

[0085] ,

[0086] in, Represents a scaling factor used to convert homogeneous coordinates to regular two-dimensional coordinates. represents the intrinsic parameter matrix of the camera, Represents the empirically estimated extrinsic parameter matrix of the lidar to camera, including rotation and translation. , represents the position of point p in the laser radar coordinate system, Represents the position of point p in the image coordinate system.

[0087] S230 , calculating the point cloud attribute consistency of the point cloud feature projection points in each image segmentation area, and obtaining the projection result of the point cloud data in the image coordinate system.

[0088] In the embodiment of the present invention, specifically Figure 3 As shown, the point cloud attribute consistency of the point cloud feature projection points in each image segmentation area is calculated, including:

[0089] S231, respectively calculating the intensity of the point cloud, the normal vector of the point cloud, and the consistency score of the segmentation category of the point cloud in each image segmentation area;

[0090] S232. Determine the consistency score of the point cloud feature projection points in the image segmentation area according to the consistency score of the point cloud intensity, the consistency score of the point cloud normal vector, and the consistency score of the point cloud segmentation category, and use the consistency score of the point cloud feature projection points in the image segmentation area as the projection result in the image coordinate system.

[0091] Specifically, for each image segmentation region , we can get a set of points falling on it :

[0092] ,

[0093] Here, Represents a binary function, when point p falls in the image segmentation area When , the function value is 1.

[0094] In the embodiment of the present invention, the formula for calculating the consistency of reflectivity (i.e., the intensity of the point cloud) is:

[0095] ,

[0096] in, Represents the intensity values ​​of all points in the point set;

[0097] The formula for calculating normal vector consistency is:

[0098] ,

[0099] in, Represents a point set The normal vector of the jth point in , Represents the average direction of all normal vectors. Indicates the number of points in the point cloud.

[0100] For each segmentation category in the point cloud, we first calculate the number of points in each category and then sort these points from large to small. The sorted points are expressed as , , ..., therefore, the calculation formula for segmentation category consistency is:

[0101] ,

[0102] ,

[0103] in, represents the number of points in the i-th category, Represents the total number of point clouds of all categories, Represents a weighting factor that reduces the contribution of a category to the total consistency score as the number of category points decreases.

[0104] In summary, the measured point set The consistency score formula is:

[0105] ,

[0106] in, represents the weight coefficient, corresponding to reflectivity consistency, normal vector consistency and segmentation category consistency respectively; Represents a point set The reflectivity consistency score, Represents a point set The normal vector consistency score is Represents a point set The segmentation category consistency score.

[0107] The final consistency score s is calculated as follows:

[0108] ,

[0109] in, Indicates the The weight of the image segmentation region, the consistency score It is The consistency score of the image segmentation area is as follows: Figure 4 The above-mentioned calculated consistency score comprehensively considers the consistency of the intensity, normal vector and segmentation category of the point cloud to evaluate the alignment between the image and the point cloud, and is used to optimize the external parameters.

[0110] It should be understood that the consistency score is mainly used to indicate the consistency difference between the point cloud features projected into the image coordinate system and the image data in the image coordinate system. Since the projection is based on the coordinate transformation matrix, the consistency difference can characterize the difference between the coordinate transformation matrix used in the projection and the desired coordinate transformation matrix. Then, by optimizing the consistency difference, the current coordinate transformation matrix can be optimized to obtain the desired coordinate transformation matrix.

[0111] In an embodiment of the present invention, the projection result of the point cloud data in the image coordinate system is iteratively optimized to obtain an optimized coordinate transformation matrix, such as Figure 5 Shown, including:

[0112] S310, determining a loss function of an iterative optimization algorithm according to a projection result of the point cloud data in the image coordinate system;

[0113] It should be understood that the embodiments of the present invention utilize a gradient descent algorithm to optimize the extrinsic parameter matrix, achieving zero-training extrinsic parameter initial calibration without the need for a calibration plate, thereby providing a precise, optimized coordinate transformation relationship for the multi-sensor system. Specifically, a loss function can be determined for the projection result, i.e., the consistency score described above. Because the consistency score indicates the difference in consistency between the point cloud features projected into the image coordinate system and the image data in the image coordinate system, a matching loss function is determined based on this consistency difference, and the loss function is then iteratively optimized.

[0114] In the embodiment of the present invention, using the gradient descent algorithm to approximate the optimal transformation matrix (external parameter matrix) is an iterative optimization process. The loss function is defined as the sum of the consistency scores of all image segmentation regions:

[0115] ,

[0116] ,

[0117] in, Represents the external parameter matrix elements.

[0118] S320, optimizing the loss function according to the iterative optimization algorithm until a preset iteration stop condition is met;

[0119] In the embodiment of the present invention, Figure 6 As shown, the loss function is optimized according to the iterative optimization algorithm until a preset iteration stop condition is met, including:

[0120] S321, updating the external parameter matrix according to the gradient descent algorithm and in the reverse direction of the gradient to minimize the loss function;

[0121] S322. Stop updating when the preset iteration stopping condition is met, and obtain the loss function when stopping updating, wherein the preset iteration stopping condition includes any one of the norm of the gradient descent algorithm being less than a preset norm threshold, the change of the loss function being less than a preset change threshold, and the number of iterations reaching a preset number of iterations.

[0122] Specifically, the gradient descent algorithm minimizes the loss function by updating the parameters in the opposite direction of the gradient. The update rule is as follows:

[0123] ,

[0124] in, Represents the learning rate, a positive scalar that controls the step size of each iteration. The process of calculating the gradient and updating the parameters is repeated until the stopping condition is met. The specific stopping condition of the iteration may include the norm of the gradient being less than a certain threshold, or the change of the loss function being less than a certain threshold, or reaching a preset number of iterations. Finally, when the gradient descent algorithm converges, the obtained external parameter matrix It is the transformation matrix from the laser radar to the camera coordinate system.

[0125] S330. Determine the corresponding optimized coordinate transformation matrix according to the loss function when the iteration stop condition is preset.

[0126] It should be understood that the coordinate transformation matrix corresponding to the loss function obtained by iteratively updating the loss function until the iteration stopping condition is met can be understood as the desired optimized coordinate transformation matrix mentioned above. In other words, when the loss function is minimized, the consistency score is minimized, which in turn indicates that the difference between the point cloud features projected into the image coordinate system and the image features is minimized. At this point, the corresponding coordinate transformation matrix is ​​the desired coordinate transformation matrix.

[0127] In an embodiment of the present invention, the above steps are repeated according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment, such as Figure 7 Shown, including:

[0128] S410, projecting the point cloud data at the current sampling moment into the image coordinate system where the image data at the current sampling moment is located according to the optimized coordinate transformation matrix at the previous sampling moment;

[0129] S420, determining a current loss function that matches the projection result at the current sampling moment;

[0130] S430. Perform iterative optimization on the current loss function according to the pre-trained neural network model until a preset iteration stop condition is met to obtain the optimized coordinate transformation matrix at the current sampling moment.

[0131] Specifically, obtaining a pre-trained neural network model includes:

[0132] 1) Determining a training data set, wherein the training data set includes a feature vector and a corresponding extrinsic parameter error, wherein the feature vector includes vehicle operating state information, environmental data information, and sensor data information;

[0133] 2) Determine the neural network model prediction function based on the training data set;

[0134] 3) Training is performed based on the training data set and the neural network model prediction function to minimize the loss function and obtain a pre-trained neural network model.

[0135] It should be understood that for the pre-trained neural network model, it is necessary to complete the data fusion of IMU and GPS sensors, vehicle state estimation and parameter update.

[0136] Specifically, the vehicle runs to save the data sample. Now assume that the initial data sample is , each sample contains the feature vector and the corresponding extrinsic parameter error :

[0137] ,

[0138] Eigenvector Constructed by:

[0139] .

[0140] The sensor data includes the received reflectivity consistency score, normal vector consistency score, and segmentation category consistency score; the vehicle status includes the three-axis angular velocity statistical features and the three-axis acceleration statistical features; the environmental data includes highways, urban roads, and unstructured roads.

[0141] Select the linear regression model, training data set and model prediction function can be expressed as:

[0142] ,

[0143] ,

[0144] The training process is achieved by minimizing the loss function, which is solved by the gradient descent optimization algorithm. :

[0145] ,

[0146] .

[0147] In an embodiment of the present invention, the growing extrinsic parameter continuous fine-tuning algorithm implemented using a neural network can be specifically understood as an adaptive machine learning strategy, which optimizes the conversion matrix between sensors by integrating the real-time posture information continuously accumulated during the operation of the vehicle.

[0148] In an embodiment of the present invention, the step of obtaining the optimized coordinate transformation matrix at the current sampling moment is repeated, and the number of samples of the optimized coordinate transformation matrix is ​​maintained at a preset threshold, so as to obtain the desired coordinate transformation matrix, such as Figure 8 Shown, including:

[0149] S510, forming a sample set of optimized coordinate transformation matrices from the optimized coordinate transformation matrices at multiple sampling moments;

[0150] S520, determining whether the number of samples in the sample set of the optimized coordinate transformation matrix reaches a preset threshold;

[0151] S530: If the preset threshold is not reached, the newly acquired optimized coordinate transformation matrix of the current sampling moment is directly added to the sample set of the optimized coordinate transformation matrix;

[0152] S540: If the preset threshold is reached, the optimized coordinate transformation matrix of the earliest sampling moment is deleted and then the newly obtained optimized coordinate transformation matrix of the current sampling moment is added to the sample set of the optimized coordinate transformation matrix;

[0153] It should be understood that, for example, if the preset threshold value of the number of samples is 1000, then after obtaining 1000 optimized coordinate transformation matrices at 1000 sampling moments, since sampling is still in progress, the 1001th optimized coordinate transformation matrix obtained after sampling at time 1001 will replace the first optimized coordinate transformation matrix and be added to the sample set, that is, each subsequent optimized coordinate transformation matrix will replace the optimized coordinate transformation matrix obtained at the earliest moment in the sample set. As a result, as the sample set is continuously and dynamically updated, the optimized coordinate transformation matrix at the latest moment will always be obtained. Therefore, the coordinate transformation matrix obtained after the sample set is trained will better match the current sensor.

[0154] S550, retraining the neural network model according to the sample set of the optimized coordinate transformation matrix to obtain a neural network model that matches the current sampling moment;

[0155] S560: Obtain an expected coordinate transformation matrix according to a neural network model that matches the current sampling moment.

[0156] It should be understood that as the vehicle continues to drive, the pose data captured by sensors such as GPS and IMU is used as training samples. The accumulation of these samples provides a rich source of learning material for the neural network. The neural network model can learn from this growing data and gradually adjust its weights and biases to more accurately predict and compensate for extrinsic parameter errors caused by vehicle motion and environmental changes. As the vehicle continues to operate, new samples are continuously added to the sample set, gradually increasing its size. The model can be retrained using new sample sets periodically or in real time to adapt to vehicle motion and environmental changes.

[0157] Specifically, if Figure 9 As shown, it can be assumed that during the vehicle running time When , the sample set is , in time When the new sample Add data, the sample set becomes , the model has parameters At that time , in time Updated to , the update rule is expressed as:

[0158] ,

[0159] ,

[0160] in, represents the learning rate, which controls the step size of each iteration. This growing algorithm is characterized by its self-optimizing capability. As time passes and the number of samples increases, the model's prediction accuracy and robustness significantly improve, enabling more accurate sensor calibration. This approach not only adapts to the dynamic changes of the vehicle under varying operating conditions but also continuously improves its performance through continuous learning, ensuring the long-term stability and reliability of the sensor fusion system.

[0161] In summary, the multi-sensor calibration method provided by the present invention utilizes the alignment consistency feature optimization of images and point clouds to achieve zero-training external parameter initial calibration and determine the initial coordinate transformation relationship between sensors; combined with the real-time posture information of road vehicles, a continuous fine-tuning method of growth-type external parameters is designed through a neural network to dynamically adjust the external parameters of the sensor to adapt to environmental changes; the initial calibration results are optimized through real-time posture information and sensor data to improve the calibration accuracy and system robustness; in complex weather conditions and vehicle vibration and noise environments, the self-growth calibration method is used to reduce the degradation of the multi-sensor perception state and improve the perception stability. This multi-sensor calibration method can significantly improve the perception performance of the autonomous driving system in a variety of environments, especially in severe weather such as rain, fog, and snow, and in conditions with large vehicle vibration and noise, ensuring the task-oriented perception stability and accuracy of multiple road vehicles. Through this method, autonomous driving vehicles can better adapt to different operating scenarios and improve their autonomous driving capabilities in complex environments such as mining areas.

[0162] As another embodiment of the present invention, a multi-sensor calibration device 100 is provided for implementing the multi-sensor calibration method described above, wherein: Figure 10 Shown, including:

[0163] An acquisition module 110 is used to acquire image data and point cloud data;

[0164] a preprocessing module 120 for preprocessing both the image data and the point cloud data so that the point cloud data is projected into an image coordinate system where the image data is located;

[0165] An iterative optimization module 130 is configured to perform iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix;

[0166] Repeatedly obtain optimized coordinate conversion matrix module 140, for repeating the above steps to obtain the optimized coordinate conversion matrix at the current sampling moment according to the image data and point cloud data at the current sampling moment;

[0167] The real-time optimization module 150 is used to repeatedly obtain the step of optimizing the coordinate transformation matrix at the current sampling moment, and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix.

[0168] The multi-sensor calibration device provided by this invention utilizes the projection of a point cloud onto an image coordinate system to derive an overlap consistency score within the image segmentation region to assess alignment. This score is then gradient-optimized to achieve initial calibration with zero-training extrinsic parameters. This calibration device does not rely on an external calibration plate, but instead uses gradient optimization to determine the transformation matrix between the camera and radar sensors. Furthermore, based on this initial calibration, a continuous fine-tuning method for growing extrinsic parameters is designed, incorporating real-time positional information from road vehicles. This method adjusts sensor extrinsic parameters in real time based on the dynamic changes of the vehicle in different operating scenarios to adapt to environmental changes, thereby improving the stability and accuracy of the perception system.

[0169] The description of the specific working process of the multi-sensor calibration device of the present invention can refer to the description of the multi-sensor calibration method above, which will not be repeated here.

[0170] As another embodiment of the present invention, a multi-sensor calibration system 10 is provided, wherein Figure 11 As shown, it includes: an image sensor 200, a laser radar sensor 300 and the multi-sensor calibration device 100 mentioned above, and the image sensor 200 and the laser radar sensor 300 are both communicatively connected to the multi-sensor calibration device.

[0171] The image sensor 200 is used to collect environmental image information to form image data.

[0172] The laser radar sensor 300 is used to detect environmental information through laser radar signals to form point cloud data.

[0173] The multi-sensor calibration device 100 is used to automatically calibrate the image data and point cloud data to obtain a coordinate transformation matrix.

[0174] The multi-sensor calibration system provided by the present invention adopts the multi-sensor calibration device mentioned above, and can adjust the external parameters of the sensors in real time according to the dynamic changes of the vehicle in different operating scenarios to adapt to changes in the environment, thereby improving the stability and accuracy of the perception system.

[0175] Specifically, the image sensor may be a camera.

[0176] The description of the specific working process of the multi-sensor calibration system of the present invention can refer to the description of the multi-sensor calibration method above, which will not be repeated here.

[0177] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-sensor calibration method, characterized in that: include: Acquire image data and point cloud data; Preprocessing the image data and the point cloud data so that the point cloud data is projected into an image coordinate system where the image data is located; Iteratively optimizing the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix; Repeat the above steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment; Repeat the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment, and maintain the number of samples of the optimized coordinate transformation matrix within a preset threshold to obtain the desired coordinate transformation matrix; Repeat the above steps based on the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment, including: Projecting the point cloud data at the current sampling moment into the image coordinate system where the image data at the current sampling moment is located according to the optimized coordinate transformation matrix at the previous sampling moment; Determine the current loss function that matches the projection result at the current sampling moment; Iteratively optimize the current loss function according to the pre-trained neural network model until the preset iteration stop condition is met to obtain the optimized coordinate transformation matrix at the current sampling moment; Repeating the steps of obtaining the optimized coordinate transformation matrix at the current sampling moment and maintaining the number of samples of the optimized coordinate transformation matrix within a preset threshold to obtain the desired coordinate transformation matrix includes: The optimized coordinate transformation matrices at multiple sampling moments are used to form a sample set of optimized coordinate transformation matrices; Determining whether the number of samples in the sample set of the optimized coordinate transformation matrix reaches a preset threshold; If the preset threshold is not reached, the newly obtained optimized coordinate transformation matrix of the current sampling moment is directly added to the sample set of the optimized coordinate transformation matrix; If the preset threshold is reached, the optimized coordinate transformation matrix of the earliest sampling moment is deleted and then the newly obtained optimized coordinate transformation matrix of the current sampling moment is added to the sample set of the optimized coordinate transformation matrix; Retraining the neural network model based on the sample set of the optimized coordinate transformation matrix to obtain a neural network model that matches the current sampling moment; The expected coordinate transformation matrix is ​​obtained according to the neural network model matching the current sampling moment.

2. The multi-sensor calibration method according to claim 1, characterized in that: Preprocessing the image data and the point cloud data so that the point cloud data is projected into the image coordinate system where the image data is located includes: Performing image semantic segmentation on the image data to obtain a plurality of image segmentation regions, and performing point cloud feature extraction on the point cloud data to obtain a plurality of point cloud features; The point cloud features are converted from the laser radar coordinate system where the point cloud data is located to the image coordinate system where the image data is located according to a preset coordinate conversion matrix, so that each image segmentation area obtains a plurality of point cloud feature projection points; The point cloud attribute consistency of the point cloud feature projection points in each image segmentation area is calculated to obtain the projection result of the point cloud data in the image coordinate system.

3. The multi-sensor calibration method according to claim 2, characterized in that: Calculate the point cloud attribute consistency of the point cloud feature projection points in each image segmentation area, including: Calculate the consistency score of the point cloud intensity, point cloud normal vector and point cloud segmentation category in each image segmentation area respectively; The consistency score of the point cloud feature projection points in the image segmentation area is determined based on the consistency score of the point cloud intensity, the consistency score of the point cloud normal vector and the consistency score of the point cloud segmentation category, and the consistency score of the point cloud feature projection points in the image segmentation area is used as the projection result in the image coordinate system.

4. The multi-sensor calibration method according to claim 1, characterized in that: Iteratively optimizing the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix includes: Determining a loss function of an iterative optimization algorithm based on a projection result of the point cloud data in the image coordinate system; Optimizing the loss function according to the iterative optimization algorithm until a preset iteration stop condition is met; The corresponding optimized coordinate transformation matrix is ​​determined according to the loss function at the preset iteration stopping condition.

5. The multi-sensor calibration method according to claim 4, characterized in that: Optimizing the loss function according to the iterative optimization algorithm until a preset iteration stop condition is met, including: According to the gradient descent algorithm, the external parameter matrix is ​​updated in the reverse direction of the gradient to minimize the loss function; When the preset iterative stopping condition is met, the update is stopped and the loss function when the update is stopped is obtained, wherein the preset iterative stopping condition includes any one of the norm of the gradient descent algorithm being less than a preset norm threshold, the change of the loss function being less than a preset change threshold, and the number of iterations reaching a preset number of iterations.

6. The multi-sensor calibration method according to claim 1, characterized in that: Obtaining pre-trained neural network models includes: Determine a training data set, the training data set including a feature vector and a corresponding extrinsic parameter error, the feature vector including vehicle operating state information, environmental data information, and sensor data information; Determine the neural network model prediction function based on the training data set; Training is performed based on the training data set and the neural network model prediction function to minimize the loss function and obtain a pre-trained neural network model.

7. A multi-sensor calibration device, used to implement the multi-sensor calibration method according to any one of claims 1 to 6, characterized in that: include: Acquisition module, used to acquire image data and point cloud data; a preprocessing module, configured to preprocess both the image data and the point cloud data so as to project the point cloud data into an image coordinate system where the image data is located; an iterative optimization module, configured to perform iterative optimization processing on the projection result of the point cloud data in the image coordinate system to obtain an optimized coordinate transformation matrix; Repeatedly obtain the optimized coordinate transformation matrix module, used to repeat the above steps according to the image data and point cloud data at the current sampling moment to obtain the optimized coordinate transformation matrix at the current sampling moment; The real-time optimization module is used to repeatedly obtain the steps of optimizing the coordinate transformation matrix at the current sampling moment, and maintain the number of samples of the optimized coordinate transformation matrix at a preset threshold to obtain the desired coordinate transformation matrix.

8. A multi-sensor calibration system, characterized in that: include: An image sensor, a laser radar sensor, and the multi-sensor calibration device according to claim 7, wherein the image sensor and the laser radar sensor are both communicatively connected to the multi-sensor calibration device, The image sensor is used to collect environmental image information to form image data. The laser radar sensor is used to detect environmental information through laser radar signals to form point cloud data. The multi-sensor calibration device is used to automatically calibrate the image data and point cloud data to obtain a coordinate transformation matrix.

Images