Parameter calibration method, system and device based on multi-modal data fusion and medium

Abstract

This invention belongs to the field of sensor calibration technology and discloses an extrinsic parameter calibration method based on multimodal data fusion, comprising the following steps: S1, acquiring image data collected by a camera and point cloud data collected by a radar; S2, projecting the point cloud data onto the image space according to the projection matrix obtained from the initial calibration of the camera and radar, generating a depth map containing depth information; S3, constructing and training an extrinsic parameter calibration model; feeding the image data and the generated depth map as input data into the trained extrinsic parameter calibration model, and having the model output the extrinsic parameter prediction result of the camera relative to the radar, thereby completing the extrinsic parameter calibration of the camera and radar. This invention can simultaneously calibrate the extrinsic parameters of multiple sensors on a vehicle, obtaining all extrinsic parameter parameters at once.

Description

Technical Field

[0001] This invention belongs to the fields of autonomous driving and sensor calibration technology, and particularly relates to external parameter calibration methods for various types of sensors such as visible light cameras, infrared cameras, lidar, and millimeter-wave radar. Background Technology

[0002] For various types of sensors, including visible light cameras, infrared cameras, lidar, and millimeter-wave radar, there are currently three calibration methods: The first method is a checkerboard-based calibration method for LiDAR and camera extrinsic parameters. The checkerboard calibration board typically uses a non-deformable material as the base, on which a flat, alternating black and white checkerboard pattern is laser-printed. This type of calibration board is mainly used for high-precision calibration of visible light and multi-beam LiDAR.

[0003] The calibration method is as follows: taking advantage of the different reflection intensities of laser light for white and black, multiple intersection points between black and white grids are extracted. The visible light camera also uses the difference in color blocks to extract corner points. By calculating the matching lidar-camera corner point pair PnP equation, the extrinsic parameters are solved.

[0004] However, this method is only suitable for extrinsic parameter calibration between high-resolution cameras and multi-beam LiDAR. For low-beam LiDAR, the sparse laser points from the reflections cannot form a dense reflective surface, leading to decreased accuracy. Furthermore, the checkerboard pattern can result in unclear edges and distortion due to lighting conditions. Additionally, this method cannot meet the calibration requirements of mobile platforms equipped with multiple sensors.

[0005] The second method is a camera and millimeter-wave radar extrinsic parameter calibration method based on corner reflectors. Corner reflectors are usually designed with materials that are sensitive to millimeter-wave radar, such as metal or alloys, and use three isosceles right-angled triangular metal plates spliced ​​together to form a triangular pyramid with three mutually perpendicular faces.

[0006] The device's reflection principle utilizes the characteristic that millimeter waves are reflected at the incident angle on the reflector, maximizing the reflection intensity for sensing. This allows the extraction of the position information of the center corner point of the corner reflector, which is then compared with the reflector corner point identified by the camera to form a 3D-2D point pair. Multiple point pairs are used to solve the PnP equation, thereby obtaining the extrinsic parameters.

[0007] This method is applicable to the extrinsic parameter calibration of cameras and millimeter-wave radars, but requires knowledge of the camera's intrinsic parameters. Conventional corner reflectors also cannot meet the calibration requirements of lidar and millimeter-wave radars, and cannot be used for the extrinsic parameter calibration of mobile platforms equipped with multiple sensors.

[0008] The third approach is to find point features, line features, and surface features directly from the environment or road without using manually designed calibration objects, and then pair them to obtain extrinsic parameters. However, this method cannot stably solve for extrinsic parameters because environmental information cannot remain absolutely static, resulting in relatively large errors, and is currently not applicable to practical applications.

[0009] The problems with the above methods are as follows: ① Multi-sensor calibration requires various specialized calibration devices, which cannot be solved in a unified way.

[0010] ② Due to prolonged use, bumpy roads, or insecure installation, the external parameters of the multiple sensors on various platforms have changed, requiring recalibration.

[0011] ③ The calibration process is complex and requires repeated calibration by professionals, which is time-consuming and laborious for users.

[0012] ④ The features used for sensor calibration (such as line features and obstacle contour features) are unstable, and the parameters calibrated in real time have large errors, which cannot meet the multi-sensor fusion requirements of daily platforms.

[0013] ⑤ Traditional calibration methods require high-precision calibration tools and cannot monitor external parameter errors in real time.

[0014] ⑥ Existing calibration methods are all one-to-one (i.e., one camera and one radar are calibrated), but there are a large number of sensors on a vehicle, and calibration using existing methods is very time-consuming. Summary of the Invention

[0015] The technical problem to be solved by this invention is that traditional calibration devices have poor versatility and cannot meet the multi-sensor fusion requirements of daily platforms.

[0016] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows: An extrinsic parameter calibration method based on multimodal data fusion includes the following steps: S1. Acquire image data captured by the camera and point cloud data captured by the radar; S2. Based on the projection matrix obtained from the initial calibration of the camera and radar, the point cloud data is projected onto the image space to generate a depth map containing depth information. S3. Construct and train an extrinsic parameter calibration model; feed the image data and the generated depth map as input data into the trained extrinsic parameter calibration model, and the model outputs the extrinsic parameter prediction results of the camera relative to the radar, thereby completing the extrinsic parameter calibration of the camera and the radar.

[0017] Furthermore, in step S2, each projected point corresponds to a three-dimensional feature (d, u, v), where d represents the depth value of the point, and u and v represent the horizontal and vertical coordinates of the point on the image plane, respectively.

[0018] Furthermore, the training steps of the extrinsic calibration model include: A cross-modal cross-enhancement module and a multi-head calibration network are constructed using convolutional layers, batch normalization layers, linear connection layers, and pooling layers. Basic deep neural networks are constructed for image data and point cloud data using convolutional layers, batch normalization layers, linear rectified layers, and pooling layers. A cross-modal cross-enhancement module and a multi-head calibration network are inserted into the basic deep neural network to construct a calibration network that fuses point cloud data and image data.

[0019] Furthermore, the cross-modal cross-enhancement module achieves feature fusion through the following fusion feature expression:

[0020] in, These represent the multi-scale features of the image and depth map, respectively. Indicates attention operation, This indicates an element-by-element concatenation operation. This represents matrix multiplication.

[0021] Furthermore, the number of calibration heads can be configured according to the number of cameras and radars to achieve one-to-one, many-to-many, one-to-many, and many-to-one camera-radar combination calibration.

[0022] Furthermore, the objective loss function expression of the extrinsic parameter calibration model is:

[0023]

[0024] in, L Indicates loss, This represents the point cloud loss weights. These represent the point cloud loss and the extrinsic parameter matrix loss, respectively. These represent translation loss, rotation loss, and matrix loss, respectively. These represent the translation loss weight, rotation loss weight, and matrix loss weight, respectively.

[0025] Furthermore, high-precision calibration plates or corner reflectors are used to obtain accurate initial extrinsic calibration parameters and camera intrinsic parameters.

[0026] The present invention also provides an extrinsic parameter calibration system based on multimodal data fusion, comprising: a camera module, a radar module, a depth map mapping module, and a calibration module; Camera module: Used to acquire image data; Radar module: Used to acquire point cloud data from the radar; Depth map mapping module: used to map point cloud data to image space based on image data to generate a depth map; Calibration module: Used to input the mapped data and image data into the extrinsic parameter calibration model for processing, obtain extrinsic parameter prediction results, and realize the extrinsic parameter calibration of camera-LiDAR.

[0027] The present invention also provides an extrinsic parameter calibration device based on multimodal data fusion, comprising: At least one processor and at least one memory; the memory is used to store at least one program; When the program is executed by the processor, the processor performs the method according to any one of claims 1-7.

[0028] The present invention also provides a storage medium storing a processor-executable program, which, when executed by a processor, is used to perform the method as described in any one of claims 1-7.

[0029] The present invention has the following advantages: (1) Real-time monitoring and timely calibration of external parameters If the error between the current calibration parameter and the actual calibration parameter exceeds the set value, the external parameter self-calibration is performed to reduce the error. For example, when the translation error exceeds 0.5m or the rotation angle error exceeds 10°, automatic self-calibration is performed to constrain the error to a range of 0.05m, 1° or even lower.

[0030] (2) Reducing the impact of external parameter errors on autonomous driving perception algorithms This method can constrain the error range in real time and control the error parameters within a controllable range (such as 0.05m, 1°), thus ensuring the stability of the autonomous driving perception algorithm.

[0031] (3) External parameters can be calibrated without the need for manual design of calibration tools and calibration rooms. A model trained on a sufficient dataset can replace manual calibration and optimize the calibration process.

[0032] (4) Enhance the robustness and accuracy of the perception algorithm Whether it's real-time monitoring of extrinsic parameter errors, constraining the range of extrinsic parameter errors, or embedding it as a network model module into autonomous driving tasks, it can reduce the sensitivity of perception algorithms to parameter errors and enhance robustness and accuracy.

[0033] (5) Multiple sensors on a vehicle can be calibrated simultaneously to obtain all external parameters at once. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating an extrinsic parameter calibration method based on multimodal data fusion in an embodiment of the present invention. Figure 2 This is a schematic diagram of depth map and training data generation in an embodiment of the present invention; Figure 3 This is a schematic diagram of the cross-modal cross-enhancement module in an embodiment of the present invention; Figure 4 This is a structural block diagram of an extrinsic parameter calibration system based on multimodal data fusion in an embodiment of the present invention; Figure 5 This is a structural block diagram of an external parameter calibration device based on multimodal data fusion in an embodiment of the present invention; Figure 6 This is a schematic diagram of the specific structure of an external parameter calibration system based on multimodal data fusion in an embodiment of the present invention; Figure 7 This is a schematic diagram of the visualization results of the nuScenes dataset in an embodiment of the present invention. Detailed Implementation

[0035] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0036] Example 1: Camera and Radar Calibration Method Step 1: After determining the specific camera and radar models, the camera can be an infrared camera or a visible light camera, and the radar can be a millimeter-wave radar or a lidar. Choose any mobile platform (unmanned vehicle, robot, etc.) and install the sensors in the preset positions. Step 2: Obtain accurate initial extrinsic calibration parameters and camera intrinsic parameters using high-precision calibration tools such as high-precision calibration boards or corner reflectors; Step 3: Move the mobile platform to any outdoor location and collect corresponding data; Step 4: Modify the installation location several times and repeatedly collect data to form a number of initial datasets; Step 5: Obtain the augmented dataset by injecting random affine transformation noise and other operations into the sensor calibration extrinsic parameters of the initial dataset. The augmentation method is as follows:

[0037] in This indicates the initial calibration of external parameters. This represents random affine noise. This represents the ground truth data of the augmented dataset. Let represent the truth rotation matrix and truth translation matrix of the augmented dataset, respectively.

[0038] Step 6 (optional): Preprocess the raw data: image data scaling, smoothing, etc., point cloud data range filtering, outlier removal, etc. Step 7: Project the LiDAR point cloud onto the 2D image plane using the camera intrinsic parameters and initial extrinsic parameters to obtain the corresponding depth image. The projection formula is:

[0039] in K Represents the camera intrinsic parameter matrix. P Represents point cloud coordinates, Let these represent the initial rotation matrix and the initial translation matrix, respectively. Indicates the depth scaling factor. These represent the horizontal and vertical coordinates corresponding to the point cloud projection onto the image coordinates, respectively. (See attached image.) Figure 2 The process of generating depth maps and datasets is demonstrated.

[0040] Step 8 (optional): Apply a preset range filter or a window smoothing filter to the depth map.

[0041] Step 9 (choose one): ① Augment the depth map to three dimensions, ② stitch the depth map and the original image together by channel dimension to form four-dimensional raw data.

[0042] Step 10: Adjust the backbone network input of the network model according to Step 8 to adapt to the input data.

[0043] Step 11: Train the corresponding calibration network using the computing platform and the dataset for the specific sensor model. (Appendix) Figure 1 The structure and training method of the network model are demonstrated. Image data and corresponding depth map data are batch-stitched or channel-wise stitched together, and then processed by a backbone network. The backbone network can be ResNet or Swin-Transformer, etc. The backbone network can be selected separately for images and depth maps, or used uniformly. The processed features are input to a feature pyramid network to obtain their respective multi-scale features. The multi-scale features are then processed by a cross-modal cross-enhancement module to obtain new features. The cross-modal cross-enhancement design is based on an attention mechanism. The fusion module can be used individually or multiple modules can be stitched together. Finally, the features are input to a calibration head to obtain extrinsic parameter data.

[0044] Step 12: The loss function used for training can be composed of one or more loss functions, such as point cloud reprojection loss, rotation quaternion angle loss, translation norm loss, and extrinsic parameter matrix loss, based on different weight coefficients. Below is an example of a loss function:

[0045] in L Indicates loss, This represents the point cloud loss weights. These represent the point cloud loss and the extrinsic parameter matrix loss, respectively. The extrinsic parameter matrix loss can be composed of the following losses:

[0046] in These represent translation loss, rotation loss, and matrix loss, respectively. These represent the translation loss weight, rotation loss weight, and matrix loss weight, respectively.

[0047] Step 13: The above calibration method was applied to the nuScenes autonomous driving dataset, and the accuracy of the calibration results is shown in the table below;

[0048] To further illustrate the effectiveness of this method, see Appendix Figure 7 The visualization results are displayed.

[0049] As attached Figure 4 As shown, this embodiment also provides an extrinsic parameter calibration system based on multimodal data fusion, including the following modules: Camera modules are used to acquire image data, and common types include visible light cameras and infrared cameras; Radar modules are used to acquire point cloud data, and common examples include 3D imaging devices such as lidar and millimeter-wave radar. The depth map mapping module is used to map point cloud data to image space based on image data, and generate a depth map corresponding to the camera. The calibration module is used to input the mapped data and image data into the calibration model for processing, obtain accurate extrinsic parameter prediction results, and realize the calibration of radar and camera.

[0050] This embodiment of the external parameter calibration system based on multimodal data fusion can execute the external parameter calibration method based on multimodal data fusion provided in the method embodiment of the present invention. It can execute any combination of implementation steps of the method embodiment and has the corresponding functions and beneficial effects of the method.

[0051] like Figure 5 As shown, this embodiment also provides an extrinsic parameter calibration device based on multimodal data fusion, including: Memory A1 is used to store computer programs; Processor A2 is used to implement the steps of the above-described extrinsic parameter calibration method based on multimodal data fusion when executing computer programs.

[0052] For details, please refer to Figure 6 This is a schematic diagram illustrating the specific structure of the extrinsic parameter calibration device based on a deep neural network model provided in this embodiment. This device can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) (e.g., one or more processors) and memory, and one or more storage media (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media can be temporary or persistent storage. The program stored in the storage media may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the central processing unit may be configured to communicate with the storage media to execute the series of instruction operations in the storage media on the deep neural network model-based extrinsic parameter calibration device.

[0053] Extrinsic parameter calibration devices based on deep neural network models may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input / output interfaces, and / or one or more operating systems. Examples include Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, etc.

[0054] The steps in the extrinsic parameter calibration method based on the deep neural network model described above can be implemented by the structure of the extrinsic parameter calibration device based on the deep neural network model.

[0055] This application also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform... Figure 1 The method shown.

[0056] This embodiment also provides a storage medium storing instructions or programs that can execute the extrinsic parameter calibration method based on multimodal data fusion provided in the method embodiment of the present invention. When the instructions or programs are run, any combination of implementation steps of the method embodiment can be executed, and the method has the corresponding functions and beneficial effects.

[0057] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0058] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0059] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, portable hard drive, read-only memory (ROM). Various media that can store program code, such as only memory, random access memory (RAM), magnetic disks or optical disks.

[0060] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0061] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0062] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0063] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0064] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

[0065] Although embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art will be able to make various modifications and improvements without departing from the principles of the present invention, and these modifications and improvements should also be considered to fall within the scope of protection of the present invention.

Claims

1. A method for extrinsic parameter calibration based on multimodal data fusion, characterized in that, Includes the following steps: S1. Acquire image data captured by the camera and point cloud data captured by the radar; S2. Based on the projection matrix obtained from the initial calibration of the camera and radar, the point cloud data is projected onto the image space to generate a depth map containing depth information. S3. Construct and train an extrinsic parameter calibration model; feed the image data and the generated depth map as input data into the trained extrinsic parameter calibration model, and the model outputs the extrinsic parameter prediction results of the camera relative to the radar, thereby completing the extrinsic parameter calibration of the camera and the radar.

2. The extrinsic parameter calibration method based on multimodal data fusion according to claim 1, characterized in that, In step S2, each projected point corresponds to a three-dimensional feature (d, u, v), where d represents the depth value of the point, and u and v represent the horizontal and vertical coordinates of the point on the image plane, respectively.

3. The extrinsic parameter calibration method based on multimodal data fusion according to claim 1, characterized in that, The training steps of the extrinsic calibration model include: A cross-modal cross-enhancement module and a multi-head calibration network are constructed using convolutional layers, batch normalization layers, linear connection layers, and pooling layers. Basic deep neural networks are constructed for image data and point cloud data using convolutional layers, batch normalization layers, linear rectified layers, and pooling layers. A cross-modal cross-enhancement module and a multi-head calibration network are inserted into the basic deep neural network to construct a calibration network that fuses point cloud data and image data.

4. The extrinsic parameter calibration method based on multimodal data fusion according to claim 3, characterized in that, The cross-modal cross-enhancement module achieves feature fusion through the following fusion feature expression: in, These represent the multi-scale features of the image and depth map, respectively. Indicates attention operation, This indicates an element-by-element concatenation operation. This represents matrix multiplication.

5. The extrinsic parameter calibration method based on multimodal data fusion according to claim 3, characterized in that, The number of calibration heads can be configured according to the number of cameras and radars, enabling one-to-one, many-to-many, one-to-many, and many-to-one camera-radar combination calibration.

6. The extrinsic parameter calibration method based on multimodal data fusion according to claim 2, characterized in that, The objective loss function expression of the extrinsic parameter calibration model is: in, L Indicates loss, This represents the point cloud loss weights. These represent the point cloud loss and the extrinsic parameter matrix loss, respectively. These represent translation loss, rotation loss, and matrix loss, respectively. These represent the translation loss weight, rotation loss weight, and matrix loss weight, respectively.

7. The extrinsic parameter calibration method based on multimodal data fusion according to claim 1, characterized in that, Precise initial extrinsic calibration parameters and camera intrinsic parameters are obtained by relying on high-precision calibration plates or corner reflectors.

8. An extrinsic parameter calibration system based on multimodal data fusion, characterized in that, include: Camera module, radar module, depth map mapping module, and calibration module; Camera module: Used to acquire image data; Radar module: Used to acquire point cloud data from the radar; Depth map mapping module: used to map point cloud data to image space based on image data to generate a depth map; Calibration module: Used to input the mapped data and image data into the extrinsic parameter calibration model for processing, obtain extrinsic parameter prediction results, and realize the extrinsic parameter calibration of camera-LiDAR.

9. An extrinsic parameter calibration device based on multimodal data fusion, characterized in that, include: At least one processor and at least one memory; The memory is used to store at least one program; When the program is executed by the processor, the processor performs the method according to any one of claims 1-7.

10. A storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the method as described in any one of claims 1-7.

Images