4d imaging millimeter wave radar-camera extrinsic parameter calibration method and device

Abstract

The application relates to the technical field of sensor calibration, in particular to a 4D imaging millimeter wave radar-camera external parameter calibration method and device. 4D imaging millimeter wave radar point cloud and camera images are acquired and preprocessed, an initial external parameter matrix of the 4D imaging millimeter wave radar-camera and a camera internal parameter matrix are initialized; point cloud features and image features under multiple scales are extracted by using an improved PointNet++ network and multiple residual network blocks, the improved PointNet++ network synchronously encodes spatial, velocity and intensity information of the point cloud; based on the point cloud features, the image features, the initial external parameter matrix and the camera internal parameter matrix, a point cloud-image fusion feature under multiple scales is acquired by using a Transformer module; the external parameter of the 4D imaging millimeter wave radar-camera is estimated by using a coarse-to-precise iterative refinement mode. Compared with the prior art, the application has the advantages that the 4D imaging millimeter wave radar and the camera external parameter online calibration can be quickly, timely and accurately performed without being affected by the environment.

Description

Technical Field

[0001] This invention relates to the field of sensor calibration technology, and in particular to a method and apparatus for calibrating the extrinsic parameters of a 4D imaging millimeter-wave radar-camera. Background Technology

[0002] Recently, autonomous driving technology has received widespread attention. To make autonomous vehicles safer and more reliable, autonomous driving systems are equipped with various perception sensors. 4D imaging millimeter-wave radar and cameras are two of the most commonly used perception sensors. Accurate extrinsic parameter calibration of 4D imaging millimeter-wave radar and cameras is an essential prerequisite for accurately fusing data. Most existing technologies use special calibration targets, such as polygonal calibration boards or infrared images, for extrinsic parameter calibration. While these methods have achieved good results, they require specific calibration environments and tools, limiting their application to initial calibration before product delivery and preventing online calibration during product use. Existing online calibration methods often use traditional feature-based matching and optimization, but these methods perform poorly on surfaces with strong textures and in environments with shadows. The development of deep learning technology has led to methods using deep learning for extrinsic parameter calibration. These methods demonstrate robustness to different environments. However, to accurately estimate the calibration matrix over a larger range, iterative networks are applied, increasing network size and reducing the efficiency of online calibration methods. Therefore, how to quickly, in real-time, and accurately perform online calibration of 4D imaging millimeter-wave radar and camera extrinsic parameters without being affected by the environment has become a problem that needs to be solved in this field. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art, which is susceptible to environmental influences, resulting in low immediacy and accuracy of online calibration of 4D imaging millimeter-wave radar and camera extrinsic parameters, and to provide a 4D imaging millimeter-wave radar-camera extrinsic parameter calibration method and apparatus.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] According to a first aspect of the present invention, a method for calibrating the extrinsic parameters of a 4D imaging millimeter-wave radar-camera is provided, comprising the following steps:

[0006] S1, acquire and preprocess the 4D imaging millimeter-wave radar point cloud and camera images, and initialize the initial extrinsic parameter matrix and camera intrinsic parameter matrix of the 4D imaging millimeter-wave radar-camera.

[0007] S2, based on the preprocessed point cloud and camera image, the improved PointNet++ network is used to extract point cloud features at multiple scales, and multiple residual network blocks are used to extract image features at multiple scales. The improved PointNet++ network synchronously encodes the spatial, velocity and intensity information of the preprocessed point cloud.

[0008] S3, Based on the point cloud features, the image features, the initial extrinsic matrix, and the camera intrinsic matrix, the Transformer module is used to obtain point cloud-image fusion features at multiple scales;

[0009] S4. Based on the point cloud-image fusion features at the multi-scale, the extrinsic parameters of the 4D imaging millimeter-wave radar-camera are estimated using an iterative refinement method from coarse to fine.

[0010] As a preferred technical solution, the process of extracting point cloud features at multiple scales using the improved PointNet++ network specifically includes:

[0011] S201, The preprocessed point cloud is sampled using the farthest point sampling method to obtain sampling points;

[0012] S202, obtain a point set with the sampling point as the center of the sphere to obtain a grouped point cloud, wherein the number of 4D radar points in the point set is the same as the preset number of search points;

[0013] S203, The improved PointNet++ network is used to perform feature aggregation on the grouped point cloud to obtain point cloud features at a single scale.

[0014] S204, based on a preset number of scales, repeat S201-S203 to obtain point cloud features at multiple scales.

[0015] As a preferred technical solution, the process of extracting image features at multiple scales using multiple residual network blocks specifically includes:

[0016] S211, based on the preprocessed camera image, first image data and residual data are obtained using a first 2D convolutional neural network and a second 2D convolutional neural network, wherein the stride of the first 2D convolutional neural network is 2 and the stride of the second 2D convolutional neural network is 1.

[0017] S212, based on the preprocessed camera image, a second image data is obtained using a third 2D convolutional neural network, and the second image data is added to the residual data to obtain the third image data, wherein the stride of the third 2D convolutional neural network is 2;

[0018] S213, Based on the third image data, image features at a single scale are obtained using an activation function;

[0019] S214, based on the preset number of residual network blocks, repeat S211-S213 to obtain image features at multiple scales, where each residual network block includes a first 2D convolutional neural network, a second 2D convolutional neural network, and a third 2D convolutional neural network.

[0020] As a preferred technical solution, S3 specifically includes:

[0021] S301, obtain sampling points of the point cloud features, and project the sampling points onto the image features using the initial extrinsic matrix and the camera intrinsic matrix to obtain the projection points and the first image feature, wherein the first image feature is the image feature corresponding to the projection points;

[0022] S302, connect the first point cloud feature and the first image feature to obtain the connection feature, wherein the first point cloud feature is the point cloud feature corresponding to the sampling point;

[0023] S303, the Transformer module is used to perform cross-modal feature interaction between the first point cloud feature, the first image feature and the connection feature to obtain cross-modal interaction features;

[0024] S304, aggregate the first point cloud features, the first image features, and the cross-modal interaction features to obtain point cloud-image fusion features.

[0025] As a preferred technical solution, the method of iterative refinement from coarse to fine specifically includes:

[0026] S401, based on the point cloud-image fusion features at the maximum scale, the initial extrinsic parameter values ​​are estimated by sequentially using a combination layer of 2D convolution-activation function-max pooling function and a combination layer of 2D convolution-ReLU activation function-average pooling function;

[0027] S402, based on the point cloud-image fusion features at other scales, the initial extrinsic parameter values ​​are iteratively optimized using a fully connected layer to obtain the final extrinsic parameter estimates.

[0028] According to a second aspect of the present invention, a 4D imaging millimeter-wave radar-camera extrinsic parameter calibration device is provided, comprising a data acquisition module, a feature extraction module, a feature fusion module, and an extrinsic parameter estimation module for signal interconnection.

[0029] The data acquisition module is used to acquire and preprocess 4D imaging millimeter-wave radar point clouds and camera images, initialize the initial extrinsic parameter matrix and camera intrinsic parameter matrix of 4D imaging millimeter-wave radar-camera, send the preprocessed point cloud and camera images to the feature extraction module, and send the initial extrinsic parameter matrix and the camera intrinsic parameter matrix to the feature fusion module.

[0030] The feature extraction module is used to receive the preprocessed point cloud and camera image, and extract point cloud features at multiple scales using an improved PointNet++ network, and extract image features at multiple scales using multiple residual network blocks. The point cloud features and the image features are sent to the feature fusion module. The improved PointNet++ network synchronously encodes the spatial, velocity and intensity information of the preprocessed point cloud.

[0031] The feature fusion module is used to receive the point cloud features, the image features, the initial extrinsic matrix and the camera intrinsic matrix, and use the Transformer module to obtain point cloud-image fusion features at multiple scales, and send the point cloud-image fusion features at multiple scales to the extrinsic parameter estimation module.

[0032] The extrinsic parameter estimation module is used to receive the point cloud-image fusion features at multiple scales and estimate the extrinsic parameters of the 4D imaging millimeter-wave radar-camera using an iterative refinement method from coarse to fine.

[0033] As a preferred technical solution, the feature extraction module includes a point cloud sampling unit for signal connectivity, a point cloud grouping unit, and a point cloud feature aggregation unit.

[0034] The point cloud sampling unit is used to sample the preprocessed point cloud using the farthest point sampling method, obtain sampling points, and send them to the point cloud grouping unit.

[0035] The point cloud grouping unit is used to receive the sampling points and obtain a point set with the sampling points as the center of the sphere, obtain the grouped point cloud and send it to the point cloud feature aggregation unit. The number of 4D radar points in the point set is the same as the preset number of search points.

[0036] The point cloud feature aggregation unit is used to receive the grouped point clouds and perform feature aggregation on the grouped point clouds using an improved PointNet++ network to obtain point cloud features at a single scale.

[0037] As a preferred technical solution, the feature extraction module further includes a first image and residual acquisition unit for signal connection, a third image acquisition unit, and an image feature acquisition unit.

[0038] The first image and residual acquisition unit is used to acquire first image data and residual data based on the preprocessed camera image using a first 2D convolutional neural network and a second 2D convolutional neural network, and send them to the third image acquisition unit. The stride of the first 2D convolutional neural network is 2, and the stride of the second 2D convolutional neural network is 1.

[0039] The third image acquisition unit is used to receive the first image data and the residual data, and based on the preprocessed camera image, uses a third 2D convolutional neural network to acquire the second image data, and adds the second image data to the residual data to acquire the third image data and sends it to the image feature acquisition unit. The stride of the third 2D convolutional neural network is 2.

[0040] The image feature acquisition unit is used to receive the third image data and use an activation function to acquire image features at a single scale.

[0041] As a preferred technical solution, the feature fusion module includes a first image feature acquisition unit for signal connectivity, a connectivity feature acquisition unit, a cross-modal interaction feature acquisition unit, and a feature aggregation unit.

[0042] The first image feature acquisition unit is used to acquire sampling points of the point cloud features, project the sampling points onto the image features using the initial extrinsic matrix and the camera intrinsic matrix, acquire the projection points and the first image feature, and send them to the connection feature acquisition unit. The first image feature is the image feature corresponding to the projection point.

[0043] The connection feature acquisition unit is used to receive the first image feature, connect the first point cloud feature and the first image feature to obtain the connection feature, and send it to the cross-modal interaction feature acquisition unit. The first point cloud feature is the point cloud feature corresponding to the sampling point.

[0044] The cross-modal interaction feature acquisition unit is used to receive the connection feature and use the Transformer module to perform cross-modal feature interaction between the first point cloud feature, the first image feature and the connection feature to obtain cross-modal interaction features and send them to the feature aggregation unit.

[0045] The feature aggregation unit is used to receive the cross-modal interaction features and aggregate the first point cloud features, the first image features and the cross-modal interaction features to obtain point cloud-image fusion features.

[0046] As a preferred technical solution, the extrinsic parameter estimation module includes an initial extrinsic parameter estimation unit and a final extrinsic parameter estimation unit for signal connections.

[0047] The initial extrinsic parameter estimation unit is used to estimate the initial extrinsic parameter values ​​based on the point cloud-image fusion features at the maximum scale, by sequentially using a combination layer of 2D convolution-activation function-max pooling function and a combination layer of 2D convolution-ReLU activation function-average pooling function, and then send the initial extrinsic parameter values ​​to the final extrinsic parameter estimation unit.

[0048] The final extrinsic parameter estimation unit is used to receive the initial extrinsic parameter values ​​and, based on the point cloud-image fusion features at other scales, iteratively optimize the initial extrinsic parameter values ​​using a fully connected layer to obtain the final extrinsic parameter estimates.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. This invention addresses extrinsic parameter correction for 4D imaging millimeter-wave radar and images. Taking advantage of the rich features of 4D radar point clouds, it improves the traditional Pointnet++ network by synchronously encoding the spatial, velocity, and intensity information of the preprocessed point cloud. This allows for full utilization of the rich point cloud information of the 4D radar. Extrinsic parameters are estimated using a coarse-to-fine layer-by-layer refinement method. By utilizing image features at different scales and 4D imaging millimeter-wave radar point cloud features together for extrinsic parameter estimation, it is less affected by external environmental factors, thus improving the accuracy of extrinsic parameter estimation.

[0051] 2. In the process of extracting multi-scale image features using multiple residual network blocks, this invention only requires a small number of convolutional neural networks to achieve multi-scale image feature extraction. It fully considers three modes: point mode, image mode, and connection mode. The Transformer module is used to perform feature interaction and feature fusion between modes. This can simultaneously utilize the 3D geometric information contained in the 4D imaging millimeter-wave radar points and the rich semantic information of the image to achieve the effect of fully fusing the 4D imaging millimeter-wave radar point features and image features. At the same time, it can avoid greatly increasing the size of the network, improve the running efficiency of extrinsic parameter estimation, and thus improve the real-time nature of online extrinsic parameter calibration.

[0052] 3. The present invention adopts a method of specifying the number of neighboring points, presets the specified number of search points, and obtains the grouped point cloud according to the preset number, avoiding the situation where neighboring points cannot be found, which can improve the efficiency of point cloud feature extraction and further improve the real-time nature of online external parameter calibration. Attached Figure Description

[0053] Figure 1 This is a flowchart illustrating the method in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating the principle of point cloud-image fusion in an embodiment of the present invention;

[0055] Figure 3 This is a schematic diagram illustrating the principle of iterative extrinsic parameter estimation in an embodiment of the present invention.

[0056] Figure 4 This is a schematic diagram of the device in an embodiment of the present invention. Detailed Implementation

[0057] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0058] Example

[0059] like Figure 1 As shown in the figure, this embodiment provides a 4D imaging millimeter-wave radar-camera extrinsic parameter calibration method, and the specific implementation process is as follows:

[0060] Step S1 involves acquiring and preprocessing the 4D imaging millimeter-wave radar point cloud and camera image, and initializing the initial extrinsic parameter matrix and camera intrinsic parameter matrix of the 4D imaging millimeter-wave radar-camera system. Specifically, the 4D imaging millimeter-wave radar point cloud is downsampled to 512 points using random sampling, and the camera image is scaled to half its original size.

[0061] In step S2, based on the preprocessed point cloud and camera images, the improved PointNet++ network is used to extract point cloud features at multiple scales, and multiple residual network blocks are used to extract image features at multiple scales. The improved PointNet++ network synchronously encodes the spatial, velocity and intensity information of the preprocessed point cloud.

[0062] The input point cloud and camera image are fed into the point cloud feature extraction network and the image feature extraction network, respectively. The point cloud feature extraction process is as follows:

[0063] Step S201: Sample the preprocessed point cloud using the farthest point sampling method to obtain sampling points. That is, use the farthest point sampling method to sample the input preprocessed point cloud, selecting a subset of points to represent the entire point cloud;

[0064] Step S202: Obtain a point set centered at each sampling point to obtain a grouped point cloud. The number of 4D radar points in the point set is the same as the preset number of search points. That is, by setting the number of search points, a point set satisfying the number of search points is obtained with each sampling point as the center, and the input preprocessed point cloud is grouped.

[0065] Step S203: The improved PointNet++ network is used to perform feature aggregation on the grouped point clouds to obtain point cloud features at a single scale. The original PointNet++ network only encodes the spatial information of the point cloud. 4D radar points not only contain spatial information but also motion and intensity information. To fully utilize the 4D radar point cloud information, the PointNet++ block is improved so that it can simultaneously encode the spatial, motion, and intensity information of the point cloud. A multilayer perceptron and other structures are used to learn features from the five dimensions of spatial, velocity, and intensity information of the grouped point clouds, and max pooling is used for feature aggregation.

[0066] Step S204: Based on a preset number of scales, repeat steps S201-S203 to obtain point cloud features at multiple scales. Specifically, repeat the above steps to obtain 4D imaging millimeter-wave radar point cloud features at three scales. The number of 4D imaging millimeter-wave radar points extracted at each scale is 256, 128, and 64, respectively. The coordinate sets of the 4D imaging millimeter-wave radar points at different scales are denoted as P0, P1, and P2, respectively, and the point cloud features extracted at different scales are denoted as RF0, RF1, and RF2, respectively.

[0067] While performing steps S201-S204, the image feature extraction process is as follows:

[0068] Step S211: Based on the preprocessed camera image, first image data and residual data are obtained using a first 2D convolutional neural network with a stride of 2 and a second 2D convolutional neural network with a stride of 1.

[0069] Step S212: Based on the preprocessed camera image, the second image data is obtained using a third 2D convolutional neural network with a stride of 2, and the second image data is added to the residual data to obtain the third image data;

[0070] Step S213: Based on the third image data, obtain image features at a single scale using an activation function.

[0071] Steps S211 to S213 involve taking the preprocessed camera image as input, first passing it through a 2D convolutional neural network with a stride of 2 and a kernel of 3×3, and then through a 2D convolutional neural network with a stride of 1 and a kernel of 3×3 to obtain the residual between the input and output first image data; then, the input image data passes through another 2D convolutional neural network with a stride of 2 and a kernel of 3×3 and is added to the previously obtained residual; finally, the image features are obtained after passing through the LeakyReLU activation function.

[0072] Step S214: Based on the preset number of residual network blocks, repeat steps S211 to S213 to obtain image features at multiple scales. Each residual network block includes a first 2D convolutional neural network, a second 2D convolutional neural network, and a third 2D convolutional neural network. That is, each residual network block consists of two 2D convolutional neural networks with a stride of 2 and a kernel size of 3×3, and one 2D convolutional neural network with a stride of 1 and a kernel size of 3×3. In this embodiment, three consecutive residual network blocks are set up to obtain image features at three scales. The size of the image features extracted at each scale is 1 / 4, 1 / 8, and 1 / 16 of the original image size, respectively. The features of the image at different scales are denoted as IF0, IF1, and IF2, respectively.

[0073] Step S3 involves using the Transformer module to obtain multi-scale point cloud-image fusion features based on point cloud features, image features, initial extrinsic matrix, and camera intrinsic matrix. For example... Figure 2 As shown, the specific process is as follows:

[0074] Step S301, use the initial extrinsic parameter matrix of the 4D imaging millimeter-wave radar-camera. With the camera intrinsic parameter matrix K, a 4D imaging millimeter-wave radar sampling point with coordinates P in the 4D imaging millimeter-wave radar coordinate system and related point cloud features RF (i.e., the first point cloud feature) can be projected onto the feature map of the RGB image to obtain the projection point p with coordinates (u,v) and the corresponding image feature IF. p (i.e., the first image feature). The projection process is as follows:

[0075]

[0076] In the formula, Z represents the depth value of the 4D imaging millimeter-wave radar point in the camera coordinate system, P represents the homogeneous coordinates of the 4D imaging millimeter-wave radar point, and T... IR After multiplying by P, take the first three coordinates so that they can be multiplied by K. If the coordinates (u,v) are not integers, use bilinear interpolation for retrieval;

[0077] Step S302, retrieve the image features IF p The connection features CF are obtained by concatenating them with the input point cloud features RF.

[0078] Step S303: Use the Transformer module to implement 4D radar point features (RF) and image features (IF). p This involves cross-modal feature interaction and multimodal feature fusion among the three features: 4D radar point features (RF) and query features (Q). P =RF·W Q Bond feature K P=RF·W K Similarly, image features are projected onto Q. I and K I For the connectivity feature CF, two linear transformations are first used to map it to different feature spaces to obtain... and After that and Projected onto value features respectively and W above Q W K , and It is a learnable linear mapping. Therefore, through attention weights W P←I =Softmax(Q I (K P ) T Obtain the contextual information of the image modality, and then through... This allows interaction with the connectivity modality to obtain connectivity features guided by the image modality. Similarly, it can be done through... Obtain the contextual information of the point modal and encode it into connectivity features guided by the point modal.

[0079] Step S304: Aggregate the original 4D radar features (RF) and image features (IF). p and features F with cross-modal interaction P←I and F I←P This yields point cloud-image fusion features, i.e. As a fusion feature.

[0080] Step S4 involves estimating the extrinsic parameters of the 4D imaging millimeter-wave radar-camera using a coarse-to-fine iterative refinement method based on multi-scale point cloud-image fusion features. For example... Figure 3 As shown, the specific process is as follows:

[0081] Step S401: Based on the point cloud-image fusion features at the largest scale, the initial extrinsic parameter values ​​are estimated by sequentially applying a combination layer of 2D convolution-activation function-max pooling function and a combination layer of 2D convolution-ReLU activation function-average pooling function. Specifically, two combination layers of 2D convolution-ReLU activation function-max pooling function and one combination layer of 2D convolution-ReLU activation function-average pooling function are applied sequentially to the fusion features at the largest scale to obtain feature matching information between the 4D imaging millimeter-wave radar and the camera.

[0082] Step S402: Based on the point cloud-image fusion features at other scales, the initial extrinsic parameter values ​​are iteratively optimized using a fully connected layer to obtain the final extrinsic parameter estimates. Specifically, this includes:

[0083] Step S4021: Based on the feature matching information from the 4D imaging millimeter-wave radar and camera, two independent fully connected layers are used to obtain the initial translation estimate and the initial rotation estimate, respectively. The initial translation is represented by a translation vector, and the initial rotation is represented by a quaternion. The extrinsic parameter matrix at the second scale can be obtained from the initial translation estimate and the initial rotation estimate.

[0084] Step S4022, using the extrinsic parameter matrix at the second scale Coordinate transformation of the first-scale 4D imaging millimeter-wave radar point set: Based on the transformed coordinates P1 ′ After the fusion step S3 and the estimation steps S401 and S4021, the extrinsic parameter matrix at the first scale can be obtained.

[0085] Step S4023, use the extrinsic parameter matrix at the first scale. Coordinate transformation of the zero-scale 4D imaging millimeter-wave radar point set: Based on the transformed coordinates P0 ′ After the fusion step S3 and the estimation steps S401 and S4021, the extrinsic parameter matrix at the zeroth scale can be obtained.

[0086] Step S4024, finally, by... Multiply them to obtain the final extrinsic parameter estimate between the 4D imaging millimeter-wave radar and the camera.

[0087] Furthermore, this embodiment also provides a 4D imaging millimeter-wave radar-camera extrinsic parameter calibration device, such as... Figure 4As shown, the device includes a data acquisition module, a feature extraction module, a feature fusion module, and an extrinsic parameter estimation module for signal connection. The data acquisition module acquires and preprocesses 4D imaging millimeter-wave radar point clouds and camera images, initializes the initial extrinsic parameter matrix and camera intrinsic parameter matrix of the 4D imaging millimeter-wave radar-camera system, sends the preprocessed point cloud and camera images to the feature extraction module, and sends the initial extrinsic parameter matrix and camera intrinsic parameter matrix to the feature fusion module. The feature extraction module receives the preprocessed point cloud and camera images, extracts multi-scale point cloud features using an improved PointNet++ network, and extracts multi-scale features using multiple residual network blocks. The image features are processed by sending point cloud features and image features to the feature fusion module. The improved PointNet++ network synchronously encodes the spatial, velocity, and intensity information of the preprocessed point cloud. The feature fusion module receives point cloud features, image features, initial extrinsic parameter matrix, and camera intrinsic parameter matrix, and uses the Transformer module to obtain multi-scale point cloud-image fusion features. These multi-scale point cloud-image fusion features are then sent to the extrinsic parameter estimation module. The extrinsic parameter estimation module receives the multi-scale point cloud-image fusion features and estimates the extrinsic parameters of the 4D imaging millimeter-wave radar-camera using an iterative refinement method from coarse to fine. Each module can implement all the aforementioned method steps, and the specific implementation process is as described above, and will not be repeated here.

[0088] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for calibrating the extrinsic parameters of a 4D imaging millimeter-wave radar-camera, characterized in that, Includes the following steps: S1, acquire and preprocess the 4D imaging millimeter-wave radar point cloud and camera images, and initialize the initial extrinsic parameter matrix and camera intrinsic parameter matrix of the 4D imaging millimeter-wave radar-camera. S2, based on the preprocessed point cloud and camera image, the improved PointNet++ network is used to extract point cloud features at multiple scales, and multiple residual network blocks are used to extract image features at multiple scales. The improved PointNet++ network synchronously encodes the spatial, velocity and intensity information of the preprocessed point cloud. The process of extracting multi-scale point cloud features using the improved PointNet++ network specifically includes: S201, The preprocessed point cloud is sampled using the farthest point sampling method to obtain sampling points; S202, obtain a point set with the sampling point as the center of the sphere to obtain a grouped point cloud, wherein the number of 4D radar points in the point set is the same as the preset number of search points; S203, the improved PointNet++ network is used to perform feature aggregation on the grouped point cloud to obtain point cloud features at a single scale; the improved PointNet++ network uses a multilayer perceptron to learn features from the five-dimensional information of the grouped point cloud, including spatial information, velocity information and intensity information, and uses max pooling operation to perform feature aggregation. S204, based on a preset number of scales, repeat S201-S203 to obtain point cloud features at multiple scales; S3, Based on the point cloud features, the image features, the initial extrinsic matrix, and the camera intrinsic matrix, the Transformer module is used to obtain point cloud-image fusion features at multiple scales; S3 specifically includes: S301, obtain sampling points of the point cloud features, and project the sampling points onto the image features using the initial extrinsic matrix and the camera intrinsic matrix to obtain the projection points and the first image feature, wherein the first image feature is the image feature corresponding to the projection points; S302, connect the first point cloud feature and the first image feature to obtain the connection feature, wherein the first point cloud feature is the point cloud feature corresponding to the sampling point; S303, the Transformer module is used to perform cross-modal feature interaction between the first point cloud feature, the first image feature and the connection feature to obtain cross-modal interaction features; S304, aggregate the first point cloud features, the first image features, and the cross-modal interaction features to obtain point cloud-image fusion features; S4. Based on the point cloud-image fusion features at the multi-scale, the extrinsic parameters of the 4D imaging millimeter-wave radar-camera are estimated using an iterative refinement method from coarse to fine. The method of iterative refinement from coarse to fine specifically includes: S401, based on the point cloud-image fusion features at the maximum scale, the initial extrinsic parameter values ​​are estimated by sequentially using a combination layer of 2D convolution-activation function-max pooling function and a combination layer of 2D convolution-ReLU activation function-average pooling function; S402, based on the point cloud-image fusion features at other scales, the initial extrinsic parameter values ​​are iteratively optimized using a fully connected layer to obtain the final extrinsic parameter estimates.

2. The 4D imaging millimeter-wave radar-camera extrinsic parameter calibration method according to claim 1, characterized in that, The process of extracting multi-scale image features using multiple residual network blocks specifically includes: S211, based on the preprocessed camera image, the residual data of the input and output first image data is obtained using a first 2D convolutional neural network and a second 2D convolutional neural network, wherein the stride of the first 2D convolutional neural network is 2 and the stride of the second 2D convolutional neural network is 1. S212, based on the preprocessed camera image, a second image data is obtained using a third 2D convolutional neural network, and the second image data is added to the residual data to obtain the third image data, wherein the stride of the third 2D convolutional neural network is 2; S213, Based on the third image data, image features at a single scale are obtained using an activation function; S214, based on the preset number of residual network blocks, repeat S211-S213 to obtain image features at multiple scales, where each residual network block includes a first 2D convolutional neural network, a second 2D convolutional neural network, and a third 2D convolutional neural network.

3. A 4D imaging millimeter-wave radar-camera extrinsic parameter calibration device, characterized in that, It includes a data acquisition module for signal connections, a feature extraction module, a feature fusion module, and an extrinsic parameter estimation module. The data acquisition module is used to acquire and preprocess 4D imaging millimeter-wave radar point clouds and camera images, initialize the initial extrinsic parameter matrix and camera intrinsic parameter matrix of 4D imaging millimeter-wave radar-camera, send the preprocessed point cloud and camera images to the feature extraction module, and send the initial extrinsic parameter matrix and the camera intrinsic parameter matrix to the feature fusion module. The feature extraction module is used to receive the preprocessed point cloud and camera image, and extract point cloud features at multiple scales using an improved PointNet++ network, and extract image features at multiple scales using multiple residual network blocks. The point cloud features and the image features are sent to the feature fusion module. The improved PointNet++ network synchronously encodes the spatial, velocity and intensity information of the preprocessed point cloud. The feature extraction module includes a point cloud sampling unit for signal connectivity, a point cloud grouping unit, and a point cloud feature aggregation unit. The point cloud sampling unit is used to sample the preprocessed point cloud using the farthest point sampling method, obtain sampling points, and send them to the point cloud grouping unit. The point cloud grouping unit is used to receive the sampling points and obtain a point set with the sampling points as the center of the sphere, obtain the grouped point cloud and send it to the point cloud feature aggregation unit. The number of 4D radar points in the point set is the same as the preset number of search points. The point cloud feature aggregation unit is used to receive the grouped point cloud and use the improved PointNet++ network to perform feature aggregation on the grouped point cloud to obtain point cloud features at a single scale. The feature fusion module is used to receive the point cloud features, the image features, the initial extrinsic matrix and the camera intrinsic matrix, and use the Transformer module to obtain point cloud-image fusion features at multiple scales, and send the point cloud-image fusion features at multiple scales to the extrinsic parameter estimation module. The feature fusion module includes a first image feature acquisition unit for signal connectivity, a connectivity feature acquisition unit, a cross-modal interaction feature acquisition unit, and a feature aggregation unit. The first image feature acquisition unit is used to acquire sampling points of the point cloud features, project the sampling points onto the image features using the initial extrinsic matrix and the camera intrinsic matrix, acquire the projection points and the first image feature, and send them to the connection feature acquisition unit. The first image feature is the image feature corresponding to the projection point. The connection feature acquisition unit is used to receive the first image feature, connect the first point cloud feature and the first image feature to obtain the connection feature, and send it to the cross-modal interaction feature acquisition unit. The first point cloud feature is the point cloud feature corresponding to the sampling point. The cross-modal interaction feature acquisition unit is used to receive the connection feature and use the Transformer module to perform cross-modal feature interaction between the first point cloud feature, the first image feature and the connection feature to obtain cross-modal interaction features and send them to the feature aggregation unit. The feature aggregation unit is used to receive the cross-modal interaction features and aggregate the first point cloud features, the first image features and the cross-modal interaction features to obtain point cloud-image fusion features; The extrinsic parameter estimation module is used to receive the point cloud-image fusion features at multiple scales and estimate the extrinsic parameters of the 4D imaging millimeter-wave radar-camera using an iterative refinement method from coarse to fine. The extrinsic parameter estimation module includes an initial extrinsic parameter estimation unit and a final extrinsic parameter estimation unit for signal connections. The initial extrinsic parameter estimation unit is used to estimate the initial extrinsic parameter values ​​based on the point cloud-image fusion features at the maximum scale, by sequentially using a combination layer of 2D convolution-activation function-max pooling function and a combination layer of 2D convolution-ReLU activation function-average pooling function, and then send the initial extrinsic parameter values ​​to the final extrinsic parameter estimation unit. The final extrinsic parameter estimation unit is used to receive the initial extrinsic parameter values ​​and, based on the point cloud-image fusion features at other scales, iteratively optimize the initial extrinsic parameter values ​​using a fully connected layer to obtain the final extrinsic parameter estimates.

4. The 4D imaging millimeter-wave radar-camera extrinsic parameter calibration device according to claim 3, characterized in that, The feature extraction module further includes a first image and residual acquisition unit for signal connection, a third image acquisition unit, and an image feature acquisition unit. The first image and residual acquisition unit is used to acquire residual data of the input and output first image data based on the preprocessed camera image using a first 2D convolutional neural network and a second 2D convolutional neural network, and send it to the third image acquisition unit. The first 2D convolutional neural network has a stride of 2, and the second 2D convolutional neural network has a stride of 1. The third image acquisition unit is used to receive the residual data, acquire second image data based on the preprocessed camera image using a third 2D convolutional neural network, add the second image data to the residual data to acquire third image data and send it to the image feature acquisition unit. The stride of the third 2D convolutional neural network is 2. The image feature acquisition unit is used to receive the third image data and use an activation function to acquire image features at a single scale.

Images