Millimeter wave radar and laser radar external parameter calibration method, device, equipment and medium

Abstract

The invention relates to a millimeter wave radar and laser radar external parameter calibration method, device, equipment and medium. The method comprises the following steps: performing data alignment on millimeter wave radar data and laser radar data to construct a data set; constructing an external parameter calibration matching model, performing key point extraction on the data set based on a preset loss function and the external parameter calibration matching model, and performing feature matching based on key points to obtain key point matching data; based on the key point matching data, distributing the key point matching data to adjacent areas by adopting a point-to-node strategy to obtain dense point matching data; and obtaining joint calibration parameters of the millimeter wave radar and the laser radar based on the dense point matching data. The method provided by the invention can maximize the effective information of the millimeter-wave radar and laser radar point clouds, solves the problems of sparse point clouds and multi-noise point clouds of the millimeter-wave radar, and improves the calibration precision and robustness.

Description

Technical Field

[0001] The present invention relates to the field of radar data processing technology, and in particular to a method, device, equipment and medium for calibrating external parameters of millimeter wave radar and laser radar. Background Art

[0002] With the development of autonomous driving, intelligent robotics, and intelligent transportation systems, millimeter-wave radar and lidar, as important sensors, have played a vital role in perception, positioning, and obstacle avoidance. However, accurate extrinsic calibration is a key step in enabling these two sensors to work together in a unified coordinate system.

[0003] Currently, 4D millimeter-wave radar calibration methods primarily rely on corner reflectors or dedicated calibration plates. While these methods have achieved promising results, they are primarily limited to offline implementation and place high demands on the hardware used for calibration. However, during routine calibration and online operation of intelligent robots or autonomous vehicles, sensor attitude drift is inevitable, potentially compromising the accuracy of initial calibration results. Furthermore, some 4D millimeter-wave radar devices lack hardware time synchronization. In sensor systems lacking hardware time synchronization, the asynchronous nature of data acquisition between sensors can lead to data misalignment during high-speed operation of robots and vehicles.

[0004] In addition, the point cloud data acquired by 4D millimeter-wave radar is relatively sparse and accompanied by a large amount of noise and false points, which seriously affects its practical application value. Compared with traditional 3D millimeter-wave radar, the point cloud quality of 4D millimeter-wave radar is significantly improved. In addition to the distance, direction, relative speed and other information acquired by traditional millimeter-wave radar, 4D millimeter-wave radar can also obtain target height information. The increase in the number of transmitting and receiving antennas greatly improves the resolution and detection capabilities of 4D millimeter-wave radar. However, its point cloud still contains more noise and is relatively sparse compared to lidar point clouds, which limits the application of 4D millimeter-wave radar. Summary of the Invention

[0005] Based on this, it is necessary to provide a millimeter-wave radar and lidar external parameter calibration method, device, equipment and medium that can solve the problems of sparse point cloud and multi-noise of point cloud in 4D millimeter-wave radar to address the above technical problems.

[0006] A method for calibrating extrinsic parameters of millimeter-wave radar and laser radar, the method comprising: Align millimeter-wave radar data and lidar data to build a data set; Constructing an external parameter calibration matching model, based on a preset loss function, the external parameter calibration matching model extracts key points from the data set, and performs feature matching based on the key points to obtain key point matching data; Based on the key point matching data, a point-to-node strategy is adopted to distribute the key point matching data to adjacent areas to obtain dense point matching data; Based on the dense point matching data, joint calibration parameters of the millimeter wave radar and the lidar are obtained.

[0007] A millimeter wave radar and laser radar extrinsic parameter calibration device, the device comprising: The dataset construction module is used to align millimeter-wave radar data and lidar data to construct a dataset; A key point matching module is used to construct an external parameter calibration matching model. Based on a preset loss function, the external parameter calibration matching model extracts key points from the data set and performs feature matching based on the key points to obtain key point matching data; A dense point matching module is used to distribute the key point matching data to adjacent areas based on the key point matching data using a point-to-node strategy to obtain dense point matching data; A joint calibration module is used to obtain joint calibration parameters of the millimeter wave radar and the lidar based on the dense point matching data.

[0008] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the millimeter wave radar and lidar extrinsic parameter calibration method when executing the computer program.

[0009] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the millimeter-wave radar and lidar extrinsic parameter calibration method.

[0010] The above-mentioned millimeter-wave radar and lidar external parameter calibration method, device, equipment and medium construct a data set by aligning the millimeter-wave radar data and the lidar data; construct an external parameter calibration matching model, and based on a preset loss function, the external parameter calibration matching model extracts key points of the data set and performs feature matching based on the key points to obtain key point matching data; based on the key point matching data, a point-to-node strategy is adopted to distribute the key point matching data to adjacent areas to obtain dense point matching data; based on the dense point matching data, the joint calibration parameters of the millimeter-wave radar and the lidar are obtained.

[0011] The present invention aligns millimeter-wave radar and lidar data in a common reference frame through data alignment, providing a more refined and accurate data foundation for subsequent calibration work. During the joint calibration process, the characteristics of the lidar's high-density point cloud are utilized to extract stable key points, and then the correspondence between the lidar's point cloud information and the millimeter-wave radar's point cloud information is established through feature matching, thereby achieving the purpose of supplementing the missing detailed information of the millimeter-wave radar. When matching dense points, the point cloud structure around each key point is considered, and the key point matching data is distributed to adjacent areas through a point-to-node strategy. This can constrain the spatial position relationship of the point cloud, eliminate isolated noise points, effectively improve matching accuracy, and enhance the quality and stability of the point cloud data. The method of the present invention does not require reliance on additional calibration targets, reduces the need for additional hardware, and simplifies the calibration process. At the same time, the method of the present invention maximizes the effective information of the millimeter-wave radar and lidar point clouds through fine point matching and optimization, solves the problems of sparse millimeter-wave radar point clouds and high noise in point clouds, and improves the accuracy and robustness of calibration. Overall, this target-free calibration framework provides reliable technical support for multi-sensor collaboration and has broad application prospects, especially in the fields of autonomous driving and intelligent robots. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0013] Figure 1 1 is a flow chart of a method for calibrating extrinsic parameters of a millimeter-wave radar and a lidar in one embodiment; Figure 2 FIG1 is a schematic diagram of projecting a millimeter-wave radar point cloud onto a map composed of laser radar point clouds using the extrinsic parameters estimated by the method proposed in the present invention in one embodiment, wherein: Figure 2 (a) is a schematic diagram of scene 1 splicing. Figure 2 (b) is a schematic diagram of the splicing of scene 2. Figure 2 (c) is a schematic diagram of the splicing of scene three. Figure 2 (d) is a schematic diagram of the splicing of scene 4; Figure 3 This is a millimeter-wave radar point cloud image spliced ​​using the laser radar pose and the extrinsic parameter data estimated by the method proposed in the present invention in one embodiment, where: Figure 3 (a) is the millimeter-wave radar point cloud image spliced ​​using external parameter data estimated by ICP. Figure 3(b) is the millimeter-wave radar point cloud image spliced ​​with the external parameter data estimated by LCR-Let. Figure 3 (c) is the millimeter-wave radar point cloud image spliced ​​with the external parameter data estimated by the method proposed in this invention. Figure 3 (d) is the millimeter-wave radar point cloud image spliced ​​using external parameter data estimated by Groundtruth; Figure 4 1 is a structural block diagram of an extrinsic parameter calibration device for millimeter-wave radar and lidar in one embodiment; Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment.

[0014] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0016] It can be understood that the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in this field can implement it. When the combination of technical solutions is mutually contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0017] The following describes the implementation of the present invention in detail with reference to the accompanying drawings in the embodiments of the present invention.

[0018] Example 1 This embodiment discloses a method for extrinsic calibration of millimeter-wave radar and lidar. This method uses data alignment to align millimeter-wave radar and lidar data in a common reference frame, providing a more refined and accurate data foundation for subsequent calibration. During the joint calibration process, the high-density point cloud of the lidar is utilized to extract stable key points. Then, through feature matching, the correspondence between the lidar point cloud information and the millimeter-wave radar point cloud information is established, thereby supplementing the missing detailed information of the millimeter-wave radar. During dense point matching, the point cloud structure around each key point is considered, and key point matching data is distributed to adjacent areas using a point-to-node strategy. This constrains the spatial positional relationship of the point cloud, eliminates isolated noise points, effectively improves matching accuracy, and enhances the quality and stability of the point cloud data. This method does not require additional calibration targets, reduces the need for additional hardware, and simplifies the calibration process. Furthermore, through precise point matching and optimization, this method maximizes the effective information of the millimeter-wave radar and lidar point clouds, addresses the issues of sparse and noisy millimeter-wave radar point clouds, and improves the accuracy and robustness of calibration. Overall, this target-free calibration framework provides reliable technical support for multi-sensor collaboration and has broad application prospects, especially in the fields of autonomous driving and intelligent robots.

[0019] like Figure 1 As shown, the millimeter wave radar and lidar extrinsic parameter calibration method provided in this embodiment includes the following steps: Step 201: align the millimeter-wave radar data and the lidar data to construct a data set.

[0020] Step 202: Construct an external parameter calibration matching model.

[0021] In step 203 , based on a preset loss function, the external parameter calibration matching model extracts key points from the data set, and performs feature matching based on the key points to obtain key point matching data.

[0022] In step 204 , based on the key point matching data, a point-to-node strategy is adopted to distribute the key point matching data to adjacent areas to obtain dense point matching data.

[0023] Step 205: Based on the dense point matching data, obtain the joint calibration parameters of the millimeter wave radar and the lidar.

[0024] In one embodiment, data alignment is performed on millimeter wave radar data and lidar data to construct a data set, including: The millimeter-wave radar data and the lidar data are temporally and spatially aligned to obtain initial alignment data; then the initial alignment data is refined to obtain precision alignment data.

[0025] Through random external parameter transformation, the precision alignment data is enhanced to obtain the enhanced data; then the yaw angle is preset, and the point cloud data with the same direction as the millimeter wave radar and lidar are selected to construct the data set.

[0026] Specifically, this embodiment uses the high-quality 4D millimeter-wave radar point cloud dataset (VoD dataset) and the MSC RAD4R dataset to further construct the datasets required for millimeter-wave radar and lidar extrinsic calibration. During alignment, due to the different sampling frequencies of the millimeter-wave radar and lidar, the timestamps of the data collected by the two sensor devices are synchronized, and the data is adjusted to a unified time base to achieve temporal alignment. Furthermore, due to the different installation locations of the millimeter-wave radar and lidar, coordinate transformation is performed based on known sensor extrinsic parameter data to ensure that the data can be aligned in a unified coordinate system, achieving spatial alignment.

[0027] After time alignment and spatial alignment, there are still slight misalignments between the two types of point clouds, which affects the accuracy of the external parameter calibration data. Therefore, further refined alignment is required. In this embodiment, the ICP method is used for refined alignment. This embodiment uses time synchronization + coordinate transformation + refined optimization to obtain more refined alignment data, avoid fusion errors caused by data deviation, and provide an accurate data foundation for subsequent calibration work. The precision alignment data includes data such as lidar point cloud, millimeter wave radar point cloud, synchronization, timestamp, and pose information.

[0028] After obtaining the precision alignment data, each frame of lidar point cloud data is enhanced by performing random translation within 5 meters in the x, y, and z directions, and random rotation within 10° in the roll, pitch, and yaw directions to generate different calibration parameters. This results in an external parameter calibration dataset with a variety of different calibration parameters, which is used to improve the generalization ability of the algorithm, avoid overfitting, and improve the robustness of the calibration algorithm in complex environments.

[0029] After data augmentation, in order to reduce redundancy, the lidar data that is roughly consistent with the direction of the millimeter-wave radar is selected through a preset yaw angle. For example, only the point cloud data located 180° in front of the lidar that is consistent with the direction of the millimeter-wave radar is retained, thereby effectively reducing redundant information and improving calibration efficiency. Furthermore, considering the sparsity of millimeter-wave radar point clouds at long distances, in order to improve the accuracy of the calibration data, the point cloud processing range is limited to only points that are no more than 60 meters away from the millimeter-wave radar for subsequent processing. This processing can avoid noise interference caused by the sparsity of long-distance point clouds, thereby ensuring data quality. To further optimize the calibration process, the two point cloud data were downsampled to a voxel size of 0.3 meters to standardize the data and prepare it for neural network input.

[0030] The dataset generated in the above way can effectively reduce redundant data and improve the processing efficiency and accuracy of calibration.

[0031] In one embodiment, the extrinsic parameter calibration matching model includes a key point extraction module, a key point matching module and a dense point matching module.

[0032] The key points of the data set are extracted through the key point extraction module; the key points are feature matched through the key point matching module to obtain key point matching data; the dense point matching module adopts a point-to-node strategy to distribute the key point matching data to adjacent areas to obtain dense point matching data.

[0033] Specifically, the extrinsic calibration matching model inputs the processed data set, and the output joint calibration parameters are the 6-DOF extrinsic parameters between the millimeter-wave radar and the lidar.

[0034] The key point extraction module extracts stable key points from the two point clouds of the dataset, and the key point matching module establishes corresponding relationships, which serve as preliminary guesses for the subsequent dense point matching module to perform neighborhood expansion.

[0035] In one embodiment, the key point extraction module serves as the starting point for stable key point extraction. It mainly uses a single shared encoder to downsample and encode the lidar point cloud data and the millimeter wave radar point cloud data layer by layer to obtain evenly distributed nodes and corresponding features. It is understandable that since mmWave radar and lidar point clouds are essentially geometric representations of the environment, adopting a single shared encoder can more effectively capture the common geometric and semantic features in both point clouds.

[0036] In order to aggregate and exchange context information between two frames of point clouds, the nodes and corresponding features are evenly distributed. , query point cloud data by linear projection Midpoint position and features Mapping to source point cloud data For all points in , the expression is: (1) Where, The query vector representing the query point cloud; The key vector representing the source point cloud; A vector of values ​​representing the source point cloud; 、 、 represents the weight matrix of the linear projection; Indicates the first The characteristics of each point; Indicates the first The characteristics of each point; 、 、 Represents the bias vector for the linear projection.

[0037] It is worth noting that both lidar point clouds and millimeter-wave radar point clouds can be either query point clouds or source point clouds, depending on the number of feature encoding layers they are in.

[0038] Then judge the query point cloud data With source point cloud data Are the point clouds in the same? If they are the same, they are directly mapped by linear projection to generate feature maps for self-attention operation; if they are different, cross-attention operation is performed to finally obtain enhanced features of point cloud interaction.

[0039] When performing the cross attention operation, the multi-layer perceptron (MLP) is used to convert the position of the point in the point cloud data into Mapping to Rotated Embedding ,in Indicates the The rotation angle is then converted into a rotation matrix using the following formula: (2) Where, express Span the rotation matrix.

[0040] Similarly, calculate the rotation matrix of another frame of point cloud data , then the rotation matrix and Applied to query vectors respectively and key vector , to realize the self-attention operation, thereby obtaining the rotational self-attention, the formula is as follows: (3) (4) Based on this, we further deduce formula (3) and obtain the expression; (5) Where, represents the attention weight; represents the rotation attention feature; express Zhang Cheng's rotation matrix; represents the matrix transpose, where A replaceable variable.

[0041] Among them, relative information Being integrated into Finally, we get the enhanced features of point cloud interaction Enhanced Features Fusion of contextual information and geometric structures.

[0042] In the key point extraction module, a set of multi-layer perceptrons (MLPs) are also used to estimate the geometric offset. Specifically, based on the enhanced features , for nodes and corresponding features Estimate the geometric offset and map it to the candidate key points to obtain the candidate key points. The expression is: (6) (7) (8) Where, Indicates geometric offset; Indicates feature offset; Represents node features; Indicates the position of the key point after offset; Represents the features after offset.

[0043] By adding offsets, multiple candidate boxes are clustered in the salient area of ​​the point cloud. Finally, the candidate key points are clustered and a candidate key point is randomly selected from each cluster as the final key point.

[0044] In one embodiment, key point matching module performs feature matching on key points to obtain key point matching data, including: Based on the key points, the similarity between the key points of millimeter wave radar and lidar is constructed by calculating the Gaussian correlation matrix to obtain the similarity elements.

[0045] Perform double normalization on the similarity elements to obtain normalized elements; select the element with the largest normalized value The normalized elements are used and their corresponding indexes are used as key point correspondences to complete feature matching and obtain key point matching data.

[0046] Specifically, calculate a Gaussian correlation matrix , used to model the normalized lidar key point features and millimeter wave radar key point features The similarity between them, get the similarity element , the process expression is: (9) Similarity elements Perform double normalization to suppress fuzzy matching and obtain normalized elements , the process expression is: (10) Where, express The Features express The Features represents the final lidar keypoints; Represents the final millimeter-wave radar key points; Indicates the Rank Similarity elements of columns; Indicates the Rank Similarity elements of the columns.

[0047] In this process, the one with the greatest similarity is selected The normalized elements are used and their corresponding indexes are used as key point correspondences to complete feature matching and obtain key point matching data.

[0048] In one embodiment, the dense point matching module adopts a point-to-node strategy to distribute key point matching data to adjacent areas to obtain dense point matching data; including: Based on the point-to-node strategy, each keypoint in the keypoint matching data is assigned to the nearest keypoint to obtain a neighborhood point set.

[0049] In the neighborhood point set, the point-level descriptor of each key point is restored by KPDecoder, and the matching score matrix is ​​calculated based on it; when calculating the matching score matrix, a non-matching subset is constructed.

[0050] Based on the matching score matrix, the soft assignment matrix is ​​solved by the Sinkhorn algorithm, and the millimeter-wave radar point cloud is assigned to the corresponding lidar point cloud or to the unmatched subset to obtain the final dense point matching data.

[0051] Specifically, since a matched keypoint pair represents the similarity between their respective neighborhoods, it implies that more point matches may be found within these regions. Therefore, we first determine the region to which each keypoint belongs, and for each keypoint in the keypoint matching data, , using the point-to-node strategy, all points in the original point cloud are assigned to the nearest key point, and the key point is obtained Neighborhood point set , the expression is: (11) Where, Represents a point in the original point cloud; represents the original point cloud; Indicates key points; Represents a set of key points.

[0052] In the neighborhood point set, the point-level descriptor of each key point is restored through KPDecoder It can be understood that for each key point correspondence , each has its corresponding point block matching . Subsequently, using point-level descriptors Calculate the matching score matrix , the expression is: (12) Where, Indicates the LiDAR key points; Indicates the LiDAR key points; No. A block of raw lidar points; Indicates the Millimeter-wave radar original point blocks; Indicates the LiDAR keypoint features; Indicates the Millimeter wave radar key point features; Represents the key point feature dimension, which is the same for millimeter wave radar and lidar.

[0053] In order to deal with unmatched points, in the matching score matrix A fill with learnable parameters is appended to the end of Then, the soft assignment matrix is ​​solved by the Sinkhorn algorithm. Considering the inherent difference in data volume between lidar and millimeter-wave radar point clouds, a row-by-row maximization strategy is adopted to assign each millimeter-wave radar point to the corresponding lidar point or to the unmatched subset to obtain the final dense point matching data. It is worth noting that if a millimeter-wave radar point is assigned to the unmatched subset, the point is classified as an unmatched point and deleted.

[0054] It's worth noting that in the dense point matching module, matching can be performed on either the top layer (denoted as RLCNet1encoder40) or the second layer (denoted as RLCNet1encoder41), that is, on the original point cloud or the downsampled point cloud, which yields better results. This is likely due to the excessive number of lidar point clouds at the top layer, which introduces redundant information and complicates the matching process. Furthermore, matching at the top layer presents significant issues with millimeter-wave radar point clouds being affected by noise and ghost points, requiring further improvement in data accuracy.

[0055] In one embodiment, the preset loss function includes a key point detection loss function, a key point matching loss function, and a dense point matching loss function, and the expressions are: (13) Where, represents the total loss function; represents the key point detection loss function; represents the key point matching loss function; represents the dense point matching loss function.

[0056] Among them, the key point detection loss should show stability in both millimeter wave radar and lidar point clouds. Therefore, the expression of the key point detection loss function is: (14) Where, Indicates the number of lidar key points; Indicates the first key points; Indicates the first key points; Indicates the number of millimeter wave radar key points; Indicates the millimeter wave radar key points; Indicates the millimeter wave radar A key point.

[0057] The key point matching loss needs to consider the overlapping circular loss to guide the network to match key points with high overlap. Therefore, the key point matching loss function is and millimeter-wave radar The average value of the overlap-aware circular loss on , is expressed as: (14) (15) (16) Where, represents the lidar keypoint matching loss; represents the millimeter-wave radar key point matching loss; Represents Positive point blocks with at least 10% overlap; Represents There are no overlapping blocks of negative points; and Both represent feature distance; express and The overlap ratio between and Represent the positive and negative weights respectively, Indicates the A block of raw lidar points; Indicates the Millimeter-wave radar original point blocks; Indicates the Millimeter-wave radar original point blocks; represents the weight ratio; Represents all lidar point blocks; The hyperparameter settings follow the convention: 、 .

[0058] The expression of the dense point matching loss function is: (17) (18) Where, Indicates the The dense point matching loss corresponding to the key point matching; Indicates the number of key point matches; ; Indicates Middle The soft assigned value of the true match of the points; Indicates Middle The soft assigned value of the true match of the points; Indicates a matching threshold True matching pairs; and They represent the row length and column width of the matrix when the unmatched subset is not added; and Respectively represent the row length and column width of the soft allocation matrix; Indicates the soft allocation value; represents a tunable hyperparameter.

[0059] In one embodiment, the superiority of the method proposed by the present invention is demonstrated. Figure 2 As shown in the figure, in an urban road scene, the millimeter-wave radar point cloud is projected onto the map constructed by the lidar point cloud using the external parameter parameters estimated by the method proposed in the present invention. The millimeter-wave radar point cloud is represented in red, and the dense map constructed by the lidar is represented in gray. It can be observed from the figure that through the accurate estimation of the external parameters by the method proposed in the present invention, the two point clouds can achieve relatively precise overlap and alignment in space. This experiment shows that the method proposed in the present invention can be used as a core module in a multi-sensor fusion system to support high-precision point cloud alignment tasks, ensure the quality of map construction, and improve the overall accuracy of the robot's environmental understanding, demonstrating its potential in subsequent cross-modal positioning.

[0060] like Figure 3 As shown in the figure, in order to further verify the accuracy and consistency of the calibration method, the estimated extrinsic parameters are applied to the cross-frame splicing of millimeter-wave radar point clouds. Specifically, the high-precision odometer pose provided by the lidar and the extrinsic parameters estimated by RLCNet are used to convert the millimeter-wave radar point clouds of consecutive frames into the world coordinate system and accumulate them to construct a millimeter-wave radar map. The results show that the extrinsic parameters obtained by the method proposed in the present invention have a clear structure of the spliced ​​millimeter-wave radar point cloud map, which is highly consistent with the layout of the real environment and is significantly better than the results obtained by splicing comparison methods such as LCR-Net and ICP. This shows that the extrinsic parameters estimated by RLCNet are still highly stable in the long-term, multi-frame data superposition process, can avoid point cloud drift, misalignment and other phenomena, and have good temporal consistency, demonstrating its potential in multi-sensor fusion odometers.

[0061] Although this embodiment Figure 1 The steps in the diagram are shown in the order indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0062] Example 2 Based on the millimeter wave radar and laser radar external parameter calibration method in Example 1, this embodiment discloses a millimeter wave radar and laser radar external parameter calibration device, such as Figure 4 As shown, the millimeter wave radar and lidar external parameter calibration device includes: a data set construction module 401, a model construction module 402, a key point matching module 403, a dense point matching module 404 and a joint calibration module 405, wherein: The data set construction module 401 is used to align the millimeter wave radar data and the lidar data to construct a data set.

[0063] The model building module 402 is used to build an external parameter calibration matching model.

[0064] The key point matching module 403 is used to extract key points from the data set based on a preset loss function and an external parameter calibration matching model, and perform feature matching based on the key points to obtain key point matching data.

[0065] The dense point matching module 404 is used to distribute the key point matching data to adjacent areas based on the key point matching data using a point-to-node strategy to obtain dense point matching data.

[0066] The joint calibration module 405 is used for the dense point matching data to obtain joint calibration parameters of the millimeter wave radar and the lidar.

[0067] In this embodiment, the specific working process and working principle of the data set construction module 401, the model construction module 402, the key point matching module 403, the dense point matching module 404 and the joint calibration module 405 are the same as those in the method of Example 1, and therefore will not be described in detail in this embodiment. Each unit module can be implemented in whole or in part by software, hardware or a combination thereof. Each unit module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above unit modules.

[0068] Example 3 like Figure 5The terminal device disclosed in this embodiment includes a transmitter, a receiver, a memory, and a processor. The transmitter is used to send instructions and data, the receiver is used to receive instructions and data, the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions stored in the memory to implement the method in the above-mentioned embodiment 1.

[0069] It should be noted that the above memory can be independent or integrated with the processor. When the memory is independently provided, the terminal device further includes a bus for connecting the memory and the processor.

[0070] Example 4 This embodiment discloses a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the method in the above-mentioned embodiment 1 is implemented.

[0071] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0072] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A method for calibrating extrinsic parameters of millimeter-wave radar and laser radar, characterized in that: The method comprises: Align millimeter-wave radar data and lidar data to build a data set; Constructing an external parameter calibration matching model, based on a preset loss function, the external parameter calibration matching model extracts key points from the data set, and performs feature matching based on the key points to obtain key point matching data; Based on the key point matching data, a point-to-node strategy is adopted to distribute the key point matching data to adjacent areas to obtain dense point matching data; Based on the dense point matching data, joint calibration parameters of the millimeter wave radar and the lidar are obtained.

2. The millimeter wave radar and laser radar extrinsic parameter calibration method according to claim 1, characterized in that: Align millimeter-wave radar data and lidar data to build a dataset, including: Performing time alignment and spatial alignment on the millimeter-wave radar data and the laser radar data to obtain initial alignment data; then performing refined alignment on the initial alignment data to obtain precision alignment data; The precision alignment data is enhanced by random external parameter transformation to obtain enhanced data. Then, the yaw angle is preset, and point cloud data with the same direction as the millimeter-wave radar and the lidar are selected to construct a data set.

3. The millimeter wave radar and laser radar extrinsic parameter calibration method according to claim 1, characterized in that: The external parameter calibration matching model includes a key point extraction module, a key point matching module and a dense point matching module; Extract key points from the data set using the key point extraction module; perform feature matching on the key points using the key point matching module to obtain key point matching data; The dense point matching module adopts a point-to-node strategy to distribute the key point matching data to adjacent areas to obtain dense point matching data.

4. The millimeter wave radar and laser radar extrinsic parameter calibration method according to claim 3, characterized in that: Extracting key points from the data set by the key point extraction module includes: The key point extraction module uses a single shared encoder to downsample and encode the laser radar point cloud data and the millimeter wave radar point cloud data layer by layer to obtain evenly distributed nodes and corresponding features. ; Nodes and corresponding features based on uniform distribution , through linear projection, the lidar point cloud data Mapping to millimeter wave radar point cloud data Then judge the lidar point cloud data Millimeter wave radar point cloud data Are the point clouds in the same? If they are the same, they are directly mapped by linear projection; if they are different, a cross-attention operation is performed to obtain enhanced features of point cloud interaction; Based on the enhanced features, the nodes and corresponding features Estimating the geometric offset and mapping it to the candidate key points to obtain the candidate key points; The candidate key points are clustered, and a candidate key point is randomly selected from each cluster as the final key point.

5. The millimeter wave radar and laser radar extrinsic parameter calibration method according to claim 3, characterized in that: Performing feature matching on the key points by the key point matching module to obtain key point matching data includes: Based on the key points, similarity between the key points of the millimeter wave radar and the lidar is constructed by calculating the Gaussian correlation matrix to obtain similarity elements; Perform double normalization on the similarity elements to obtain normalized elements; select the element with the largest normalized value The normalized elements are used and their corresponding indexes are used as key point correspondences to complete feature matching and obtain key point matching data.

6. The millimeter wave radar and laser radar extrinsic parameter calibration method according to claim 3, characterized in that: The dense point matching module adopts a point-to-node strategy to distribute the key point matching data to adjacent areas to obtain dense point matching data; including: Based on the point-to-node strategy, each key point in the key point matching data is assigned to the nearest key point to obtain a neighborhood point set; In the neighborhood point set, the point-level descriptor of each key point is restored by KPDecoder, and a matching score matrix is ​​calculated based on the descriptor; when calculating the matching score matrix, an unmatched subset is constructed; Based on the matching score matrix, the soft assignment matrix is ​​solved by the Sinkhorn algorithm, and the millimeter-wave radar point cloud is assigned to the corresponding lidar point cloud or to the unmatched subset to obtain the final dense point matching data.

7. The millimeter wave radar and laser radar extrinsic parameter calibration method according to claim 3, characterized in that: The pre-set loss functions include key point detection loss function, key point matching loss function and dense point matching loss function; the expressions are: ； in, ； ； ； Where, represents the total loss function; represents the key point detection loss function; represents the key point matching loss function; represents the dense point matching loss function; Indicates the number of lidar key points; Indicates the first key points; Indicates the number of millimeter wave radar key points; Indicates the millimeter wave radar key points; represents the lidar keypoint matching loss; represents the millimeter-wave radar key point matching loss; Indicates the number of key point matches; Indicates the The dense point matching loss corresponding to the key point matching.

8. A millimeter wave radar and laser radar external parameter calibration device, characterized in that: The device comprises: The dataset construction module is used to align millimeter-wave radar data and lidar data to construct a dataset; A key point matching module is used to construct an external parameter calibration matching model. Based on a preset loss function, the external parameter calibration matching model extracts key points from the data set and performs feature matching based on the key points to obtain key point matching data; A dense point matching module is used to distribute the key point matching data to adjacent areas based on the key point matching data using a point-to-node strategy to obtain dense point matching data; A joint calibration module is used to obtain joint calibration parameters of the millimeter wave radar and the lidar based on the dense point matching data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the millimeter wave radar and lidar external parameter calibration method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the millimeter wave radar and lidar external parameter calibration method described in any one of claims 1 to 7 are implemented.

Images