Centimeter-level millimeter wave radar point cloud registration system for multi-vehicle cooperative perception

Abstract

The invention discloses a centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle cooperative perception, and the system is characterized in that each point cloud generation assembly is used for generating a 3D point cloud of a corresponding intelligent driving vehicle based on SAR imaging, and a frame synchronization assembly is used for carrying out the frame synchronization of the 3D point cloud generated by each intelligent driving vehicle; the shared target screening assembly is used for screening a shared target according to the target detection result set after frame synchronization correction, and the point cloud registration assembly is used for registering 3D point cloud data of multiple vehicles according to the shared target between different vehicles. According to the method, features can be effectively extracted from sparse, disordered and semantic information lacking millimeter wave radar data, so that registration is carried out on millimeter wave radar point clouds among vehicles, and the perception capability of the automatic driving vehicle in a multi-vehicle cooperative perception scene is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of millimeter-wave radar point cloud registration. More specifically, it relates to a centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle collaborative perception. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, multi-vehicle collaborative perception has gradually become an important technical paradigm for enhancing the safety of autonomous vehicles. By sharing sensor data among vehicles, this technology not only significantly expands the perception range but also effectively improves the target perception accuracy, thereby enhancing the reliability and safety of the autonomous driving system in complex traffic scenarios. In this technical framework, accurate registration of sensor data between vehicles is particularly crucial. In particular, registering the raw data can provide the vehicle with finer-grained and more complete vision information, thereby improving the performance of diverse applications such as path planning and target detection. As an emerging perception modality, millimeter-wave radar has become a commonly equipped sensor for autonomous vehicles due to its strong penetration and low computational cost. However, while vision-based point cloud registration technology has been widely studied in traffic scenarios, the registration of millimeter-wave radar point clouds has not been fully explored.

[0003] Currently, many methods have been proposed for the research of point cloud registration. Some methods perform registration by extracting key points between vehicles, but there are three main limitations as follows: (1) The registration process usually requires a large amount of iterative calculation and even relies on complex deep learning models, resulting in a high computational cost; (2) Due to the dynamic change of the perspective caused by vehicle movement, the significant perspective difference between vehicles will make the registration result unstable; (3) Each point cloud frame contains hundreds of thousands of points, and the density, noise, and outliers of the points vary greatly, which further increases the difficulty of registration. There are also some methods that focus on the utilization of object-level and feature-level information, and obtain the transformation matrix between vehicles by extracting the semantic information and significant features of the object. However, the performance of these methods highly depends on the accuracy of object semantic or feature extraction, so they are not applicable to sparse, disordered, and semantically information-lacking millimeter-wave radar point clouds. In addition, in complex traffic scenarios, factors such as the high-speed movement of vehicles, multi-perspective changes, and multi-path reflections further exacerbate the challenge of millimeter-wave radar point cloud registration. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle collaborative perception, which can effectively extract features from sparse, disordered, and semantically information-lacking millimeter-wave radar data, thereby registering the millimeter-wave radar point clouds between vehicles to effectively improve the perception ability of autonomous vehicles in multi-vehicle collaborative perception scenarios.

[0005] To achieve the above invention object, the centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle collaborative perception of the present invention includes a point cloud generation component, a frame synchronization component, a shared target screening component, and a point cloud registration component. Denote the number of intelligent driving vehicles in the multi-vehicle collaborative perception system as M, and select one intelligent driving vehicle as the registration vehicle from them. A point cloud generation component is respectively deployed on each intelligent driving vehicle, and the frame synchronization component, the shared target screening component, and the point cloud registration component are deployed on the registration vehicle; where:

[0006] Each point cloud generation component is respectively used to generate the 3D point cloud of the corresponding intelligent driving vehicle based on SAR imaging and send it to the frame synchronization component of the registration vehicle; the specific method for generating the 3D point cloud is:

[0007] S1.1: Use SAR imaging to generate N p 2D planar images A n , where n = 1, 2,..., N p , N p represents the number of transmit / receive antenna pairs in the millimeter-wave radar system;

[0008] S1.2: For each pixel (x i , y i ), estimate its beam vector using the following formula

[0009]

[0010] where R is the spatial covariance matrix, the superscript -1 represents taking the inverse matrix, and the superscript H represents taking the conjugate transpose;

[0011] Then, according to the formula the height h i of the scattering point corresponding to the pixel (x i , y i ) can be calculated, where represents the spatial frequency of the y-axis origin, and σ v represents the spacing between the transmit / receive antennas;

[0012] S1.3: For each pixel in the 2D planar image A n , calculate the amplitude standard deviation of its adjacent pixels, and determine whether the reflection amplitude of this pixel is lower than one standard deviation. If so, regard it as an alternative significant pixel, otherwise do nothing; set the reflection amplitude threshold of the significant pixel according to the actual situation, and regard the pixel with a reflection amplitude greater than this reflection amplitude threshold among the alternative significant pixels as the significant pixel; then keep the reflection amplitude of the significant pixels in the 2D planar image A n unchanged, and set the reflection amplitude of the remaining pixels to 0;

[0013] S1.4: Determine the 3D coordinates (x p , y i , h i ) of each scattering point based on the coordinates and height of each scattering point in the N i 2D planar images, and generate a 3D point cloud according to the reflection amplitude of each scattering point determined from the denoised 2D planar image;

[0014] The frame synchronization component is used to perform frame synchronization correction on the 3D point cloud generated by each intelligent driving vehicle, and then send the frame-synchronized and corrected 3D point cloud to the shared target determination component. The specific method for 3D point cloud frame synchronization is as follows:

[0015] S2.1: Select the latest 3D point cloud from the 3D point clouds sent by the point cloud generation component of each intelligent driving vehicle, and record the corresponding time as t m . The 3D point cloud is At the same time, obtain the 3D point cloud at time t m - 1

[0016] S2.2: Use a preset target detection model to extract 3D targets from each 3D point cloud respectively. Denote the target detection result of the 3D point cloud P m,τ as where τ ∈ {t m , t m - 1}, K m,τ represents the number of 3D targets in the 3D point cloud P m,τ , represents the k-th 3D target point cloud cluster in the 3D point cloud P m,τ , k = 1, 2,..., K m,τ , represents the category of the target point cloud cluster , represents the center coordinates of the target point cloud cluster , represents the 3D bounding box data of the target point cloud cluster , represents the orientation angle of the target point cloud cluster , represents the confidence level of the target point cloud cluster ;

[0017] S2.3: For each intelligent driving vehicle, perform target tracking based on the target detection results at time t m - 1 and time t m to obtain the matching target set and the new target set and obtain the target trajectory R of each target in the matching target set ​m,q and the motion speed s m,q , q = 1, 2, …, Q m , Q m represents the set of matching targets the number of targets in the set;

[0018] S2.4: Denote the serial number of the registration vehicle as m * , then its 3D point cloud is denoted as The time of the latest frame is denoted as For the 3D point cloud of the intelligent driving vehicle received by the registration vehicle Calculate its time difference Then use the following formula to calculate the coordinates of the matching target point cloud cluster of each intelligent driving vehicle at time :

[0019]

[0020] where respectively represent the set of matching targets the coordinates of the center of the q-th target in the set at time t m , ;

[0021] Thus, the target detection result set after frame synchronization correction is obtained represents the q-th 3D target point cloud cluster after frame synchronization correction, respectively represent the category, 3D bounding box data, orientation angle and confidence of this 3D target point cloud cluster ;

[0022] The shared target screening component is used to screen shared targets according to the target detection result set after frame synchronization correction and send the shared targets to the point cloud registration component; the shared target screening component includes a node feature encoder, an edge feature encoder, a message passing graph neural network, an edge classifier and a shared target determination module, where:

[0023] The node feature encoder is used to encode the data of each 3D target point cloud cluster as the initial embedding feature of the corresponding node and send it to the message passing graph neural network, where j = 1, 2, …, J,

[0024] The edge feature encoder is used to encode the edge data between every two nodes as the initial embedding feature of the corresponding edge and send it to the message passing graph neural network, where the edge data data between any two nodes j,j′It is determined by the following method:

[0025]

[0026] where j, j′ = 1, 2, …, J, and j ≠ j′, dis j,j′ represents the distance between the central coordinates of node j and node j′, ω j,j′ represents the angle of the edge between node j and node j′, r j and r j′ are the distances from node j and node j′ to the origin of the corresponding coordinate system respectively, w j and w j′ represent the angles of the perspectives of node j and node j′ respectively;

[0027] The message-passing graph neural network is used to perform B times of space-aware embedding updates on the graph G(V, E) with 3D object point cloud clusters in the set O of M target detection results as nodes. V represents the node set, and E represents the edge set. The edge embedding features obtained by the update m are sent to the edge classifier; the specific method of edge embedding update is as follows: In the b-th embedding update, b = 1, 2, …, B, for the edges between two nodes belonging to the same vehicle, the following formula is used for edge embedding update:

[0028]

[0029]

[0030] where φ e v [] represents a preset mapping function;

[0031] For the edges between two nodes across vehicles, the following formula is used for edge embedding update:

[0032]

[0033] where φ v [] represents a preset mapping function;

[0034] The edge classifier is used to predict the association probability of the corresponding edge after receiving the edge embedding features and send the association probability to the shared target determination module;

[0035] The shared target determination module is used to judge whether the association probability of the edge between two nodes across vehicles is greater than a preset threshold. If so, the two nodes connected by the edge are the shared targets of two intelligent driving vehicles, otherwise not;

[0036] The point cloud registration component is used to register the 3D point cloud data of M vehicles according to the shared targets between different vehicles.

[0037] The present invention is a centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle collaborative perception. Each point cloud generation component is respectively used to generate the 3D point cloud of the corresponding intelligent driving vehicle based on SAR imaging. The frame synchronization component is used to perform frame synchronization on the 3D point clouds generated by each intelligent driving vehicle. The shared target screening component is used to screen shared targets according to the set of target detection results corrected by frame synchronization. The point cloud registration component is used to register the 3D point cloud data of multiple vehicles according to the shared targets between different vehicles.

[0038] The present invention has the following beneficial effects:

[0039] 1) The present invention proposes a point cloud generation method based on SAR imaging, which solves the problem of difficult semantic extraction of sparse and disordered radar point clouds;

[0040] 2) A frame synchronization method for motion perception is proposed, which solves the problem of spatio-temporal consistency of asynchronous radar frames;

[0041] 3) A point cloud registration method based on shared targets is proposed, which solves the problem of difficult matching of millimeter-wave radar point clouds of multiple vehicles and realizes centimeter-level point cloud registration. Description of the Drawings

[0042] Figure 1 It is a structural diagram of the specific implementation manner of the centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle collaborative perception of the present invention;

[0043] Figure 2 It is a flowchart of generating 3D point clouds based on SAR imaging in the present invention;

[0044] Figure 3 It is a flowchart of frame synchronization in the present invention;

[0045] Figure 4 It is a structural diagram of the shared target screening component in the present invention. Specific Embodiments

[0046] The following describes the specific implementation manner of the present invention with reference to the drawings, so that those skilled in the art can better understand the present invention. It should be particularly noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0047] Embodiment

[0048] Figure 1 It is a structural diagram of the specific implementation manner of the centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle collaborative perception of the present invention. As Figure 1As shown in the figure, the centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle collaborative perception of the present invention includes multiple point cloud generation components, a frame synchronization component, a shared target screening module, and a point cloud registration module. For the point cloud generation components, the frame synchronization component, the shared target screening component, and the point cloud registration component, denote the number of intelligent driving vehicles in the multi-vehicle collaborative perception system as M, and select one intelligent driving vehicle as the registration vehicle from them. A point cloud generation component is respectively deployed on each intelligent driving vehicle, and a frame synchronization component, a shared target determination component, and a point cloud registration component are deployed on the registration vehicle. Next, each module will be described in detail.

[0049] Each point cloud generation component is respectively used to generate a 3D point cloud of the corresponding intelligent driving vehicle based on SAR imaging and send it to the frame synchronization component of the registration vehicle.

[0050] Considering that the 3D point cloud generated by the vehicle millimeter-wave radar is sparse and disordered, and the target semantic information is difficult to be effectively extracted by the target detection model, inspired by the SAR (Synthetic Aperture Radar) imaging technology, the present invention proposes a point cloud generation component based on SAR imaging. This component uses SAR imaging technology to improve the resolution and density of the radar point cloud, providing support for subsequent point cloud registration.

[0051] Typical SAR imaging generates a 2D image only along the x-y plane by moving the radar antenna, which stacks targets at different heights onto a plane, resulting in occlusion problems. Although the small aperture of the vehicle radar limits its ability to directly image in the vertical direction, by combining the transmit / receive (Tx / Rx) antennas in the vertical direction and digital beamforming technology, SAR imaging can be performed at different elevation angles, effectively distinguishing the height information of the targets. However, this method has two limitations: (i) High-quality SAR imaging depends on synthesizing a large amount of elevation data, which requires extensive beam scanning and is thus not applicable to time-sensitive autonomous driving scenarios; (ii) Due to the diversity of targets in the traffic scene, the height resolution is non-uniform, and the resolution significantly decreases as the detection distance increases. Therefore, the point cloud generation component proposed by the present invention uses aligned pixels to estimate the height information of the target scattering points and integrates the height information into the 2D plane image to construct a 3D point cloud, thereby providing a more accurate target representation. Figure 2 is the flow chart of generating a 3D point cloud based on SAR imaging in the present invention. As Figure 2 shown, the specific steps of generating a 3D point cloud based on SAR imaging in the present invention include:

[0052] S201: Obtain a 2D plane image:

[0053] Use SAR imaging to generate N p 2D plane images containing targets, where N pRepresents the number of transmit / receive antenna pairs in a millimeter-wave radar system. To improve the quality of 2D plane images, in practical applications, matched filtering and interpolation can be performed on each 2D plane image separately.

[0054] S202: Estimate the point cloud height:

[0055] The present invention needs to extract height information from the pixels of a 2D plane image. To achieve this goal, first, the phase response of each pixel is modeled. Specifically, for each 2D plane image, the phase response ∠F(s i ,y i ) of the scatterer at coordinates (x x ,s y ) can be expressed as:

[0056] ∠F(s x ,s y ) = -s x (x - x i ) - s y (y - y i )

[0057]

[0058] where F(s x ,s y ) represents the cross-range spectrum, σ i represents the sampling interval, s x and s y represent the spatial frequencies of the x-axis and y-axis respectively, represents the spatial frequency of the origin of the y-axis, f and T represent the center frequency and period of the pulse respectively, γ represents the rate of linear increase of the signal frequency, and c is the speed of light.

[0059] After applying the inverse fast Fourier transform, the phase ∠f(x, y) of the scatterer can be defined as:

[0060]

[0061] where, represents the spatial frequency of the origin of the y-axis, is a constant factor that is the same between pixels, and carries information related to distance, which is crucial for estimating the height and three-dimensional spatial depth of the target.

[0062] Based on the above analysis, the present invention applies the angle-of-arrival (AoA) estimation technique used in the antenna array to the virtual pixel array. Assume that the height of the scatterer corresponding to a certain pixel (x i ,y i ) is h i , and its elevation angle AoA can be expressed as μi = arccos(h i / y i ). For the antenna array, the array beam vector is a function of the height h i to capture the phase changes of the pixels, which depend on the relative height of the scatterers and the angle of arrival AoA. The array beam vector can be expressed by the following formula:

[0063]

[0064] where σ v represents the spacing between the transmit / receive antennas, N p represents the number of transmit / receive antenna pairs, and λ is the wavelength.

[0065] Next, the present invention uses the Capon algorithm to estimate the height of the pixels by optimizing the beam vector to maximize the point intensity from the desired direction (i.e., the incident signal) while minimizing the signal interference in other directions. Mathematically, the beam vector i for the target height h can be estimated by the following optimization problem:

[0066]

[0067] where R is the spatial covariance matrix used to capture the power distribution of signals in different directions, the superscript -1 represents taking the inverse matrix, and the superscript H represents taking the conjugate transpose.

[0068] For each pixel (x i , y i ), after estimating the beam vector using the above formula, the height h of the scatterer corresponding to the pixel (x i , y i ) can be calculated according to the formula i , where represents the spatial frequency of the origin of the y-axis, and σ v represents the spacing between the transmit / receive antennas.

[0069] S203: Extract significant pixels:

[0070] In remote sensing applications, SAR is usually deployed on aircraft or satellites to provide a top-down view, where most pixels correspond to ground targets in the real world. However, when SAR is installed on a vehicle with a lateral field of view, most of the transmitted signals pass through open space, resulting in only a small amount of reflection and causing weak pixel clusters related to the target. Therefore, it is necessary to perform noise processing on the original data to reduce unnecessary calculations. A common method is to filter pixels based on the reflection amplitude of the target. However, since the reflection intensities of targets with different materials and distances are different, setting a fixed amplitude threshold is ineffective. Therefore, the present invention improves this method to highlight significant pixels. The specific method is as follows:

[0071] For each pixel in the 2D planar image A n calculate the standard deviation of the amplitudes of its adjacent pixels, and determine whether the reflection amplitude of this pixel is lower than one standard deviation. If so, regard it as an alternative significant pixel; otherwise, do nothing. Set the reflection amplitude threshold for significant pixels according to the actual situation, and regard the pixels among the alternative significant pixels whose reflection amplitudes are greater than this reflection amplitude threshold as significant pixels. Then keep the reflection amplitudes of the significant pixels in the 2D planar image A n unchanged, and set the reflection amplitudes of the remaining pixels to 0, thereby obtaining the processed 2D planar image

[0072] By adopting the above method, the saliency of the target can be enhanced, thus facilitating subsequent target extraction. It should be noted that including more adjacent pixels can enhance the saliency of the central pixel, but it also increases the risk of mis-screening valid points. Therefore, the threshold size can be adjusted to balance the point cloud density and the number of noise false alarms.

[0073] In addition to the above method, the 2D planar image can also be further denoised and optimized based on PCA analysis. The specific method is as follows:

[0074] Convert N p 2D planar images into vectors respectively, and form a two-dimensional matrix with N p vectors as column vectors. Perform PCA analysis on this two-dimensional matrix, select the first principal component, and reverse process and restore it to N p 2D planar images A n '. Then, based on each 2D planar image A 1 ', use the L n -TV model to derive the optimized 2D planar image n that is closest to the original 2D planar image A

[0075] Through the above denoising optimization based on PCA analysis, the effective pixels can be more prominent and the pixels affected by noise can be suppressed, further reducing the noise during pixel extraction, thereby minimizing the influence of random noise to the greatest extent.

[0076] S204: Generate 3D point cloud:

[0077] According to the coordinates and heights of each scattering point in N p 2D planar images, determine the 3D coordinates (x i , y i , h i ) of each scattering point, and determine the reflection amplitude of each scattering point according to the denoised 2D planar image to generate a 3D point cloud.

[0078] During the process of generating the 3D point cloud, the depth information d i , y i , h i ) of each scattering point can be calculated, and the calculation formula is: i For each scattering point, its intensity information can be calculated through beamforming, and the beam steering vector b

[0079]

[0080] : i :

[0081]

[0082] Among them, represents the vector after alignment of the corresponding transmit / receive antenna pair.

[0083] The frame synchronization component is used to perform frame synchronization correction on the 3D point cloud generated by each intelligent driving vehicle, and then send the frame synchronization corrected 3D point cloud to the shared target determination component.

[0084] Due to problems such as inconsistent frame rates, data transmission delays, and frame loss between different intelligent driving vehicles, data is often unable to be aligned in time and space, affecting the accuracy of multi-vehicle collaborative perception. Therefore, in the present invention, the frame synchronization component is used to maintain the spatio-temporal consistency of the scene by performing lightweight tracking on the target motion trajectory, effectively alleviating the influence of asynchronous radar frames and unpredictable transmission and calculation delays between multiple vehicles. At the same time, the radar frames between multiple vehicles are dynamically corrected according to the motion speed of the target, thereby achieving precise alignment of radar data and ensuring the spatio-temporal consistency of radar data between multiple vehicles. Figure 3 is the flowchart of the frame synchronization correction of the present invention. As Figure 3 shown, the specific steps of the frame synchronization correction in the present invention include:

[0085] S301: Obtain 3D point cloud:

[0086] Select the latest 3D point cloud from the 3D point clouds sent by the point cloud generation components of each intelligent driving vehicle, and record the corresponding moment as t m , and the 3D point cloud is At the same time, obtain the 3D point cloud at time t m -1 m = 1, 2, …, M.

[0087] S302: Extract 3D targets:

[0088] Use a preset target detection model to extract 3D targets from each 3D point cloud respectively, and record the target detection result of the 3D point cloud P m,τ as where τ ∈ {t m , t m -1}, K m,τ represents the number of 3D targets in the 3D point cloud P m,τ , represents the k-th 3D target point cloud cluster in the 3D point cloud P m,τ , k = 1, 2, …, K m,τ , represents the category of the target point cloud cluster , represents the center coordinates of the target point cloud cluster , represents the 3D bounding box data of the target point cloud cluster , represents the orientation angle of the target point cloud cluster , represents the confidence level of the target point cloud cluster .

[0089] In this embodiment, the target detection model uses the Occ3D model. It should be noted that the target detection of each intelligent driving vehicle is performed independently, and its detection result is relative to its own radar coordinate system.

[0090] S303: Target tracking:

[0091] For each intelligent driving vehicle, perform target tracking based on the target detection results at time t m -1 and time t m to obtain the matching target set and the new target set and obtain the target trajectory R and the motion speed v m,q of each target in the matching target set m,q , q = 1, 2, …, Q m , Qm Represents the matching target set The number of targets in it.

[0092] Current target tracking technologies rely on deep learning models. Although they have good effects, they have high computational costs and are therefore not suitable for latency-sensitive applications in traffic scenarios. The target tracking algorithm adopted by the present invention uses a combined algorithm of the Kalman filter algorithm and the Hungarian algorithm to achieve low-latency target tracking. This algorithm is a commonly used algorithm in the field of target tracking, and its specific process will not be elaborated here.

[0093] S304: Frame synchronization correction:

[0094] In practical applications, the 3D point cloud frames of multiple vehicles are not always synchronized. The main reasons are as follows: (i) Frame difference: The commonly used data sampling frame rate of millimeter-wave radars is 20Hz, which may cause frame offsets between vehicles to reach dozens of milliseconds; (ii) Frame loss: Due to the heterogeneity of the computing capabilities of different vehicles, target detection may be performed at different frequencies. In addition, packet loss may occasionally occur in the data transmission between multiple vehicles. Therefore, the vehicle serving as the registration server may lose some radar frames from other vehicles. For this reason, the present invention propagates the target state of 3D target tracking to the current frame to estimate its state in the next frame.

[0095] Denote the serial number of the registered vehicle as m * , then its 3D point cloud is denoted as The time of the latest frame is denoted as For the 3D point cloud of the intelligent driving vehicle received by the registered vehicle Calculate its time difference Then use the following formula to calculate the central coordinates of the matching target point cloud cluster of each intelligent driving vehicle at time :

[0096]

[0097] Among them, respectively represent the matching target set The central coordinates of the q-th matching target point cloud cluster in it at time t m , .

[0098] For other states, through experimental observation, other target states usually change very little within milliseconds. Therefore, the present invention adopts a constant velocity model to minimize the inter-frame calculation and communication overhead to efficiently synchronize the radar data of the registered vehicle and other vehicles. Thus, the target detection result set after frame synchronization correction is obtained Represents the q-th 3D target point cloud cluster after frame synchronization correction, respectively represent this 3D target point cloud cluster category, 3D bounding box data, orientation angle, and confidence.

[0099] The shared target screening component is used to screen shared targets according to the set of object detection results after frame synchronization correction, and send the shared targets to the point cloud registration component.

[0100] After synchronizing the 3D point cloud frames of other vehicles, the next task of the registering vehicle is to identify shared targets from two different object detection results for accurate registration. A straightforward approach is to construct a detected object graph and compare the similarity of nodes and edges to identify shared targets. However, this method has two limitations: (i) In complex and dynamic traffic scenarios, similar vehicles may create locally indistinguishable graph structures, making it difficult for the registering vehicle to accurately identify shared targets; (ii) Existing methods usually rely on speed estimation to establish relationships between nodes. However, even with the enhanced density of radar point clouds, the accuracy of speed estimation is still relatively low compared to high-resolution LiDAR data. Therefore, different from simply comparing local structural similarities, the present invention adopts a learning-based method to better understand and associate multiple vehicles in the view. Figure 4 is the structural diagram of the shared target screening component in the present invention. As Figure 4 shown, the shared target screening component in the present invention includes a node feature encoder, an edge feature encoder, a Message Passing Neural Network (MPNN), an edge classifier, and a shared target determination module, where:

[0101] The node feature encoder is used to encode the data of each 3D object point cloud cluster as the initial embedding feature of the corresponding node and send it to the Message Passing Neural Network, where j = 1, 2,..., J, It can be seen that in the present invention, the original 3D object point cloud cluster features are not directly input into the graph, but the 3D object point cloud cluster features are feature-encoded to obtain the initial embedding features of the corresponding target nodes.

[0102] The edge feature encoder is used to encode the edge data between every two nodes as the initial embedding feature of the corresponding edge and send it to the Message Passing Neural Network, where the edge data data j,j′ between any two nodes is determined by the following method:

[0103]

[0104] where j, j' = 1, 2,..., J, and j ≠ j', dis j,j′denotes the distance between the central coordinates of node j and node j′, ω j,j′ denotes the angle of the edge between node j and node j′, r j and r j′ are the distances from node j and node j′ to the origin of the corresponding coordinate system respectively, w j 、w j′ denote the angles of the perspectives of node j and node j′ respectively. All the above information can be calculated from the data of the 3D object point cloud cluster.

[0105] It can be seen that in the present invention, the edges are classified. One type is the edges connecting nodes within the same vehicle, and the other type is the edges connecting nodes across different vehicle spaces. Subsequently, the embedding feature updates of the two types of edges will be processed separately.

[0106] The message passing graph neural network is used to perform B times of space-aware embedding updates on the graph G(V, E) with 3D object point cloud clusters in the M target detection result sets O m as nodes, where V represents the node set and E represents the edge set, and the updated edge embedding features are sent to the edge classifier.

[0107] To avoid misidentifying shared targets in the local structure, the present invention focuses on capturing the global position of each node in the graph through the message passing graph neural network MPNN. The message passing graph neural network MPNN can play an important role in propagating node and edge embedding information in the graph. However, existing MPNN-based learning methods usually focus on the continuous frame association of a single vehicle and are not applicable to multi-vehicle cooperation scenarios, and nodes and edges exist in two different coordinate systems. For this reason, the present invention proposes a space-aware embedding update method to learn and interpret the global position of each node while updating the node and edge embeddings in the local vehicle and global coordinate spaces. Since the registered vehicle and other vehicles are in different coordinate systems, according to the construction method of the multi-vehicle target graph, the updates of the two types of edge embeddings are processed separately. The specific method is as follows:

[0108] In the b-th embedding update, b = 1, 2, …, B, for the edges between two nodes belonging to the same vehicle, the following formula is used for edge embedding update:

[0109]

[0110] where, φ e [] represents a preset mapping function.

[0111] For the edges between two nodes across vehicles, the following formula is used for edge embedding update:

[0112]

[0113] where, φv [] indicates a preset mapping function.

[0114] The edge classifier is used to receive the edge embedding features Then, predict the association probability of the corresponding edge And send the association probability to the shared target determination module.

[0115] The shared target determination module is used to determine whether the association probability of the edge between two nodes across the vehicle is greater than a preset threshold. If so, the two nodes connected by the edge are shared targets of the two intelligent driving vehicles, otherwise not.

[0116] In practical applications, it is necessary to first collect several pairs of shared target screening components for training. The calculation formula of the loss function ξ used in the training of the shared target screening component in this embodiment is as follows:

[0117]

[0118] Among them, y j,j′ represents the true probability of whether node j and node j′ are shared targets, y j,j′ =1 means that node j and node j′ are shared targets, y j,j′ = 0 indicates that node j and node j′ are not shared targets, and w represents a weight factor used to compensate for the class imbalance between edges representing shared targets and other targets.

[0119] The point cloud registration component is used to register the 3D point cloud data of M vehicles according to the shared targets between different vehicles.

[0120] Since the bounding box and 3D position of the shared target have been obtained in the shared target screening component, the bounding box corner points can be considered as key point pairs and input into the RANSAC algorithm for registration, and the transformation matrix between the vehicle to be registered and other vehicles can be iteratively calculated. It should be noted that the accuracy of the transformation matrix depends to a large extent on the performance of target tracking and shared target matching. Fortunately, the point cloud generation component proposed in the present invention can enhance the quality of sparse radar point clouds and efficiently improve the performance of both. Therefore, the calculated transformation matrix has strong robustness. Ultimately, the present invention can achieve centimeter-level millimeter-wave radar point cloud registration in multi-vehicle scenarios and serve target perception applications with high precision requirements.

[0121] The experimental results show that the present invention significantly improves the perception range and perception accuracy of the vehicle. The effective perception range of the vehicle is increased by 117%, the relative translation error is reduced to 0.08 m, the relative rotation error is reduced to 1.83°, and the success rate of point cloud registration reaches 95.65%. At the same time, if the interval frame registration strategy is adopted, a frame processing speed of 20 fps can be achieved, meeting the real-time requirements of autonomous vehicles.

[0122] Although the above-described illustrative specific embodiments of the present invention have been described to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions made using the concept of the present invention are within the scope of protection.

Claims

1. A centimeter-level millimeter-wave radar point cloud registration system for multi-vehicle collaborative perception, characterized in that: It includes a point cloud generation component, a frame synchronization component, a shared target screening component and a point cloud registration component. The number of intelligent driving vehicles in the multi-vehicle collaborative perception system is M, and one intelligent driving vehicle is selected as the registration vehicle. A point cloud generation component is deployed on each intelligent driving vehicle, and a frame synchronization component, a shared target screening component and a point cloud registration component are deployed on the registration vehicle; wherein: Each point cloud generation component is used to generate a 3D point cloud corresponding to the intelligent driving vehicle based on SAR imaging and send it to the frame synchronization component of the registration vehicle; the specific method of generating a 3D point cloud is: S1.1: Generate N containing the target using SAR imaging p A 2D plane image A n , where n = 1, 2, ..., N p , N p Represents the number of transmit / receive antenna pairs in the millimeter wave radar system; S1.2: For each pixel (x i ,y i ) uses the following formula to estimate its beam vector Where R is the spatial covariance matrix, the superscript -1 indicates that the inverse matrix is ​​obtained, and the superscript H indicates that the conjugate transpose is obtained; Then according to the formula The pixel (x i ,y i ) corresponds to the height h of the scattering point i ,in represents the spatial frequency of the origin of the y-axis, σ v Indicates the spacing between the transmitting / receiving antennas; S1.3: For a 2D plane image A n For each pixel in the image, calculate the amplitude standard deviation of its adjacent pixels, and determine whether the reflection amplitude of the pixel is lower than one standard deviation. If so, take it as a candidate significant pixel, otherwise do nothing. Set the reflection amplitude threshold of the significant pixel according to the actual situation, and take the pixels whose reflection amplitude is greater than the reflection amplitude threshold among the candidate significant pixels as significant pixels. Then take the 2D plane image A n The reflection amplitude of the significant pixels in the image remains unchanged, and the reflection amplitude of the remaining pixels is set to 0; S1.4: According to N p The coordinates and height of each scattering point in the 2D plane image are determined to determine the 3D coordinates (x i ,y i ,h i ), determine the reflection amplitude of each scattering point according to the denoised 2D plane image and generate a 3D point cloud; The frame synchronization component is used to perform frame synchronization correction on the 3D point cloud generated by each intelligent driving vehicle, and then send the 3D point cloud after frame synchronization correction to the shared target determination component. The specific method of 3D point cloud frame synchronization is: S2.1: Select the latest 3D point cloud from the 3D point cloud sent by the point cloud generation component of each intelligent driving vehicle, and record its corresponding time as t m , the 3D point cloud is At the same time, get t m -1 3D point cloud at a moment m=1,2,…,M; S2.2: Use the preset target detection model to detect each 3D point cloud Extract the 3D target from the 3D point cloud P m,τ The target detection result is Among them, τ∈{t m ,t m -1}, K m,τ Represents a 3D point cloud P m,τ The number of 3D objects in Represents a 3D point cloud P m,τ The kth 3D target point cloud cluster in, k = 1, 2, ..., K m,τ , Represents the target point cloud cluster Category, Represents the target point cloud cluster The center coordinates of Represents the target point cloud cluster 3D bounding box data, Represents the target point cloud cluster The orientation angle, Represents the target point cloud cluster Confidence level; S2.3: For each intelligent driving vehicle according to t m -1 moment and t m The target detection results at each moment are used to track the target and obtain the matching target set and the new target set And get the matching target set The target trajectory R of each target in m,q and the movement speed s m,q ,q=1,2,…,Q m , Q m Represents the matching target set Number of targets; S2.4: The serial number of the registered vehicle is m * , then its 3D point cloud is recorded as The time of the latest frame is recorded as For the 3D point cloud of the intelligent driving vehicle received by the registration vehicle Calculate the time difference Then the following formula is used to calculate the time of each intelligent driving vehicle The matching target point cloud cluster coordinates: in, Respectively represent the matching target set The qth matching target point cloud cluster in time t m , The center coordinates of Thus, we can obtain the target detection result set after frame synchronization correction. represents the qth 3D target point cloud cluster after frame synchronization correction, Respectively represent the 3D target point cloud cluster Category, 3D bounding box data, orientation angle and confidence; The shared target screening component is used to screen the shared targets according to the target detection result set after frame synchronization correction, and send the shared targets to the point cloud registration component; the shared target screening component includes a node feature encoder, an edge feature encoder, a message passing graph neural network, an edge classifier and a shared target determination module, wherein: The node feature encoder is used to encode each 3D target point cloud cluster Data Encode as the initial embedding feature of the corresponding node And sent to the message passing graph neural network, where j = 1, 2, ..., J, The edge feature encoder is used to encode the edge data between every two nodes as the initial embedding feature of the corresponding edge. And sent to the message passing graph neural network, where the edge data between any two nodes j,j′ Determine using the following method: Among them, j, j′=1,2,…,J, and j≠j′, dis j,j′ represents the distance between the center coordinates of node j and node j′, ω j,j′ represents the angle of the edge between node j and node j′, r j and r j′ are the distances from node j and node j′ to the origin of the corresponding coordinate system, w j 、w j′ denote the angles of the viewing angles of node j and node j′ respectively; The message passing graph neural network is used to detect the M target detection results set O m The graph G(V,E) with 3D target point cloud clusters as nodes is updated B times based on spatial perception. V represents the node set and E represents the edge set. The updated edge embedding features Sent to the edge classifier; the specific method of edge embedding update is: In the b-th embedding update, b = 1, 2, ..., B, for the edge between two nodes belonging to the same vehicle, the following formula is used to perform edge embedding update: Among them, φ e [] indicates the preset mapping function; For the edge between two nodes across vehicles, the following formula is used to update the edge embedding: Among them, φ v [] indicates the preset mapping function; The edge classifier is used to receive the edge embedding features Then, predict the association probability of the corresponding edge And send the association probability to the shared target determination module; The shared target determination module is used to determine whether the association probability of the edge between two nodes across the vehicles is greater than a preset threshold. If so, the two nodes connected by the edge are shared targets of the two intelligent driving vehicles, otherwise not; The point cloud registration component is used to register the 3D point cloud data of M vehicles according to the shared targets between different vehicles.

2. The centimeter-level millimeter-wave radar point cloud registration system according to claim 1, characterized in that: In step S1.3, the 2D plane image is also analyzed based on PCA. Further denoising optimization is performed. The specific method is: p 2D plane image Convert them into vectors respectively, and convert N p vectors as column vectors to form a two-dimensional matrix; PCA analysis is performed on the two-dimensional matrix, the first principal component is selected, and the two-dimensional matrix is ​​reversely processed and restored to N according to the composition method of the two-dimensional matrix. p A 2D plane image A′ n ; Then use L 1 -TV model is based on each 2D plane image A′ n Derive the image A closest to the original 2D plane n Optimized 2D plane image 3. The centimeter-level millimeter-wave radar point cloud registration system according to claim 1, characterized in that: The calculation formula of the loss function ξ used by the shared target screening component during training is as follows: Among them, y j,j′ Indicates the true probability of whether node j and node j′ are shared targets, y j,j′ =1 means that node j and node j′ are shared targets, y j,j′ =0 indicates that node j and node j′ are not shared targets, and w indicates the weight factor.

Images