Millimeter wave radar point cloud registration method and device for vehicle-road cooperation automatic driving

Abstract

The application discloses a kind of millimeter wave radar point cloud registration methods and devices for vehicle-road cooperation automatic driving, belong to point cloud registration field.The application utilizes the complementary advantages of camera and millimeter wave radar, realizes the accurate registration of vehicle-infrastructure sparse disorder 3D radar point cloud, solves the problem of difficult extraction of sparse disorder radar point cloud semantic features and point pair difficult correspondence.Meanwhile, adaptive scheduling strategy is designed, considering the triggering factor of vehicle and the change degree of point cloud in infrastructure field of view to adaptively trigger data request to ensure real-time registration between multiple vehicles-infrastructure.The application takes camera data and radar point cloud data of vehicle and infrastructure as input and finally completes point cloud registration, while meeting the real-time and accuracy of millimeter wave radar point cloud registration, realizes the improvement of vehicle perception range and perception accuracy in heavy fog weather, and supports the further improvement of millimeter wave radar performance in target detection, path planning and other applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud registration, and particularly relates to a millimeter wave radar point cloud registration method and device for vehicle-road cooperation automatic driving. BACKGROUND

[0002] Automatic driving has the potential to change the way people travel, however, the safety and reliability of the automatic driving system still need to be improved. The root cause of analyzing the safety hazards of the automatic driving system lies in the limited perception range of the single vehicle perspective. Generally speaking, there are mainly two limitations: (1) the inherent properties of the sensor itself limit its perception range, and it is easy to be blocked; (2) adverse conditions, such as fog, limit the perception range and accuracy.

[0003] Infrastructure-aided automatic driving (i.e. vehicle-road cooperation) is an emerging paradigm, which utilizes roadside infrastructure, such as intelligent lamp posts equipped with perception and computing units, to share real-time information such as target detection results and traffic conditions with the automatic driving vehicles. A key enabling technology in this paradigm is to register two 3D point clouds generated by the sensors (such as laser radars, millimeter wave radars) configured by the vehicles and the infrastructure. The registered results can expand the perception range and enhance the perception ability. Existing automatic driving vehicles and intelligent roadside infrastructure are usually equipped with millimeter wave radars, which have obvious advantages in dealing with adverse conditions compared with cameras and laser radars. Therefore, it is very important to register 3D millimeter wave radar point clouds to solve the above two limitations.

[0004] At present, there are many point cloud registration methods, but they are not specifically designed for vehicle-road cooperation scene registration. When these methods are applied to the vehicle-road scene, they will produce high computational overhead and low accuracy, because they rely on similar positions and large-scale overlap between point clouds. Recently, the research on vehicle-road cooperation point cloud registration mainly focuses on two technical routes. The first one is to fuse the target detection results, which can meet the real-time requirement, but at the expense of registration accuracy. The other one is to fuse the original laser radar point clouds, which can achieve centimeter-level accuracy, but multi-vehicle fusion requires high bandwidth consumption, for example, the bandwidth required by 10 vehicles reaches 960Mbps. At the same time, the performance of these two technical routes depends heavily on semantic segmentation, but this is not suitable for millimeter wave radars, because radar point clouds are sparse and disordered, and semantic information is difficult to extract.

[0005] In summary, millimeter wave radar point cloud registration faces two unique challenges of real-time and accuracy:

[0006] (1) How to obtain an accurate vehicle-infrastructure transformation matrix using sparse and unordered radar point clouds. Existing models cannot extract the semantics of radar point clouds, and if we directly match the point clouds between the vehicle and the infrastructure, this will result in a large registration error, leading to low accuracy of the obtained transformation matrix.

[0007] (2) How to achieve real-time multi-vehicle-infrastructure registration. Consider a real traffic scenario where a person is about to cross the road but is blocked by a private car on the roadside: given a vehicle speed of 40 km / h, a braking distance of about 12 meters, and a vehicle distance from the pedestrian of 20 meters, the vehicle is allowed to use the registration result to avoid traffic accidents, and registration and target detection must be completed within (20-12) / 40 = 200 ms. The latest target detection model also needs 48 ms to detect targets, so registration must be completed within 152 ms, which does not take into account other delays such as transmission delays.

[0008] Therefore, it is necessary to develop a new method to simultaneously satisfy the real-time and accuracy of millimeter wave radar point cloud registration. SUMMARY

[0009] The purpose of the present application is to propose a millimeter wave radar point cloud registration method and device for vehicle-road cooperative autonomous driving, which utilizes the complementary advantages of millimeter wave radar and vision to achieve real-time and accurate millimeter wave radar point cloud registration between multiple vehicles and road infrastructure.

[0010] To achieve the above purpose, the present application provides the following technical solutions:

[0011] On the one hand, the present application provides a millimeter wave radar point cloud registration method for vehicle-road cooperative autonomous driving, comprising the following steps:

[0012] S1, salient target segmentation: camera data of roadside facilities and multiple vehicles are obtained using a camera, and a salient target segmenter is used to segment salient targets from images of roadside facilities and multiple vehicles, respectively, to provide static semantics for the radar branch;

[0013] S2, salient region extraction: based on static semantics and using a millimeter wave radar, radar point cloud data of roadside facilities and multiple vehicles are obtained, and a salient region extractor is used to align the camera and millimeter wave radar branches in time and spatial dimensions to obtain 3D point cloud regions corresponding to the salient targets;

[0014] S3, 3D point cloud registration: according to the 3D point cloud regions corresponding to the salient targets, key points are selected and the selected key points are iterated to obtain a transformation matrix, and the registration result is output;

[0015] The method realizes real-time point cloud registration of multiple vehicles through a multi-vehicle adaptive scheduling component, which includes a request scheduler and an adaptive trigger; the request scheduler selects key frames by weighing the number of requests, registration delay, success rate and the degree of change in vehicle perspective; for non-key frames, the request scheduler uses the state information of the target in the previous frame to update the point cloud data of the next frame; the adaptive trigger simultaneously considers the vehicle factors and the degree of change in infrastructure perspective to adaptively transmit the corresponding key points.

[0016] Further, in step S1, the salient target segmenter uses a 2D target detector to detect and segment targets from vehicles and infrastructure.

[0017] Further, in step S2, a salient region extractor is used to obtain the 3D point cloud region corresponding to the salient target, and in time, the frame of the camera branch is associated with its nearest radar frame; in space, the camera frame and radar frame measurements are converted to the same coordinate system.

[0018] The specific method is as follows:

[0019] Convert the 3D point cloud coordinates to the pixel coordinate system under the camera perspective, and the formula is as follows:

[0020]

[0021] The above formula is rewritten as:

[0022] P uv =KTP w

[0023] Where P uv represents the pixel homogeneous coordinates (u, v, 1), P w refers to the point coordinates (X w , Y w , Z w , 1) obtained by taking the millimeter wave acquisition device as the coordinate origin, K is the camera intrinsic matrix, f x and f y are the focal lengths on the X and Y axes; u0 and v0 are the principal point coordinates, and T is the camera extrinsic matrix.

[0024] The above formula maps the 2D plane of the camera branch to the 2D plane of the real-world radar point cloud to obtain the saliency region of the radar branch, then assigns the corresponding semantics to the 2D point cloud region, and maps it back to the 3D representation, thereby realizing 3D saliency region extraction of the radar point cloud.

[0025] Further, in step S3, an improved RANSAC algorithm is used to iterate on the selected key points, and the improved RANSAC algorithm includes two strategies:

[0026] (i) Key point extraction strategy

[0027] Use the geometric features of prominent targets to wrap them with spheres to obtain the corresponding key points;

[0028] (ii) Early termination strategy

[0029] By clustering and similarity calculations on salient targets, the correct direction of the transformation matrix is ​​determined. If the direction calculated by matching key points is consistent with the correct direction of the transformation matrix, the iteration process is terminated early.

[0030] Furthermore, the specific method of the key point extraction strategy is as follows: select the center point and two radius points of the sphere as key points, convert the point cloud of the salient target into a binary image, and then fit the geometric features of the target with the convex hull.

[0031] Furthermore, the specific method of the early termination strategy is as follows: cluster the pixel coordinates x of the static semantic targets extracted from the infrastructure and vehicles, and divide these targets into left and right directions; determine the direction between the vehicle and the infrastructure by comparing the number of static targets distributed on the left and right sides and the similarity of the semantic targets.

[0032] Furthermore, for keyframes, when the average number of new target points increases by k, the request scheduler sets a threshold of k to determine the degree of view change. When the change in the vehicle's field of view is greater than k, the request scheduler considers both the number of requests within 1 second and the success rate of the previous frame to decide whether to perform registration. If the success rate is lower than the set lower limit, and the remaining time is sufficient to execute multiple requests, then registration is performed. The optimization problem of maximizing the number of requests to improve the overall success rate is expressed as follows:

[0033]

[0034] Among them, A i is the success rate of the i-th request, N is the number of requests, and L is the registration delay.

[0035] Furthermore, for non-critical frames, the request scheduler uses the Doppler velocity of the point cloud for updates:

[0036] P(x i+1 y i+1 , z i+1 )=P(x i y i , z i )v i Δt

[0037] Where, x i y i , zi is the 3D coordinate of the i-th point, v i is the velocity of the i-th point, and Δt is the interval time between frames.

[0038] Further, based on the key frame selected by the request scheduler, the vehicle selectively requests the infrastructure to send the corresponding key points for point cloud registration through the adaptive trigger.

[0039] On the other hand, the application also provides a millimeter wave radar point cloud registration device for vehicle-road cooperative automatic driving, which comprises the following components to realize the millimeter wave radar point cloud registration method for vehicle-road cooperative automatic driving according to any one of the above:

[0040] a camera for acquiring camera data of roadside facilities and multiple vehicles;

[0041] a millimeter wave radar for acquiring radar point cloud data of roadside facilities and multiple vehicles;

[0042] a point cloud registration network component composed of a salient target segmenter, a salient region extractor, and a 3D point cloud registration module; wherein the salient target segmenter is used to segment salient targets from images of roadside facilities and multiple vehicles to provide static semantics for the radar branch; the salient region extractor is used to align the camera and the millimeter wave radar branches in time and space dimensions based on the static semantics to obtain the 3D point cloud region corresponding to the salient target; and the 3D point cloud registration module is used to select key points according to the 3D point cloud region corresponding to the salient target and iteratively process the selected key points to obtain a transformation matrix and output a registration result;

[0043] a multi-vehicle adaptive scheduling component including a request scheduler and an adaptive trigger, used to realize real-time point cloud registration of multiple vehicles; wherein the request scheduler selects key frames by weighing the number of requests, registration delay, success rate, and the degree of change in the vehicle perspective; for non-key frames, the request scheduler uses the state information of the target in the previous frame to update the point cloud data of the next frame; and the adaptive trigger simultaneously considers the vehicle factors and the degree of change in the infrastructure perspective to adaptively transmit the corresponding key points.

[0044] Compared with the prior art, the application has the following beneficial effects:

[0045] The millimeter wave radar point cloud registration method and device for vehicle-road cooperative automatic driving provided by the application first utilize the complementary advantages of cameras and millimeter wave radars to realize accurate registration of vehicle-infrastructure sparse and disordered 3D radar point clouds, and solve the problems of difficult extraction of semantic features of sparse and disordered radar point clouds and difficult correspondence of points.

[0046] The adaptive scheduling strategy is designed, and the triggering factors of the vehicle and the change degree of the point cloud in the field of view of the infrastructure are considered to adaptively trigger data request, so as to ensure real-time registration between multiple vehicles and infrastructure.

[0047] To sum up, the camera data and radar point cloud data of the vehicle and the infrastructure are taken as inputs, and the point cloud registration is finally completed, while the real-time and accuracy of the millimeter wave radar point cloud registration are met, the vehicle perception range and perception accuracy are improved in heavy fog weather, and the performance of the millimeter wave radar in target detection, path planning and other applications is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0049] Figure 1 The system architecture diagram of the millimeter wave radar point cloud registration method for vehicle-road cooperative automatic driving provided by the embodiments of the present application. DETAILED DESCRIPTION

[0050] In order to better understand the technical solutions, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described examples are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0051] The millimeter wave radar point cloud registration method and device for vehicle-road cooperative automatic driving proposed in the present application, the system architecture is as shown in Figure 1 The camera, millimeter wave radar, point cloud registration network component and multi-vehicle adaptive scheduling component are included. Among them:

[0052] The point cloud registration network component solves the problem of extracting target semantics from millimeter wave radar point clouds by utilizing the complementary advantages of cameras and radars. Specifically, the invention designs a salient target segmenter to segment salient targets (such as traffic signs) from images to provide semantics for the radar branch. Then a salient region extractor is used to map the corresponding 3D point cloud region. In addition, the invention takes into account the sparsity and disorder of point clouds, which makes it difficult to form an accurate one-to-one correspondence, resulting in a large registration error. To solve this problem, the invention designs a 3D point cloud registration module, which includes two strategies: (i) key point extraction strategy: using the geometric features of salient targets to wrap them with spheres to obtain corresponding key points; (ii) early termination strategy: by clustering and similarity calculation on salient targets, the correct direction of the transformation matrix is determined, and if the direction calculated by the matching key points is consistent with the correct direction of the transformation matrix, the iteration process can be terminated in advance. Finally, the invention uses the RANSAC algorithm to iterate on the selected key points and outputs the transformation matrix to achieve accurate point cloud registration.

[0053] The multi-vehicle adaptive scheduling component ensures real-time point cloud registration of multiple vehicles by designing two key modules: request scheduler and adaptive trigger. (i) The request scheduler is designed to reasonably select key frames by weighing various factors: request times, registration delays, success rates, and changes in vehicle perspectives; for non-key frames, the scheduler uses the state information of the target in the previous frame to update the point cloud data in the next frame; (ii) The adaptive trigger considers both vehicle factors and the degree of change in infrastructure perspectives to adaptively transmit corresponding key points to reduce bandwidth consumption and multi-vehicle registration running time.

[0054] The functions and methods of the point cloud registration network component are described as follows.

[0055] The point cloud registration network component mainly consists of three modules: salient target segmenter, salient region extractor, and 3D point cloud registration.

[0056] 1) Salient target segmenter: The invention uses a 2D target detector Yolo-v5 to detect and segment targets from vehicles and infrastructure. Some dynamic targets in the scene change continuously between consecutive frames, which is meaningless for point cloud registration. Therefore, during model training, dynamic targets such as vehicles and pedestrians are automatically filtered to reduce computational cost. At the same time, only a common static target (traffic sign) is further processed to ensure the accuracy of registration under highly dynamic traffic conditions.

[0057] 2) salient region extractor: based on the extracted static semantic targets, a salient region extractor is designed to obtain 3D regions of the radar point cloud. First, the camera and millimeter wave radar branches need to be aligned in time and space dimensions. In time, the frames of the camera branch are associated with their nearest radar frames. In space, the camera frame and radar frame measurements are transformed to the same coordinate system. Specifically, the 3D point cloud coordinates are converted to the pixel coordinate system under the camera perspective, as follows:

[0058]

[0059] The above formula can be rewritten as:

[0060] P uv = KTP w

[0061] where P uv represents the pixel homogeneous coordinates (u, v, 1), P w refers to the point coordinates (X w , Y w , Z w , 1) obtained with the millimeter wave acquisition device as the coordinate origin, K is the camera intrinsic matrix (f x and f y are the focal lengths on the X and Y axes; u0 and v0 are the principal point coordinates), and T is the camera extrinsic matrix. Since the relative positions of the two sensors are fixed, they can be calculated offline. Through the above formula, the 2D plane of the camera branch is mapped to the two-dimensional plane of the real-world radar point cloud to obtain the salient region of the radar branch, and then the corresponding semantics is assigned to the 2D point cloud region, and is mapped back to the 3D representation, thereby realizing the 3D salient region extraction of the radar point cloud.

[0062] 3) 3D point cloud registration module: According to the obtained salient region, the key points are selected as the input of the RANSAC algorithm to quickly find the transformation matrix. However, the traditional RANSAC algorithm consumes most of the time to calculate the inliers, which is an O(n) operation. Therefore, the present application designs two key strategies for the traditional RANSAC to achieve accurate point cloud registration. (i) Key point extraction strategy: the present application designs a sphere model to wrap the salient target point cloud, and selects the center point and two radius points of the sphere as the key points. Specifically, the salient target point cloud is converted into a binary image, and then the convex hull is fitted with the geometric features of the target. We find that the 3D space shape of the target point cloud is similar to a sphere, on this basis, a sphere model is used to wrap the target point cloud, and the point cloud is clustered in the envelope using the features and occupancy information of the point cloud, a center point and two radius points are selected, this method can effectively solve the non-correspondence of key points, so as to improve the registration performance and reduce the search space of matching key points. (ii) Early termination strategy: the static semantic targets extracted from the infrastructure and the vehicle are clustered according to their pixel coordinates x, and these targets are divided into left and right directions; by comparing the number of static targets distributed on the left and right sides and the similarity of semantic targets, the direction between the vehicle and the infrastructure is determined. For example, if the number of targets on the left side between the vehicle and the infrastructure is the same, and there are salient targets with high similarity, then the direction is considered to be the same, otherwise it is opposite. On this basis, the problem of difficult determination of the direction of the transformation matrix can be solved. Once a correct transformation matrix is obtained, the RANSAC algorithm is used to estimate the number of inliers and determine whether to end the registration.

[0063] The functions and methods of the multi-vehicle adaptive scheduling component are specifically introduced as follows.

[0064] 1) Request scheduler:

[0065] To balance the real-time performance and the registration accuracy, a request scheduler needs to weigh several key factors: success rate A, registration latency L, request number N, and the degree of view change ε. For example, according to the statistical results of the dataset, the number of points of a new target increases by an average of 4, and the threshold is set to 4 to determine the degree of view change. When ε > 4, the request scheduler considers both the number of requests within 1s and the success rate of the last frame to determine whether to perform registration. If the success rate is lower than the set lower limit (in order to better serve the vehicle, the lower limit of the success rate is set to 90% in the preferred embodiment), and the remaining time can meet the execution of multiple requests, registration will be performed. Through actual measurement of the roadside equipment, the average time of one registration is 85.77ms. Based on this, the upper limit of the request number can be calculated. Once the registration speed of 20fps is met, the request scheduler will strive to maximize the request number to further improve the overall success rate, and the optimization problem can be expressed as follows:

[0066]

[0067] Wherein, A i is the success rate of the i-th request.

[0068] For non-key frames, the request scheduler uses a lightweight update strategy to obtain the point cloud data of the next frame. The existing visual odometry method has high calculation cost and is not suitable for real-time point cloud registration, and the perception range of the camera in foggy weather will be severely limited, which makes some medium and long distance targets unable to be detected. Therefore, the request scheduler uses the Doppler velocity of the point cloud to update the non-key frame:

[0069] P(x i+1 , y i+1 , z i+1 ) = P(x i , y i , z i ) v i Δt

[0070] Wherein, x i , y i , z i are the 3D coordinates of the i-th point, v i is the velocity of the i-th point, and Δt is the interval time between frames.

[0071] 2) Adaptive trigger: based on the key frame selected by the request scheduler, the vehicle selectively requests the infrastructure to send the corresponding key points for point cloud registration. Point cloud registration is a collaboration between the vehicle and the infrastructure. Therefore, it is also necessary to consider whether some factors in the infrastructure will affect the safe driving of the vehicle, for example, some new targets will appear in the field of view of the infrastructure, but the vehicle does not detect the changed target. In this case, the vehicle may have an accident because it is not timely registered. Therefore, the present application designs an adaptive trigger to consider the triggering factors of the vehicle (success rate A, registration delay L, request number N) and the degree of change of the vehicle field of view ε to adaptively trigger data requests, to ensure real-time registration between multiple vehicles and infrastructure. In this embodiment, according to the statistical results of the actually collected data, that is, the number of changes in point cloud when a target appears or disappears, the triggering threshold of the infrastructure is set to 5. This data can be adaptively changed according to different equipment.

[0072] The application provides a millimeter wave radar point cloud registration method and device for vehicle-road cooperative automatic driving, which utilizes the complementary advantages between a camera and a millimeter wave radar, and expands the perception ability of a vehicle (especially in harsh conditions).

[0073] The experimental results show that, from the application level, the effective perception distance of the outdoor scene is increased by 48%-89%, and the target point cloud density is increased by 65%. For the registration result, the average registration error (RTE) of the outdoor scene is 0.32 m, and the average success rate is 92.75%; in addition, the average bandwidth consumption of each vehicle is 0.45 Mbps for data transmission, and 27 / 114 vehicles can be supported for real-time registration in a 4G / 5G network. The registration result of the application can be used in target detection, path planning and other applications, and effectively solves the problems of limited perception range and low perception accuracy of the vehicle in harsh conditions.

[0074] The above only describes the preferred embodiments and principles of the application in detail, and is not used to limit the protection scope of the application. For ordinary skilled persons in the art, according to the idea provided by the application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application should be considered as the protection scope of the application.

Claims

1. A method for millimeter wave radar point cloud registration for vehicle-road cooperative automatic driving, characterized in that, The method comprises the following steps: S1, significant target segmentation: camera data of roadside facilities and multiple vehicles are acquired respectively, and a significant target segmenter is used to segment significant targets from images of the roadside facilities and the multiple vehicles respectively to provide static semantics for a radar branch; S2, significant region extraction: based on the static semantics and radar point cloud data of the roadside facilities and the multiple vehicles acquired respectively by using a millimeter wave radar, a significant region extractor is used to align the camera and the millimeter wave radar branches in time and space dimensions to obtain 3D point cloud regions corresponding to the significant targets; S3, 3D point cloud registration: according to the 3D point cloud regions corresponding to the significant targets, key points are selected and iteration is performed on the selected key points to obtain a transformation matrix, and a registration result is output; The method realizes real-time point cloud registration of multiple vehicles through a multi-vehicle adaptive scheduling component, and the multi-vehicle adaptive scheduling component comprises a request scheduler and an adaptive trigger; The request scheduler selects a key frame by balancing the number of requests, registration delay, success rate and the degree of change of the vehicle perspective; for a non-key frame, the request scheduler uses the state information of the target in the previous frame to update the point cloud data of the next frame; The adaptive trigger simultaneously considers the vehicle factor and the degree of change of the infrastructure perspective to adaptively transmit corresponding key points; In step S3, the selected key points are iterated by using an improved RANSAC algorithm, and the improved RANSAC algorithm comprises two strategies: (i) key point extraction strategy The corresponding key points are obtained by wrapping the significant target with a sphere according to the geometric features of the significant target; (ii) early termination strategy The correct direction of the transformation matrix is determined by clustering and similarity calculation of the significant target, and if the direction calculated by the matching key points is consistent with the correct direction of the transformation matrix, the iteration process is terminated in advance.

2. The method of claim 1, wherein, In step S1, the significant target segmenter uses a 2D target detector to detect and segment targets from vehicles and infrastructure.

3. The method of claim 1, wherein, In step S2, the significant region extractor is used to obtain the 3D point cloud region corresponding to the significant target, in time, the frame of the camera branch is associated with its nearest radar frame; in space, the camera frame and the radar frame measurement values are converted to the same coordinate system; The specific method is as follows: The 3D point cloud coordinates are converted to pixel coordinates in the camera perspective, and the formula is as follows: The above formula is rewritten as: P uv = KTP w where P uv represents the pixel homogeneous coordinates (u, v, 1), P w w w w x y is the camera intrinsic matrix, f x and f y are the focal lengths in the X and Y axes; u0and v0are the principal point coordinates, and T is the camera extrinsic matrix;​ The 2D plane of the camera branch is mapped to the 2D plane of the real world radar point cloud by the above formula to obtain the significant region of the radar branch, and then the corresponding semantics is assigned to the 2D point cloud region, and then it is mapped back to the 3D representation, thereby realizing the 3D significant region extraction of the radar point cloud.

4. The method of claim 1, wherein, The specific method of the key point extraction strategy is to select the center point and two radius points of the sphere as the key points, convert the point cloud of the significant target to a binary image, and then use the convex hull to fit the geometric features of the target.

5. The method of claim 1, wherein, The specific method of the early termination strategy is to cluster the pixel coordinates x of the static semantic targets extracted from the infrastructure and the vehicle, and divide these targets into left and right directions; the number of static targets distributed on the left and right sides and the similarity of the semantic targets are compared to determine the direction between the vehicle and the infrastructure.

6. The method of claim 1, wherein, For key frames, when the number of new targets increases by k on average, the request scheduler sets the threshold to k to determine the degree of change in view, when the degree of change in the vehicle's field of view is greater than k, the request scheduler considers the number of requests within 1s and the success rate of the previous frame to decide whether to perform registration; If the success rate is below the set lower limit, and the remaining time can meet the execution of multiple requests, perform registration; the request scheduler maximizes the number of requests to improve the overall success rate of the optimization problem as follows: where A i is the success rate of the ith request, N is the number of requests, and L is the registration latency.

7. The method of claim 1, wherein, For non-key frames, the request scheduler updates the Doppler velocity of the point cloud: P(x i+1 , y i+1 , z i+1 ) = P(x i , y i , z i ) v i Δt, where x i , y i , z i are 3D coordinates of the i-th point, v i is the velocity of the i-th point, and Δt is the interval time between frames.

8. The method of claim 1, wherein, Based on the key frames selected by the request scheduler, the vehicle selectively requests the infrastructure to send corresponding key points for point cloud registration through the adaptive trigger.

9. A millimeter wave radar point cloud registration device for vehicle-road cooperative automatic driving, characterized by, The method comprises the following modules to realize the millimeter wave radar point cloud registration method for vehicle infrastructure cooperation automatic driving according to any one of claims 1-8: a camera for acquiring camera data of roadside facilities and multiple vehicles; a millimeter wave radar for acquiring radar point cloud data of roadside facilities and multiple vehicles; a point cloud registration network component composed of a salient target segmenter, a salient region extractor and a 3D point cloud registration module; wherein the salient target segmenter is used to segment salient targets from the images of roadside facilities and multiple vehicles to provide static semantics for the radar branch; the salient region extractor aligns the camera and millimeter wave radar branches in time and space dimensions based on the static semantics to obtain the corresponding 3D point cloud region of the salient target; the 3D point cloud registration module selects key points and iterates the selected key points according to the corresponding 3D point cloud region of the salient target to obtain a transformation matrix and outputs a registration result; a multi-vehicle adaptive scheduling component including a request scheduler and an adaptive trigger for realizing real-time point cloud registration of multiple vehicles; wherein the request scheduler selects key frames by weighing the number of requests, registration delay, success rate and the degree of change in the vehicle's view; for non-key frames, the request scheduler uses the state information of the targets in the previous frame to update the point cloud data of the next frame; the adaptive trigger considers both the vehicle factors and the degree of change in the infrastructure view to adaptively transmit corresponding key points.

Images