A multi-vehicle cooperative 3D target detection method and system based on radar data

Abstract

The application discloses a kind of multi-vehicle coordination 3D target detection method and system based on radar data, method includes: obtaining the local environment information of main vehicle and each coordination vehicle, and obtains radar data by adding noise;The radar data of coordination vehicle is shared to main vehicle, and radar data is converted to the coordinate system of main vehicle;Radar data is converted into pseudo image, and the feature of coordination vehicle and main vehicle is extracted;The feature of extracted coordination vehicle removes time and space noise;Make the feature of coordination vehicle and main vehicle interact fusion;Generate the three-dimensional target detection frame of surrounding environment;Through calculating loss function, the target detection model is cyclically optimized, and the 3D target detection of multi-vehicle coordination is completed.The application embodiment can improve the precision and robustness of target detection, so that automatic driving system can identify target in real time, accurately in dynamic, complex road conditions, enhance the safety of automatic driving system, and can be widely applied in intelligent traffic technology field.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and in particular to a multi-vehicle cooperative 3D target detection method and system based on radar data. Background Technology

[0002] As autonomous driving technology matures, a vehicle's environmental perception capabilities have become a key factor in ensuring safety and improving driving efficiency. Radar, especially millimeter-wave radar, is widely used in autonomous driving systems due to its stable all-weather, all-time operation. Radar can operate stably in adverse weather conditions (such as fog, rain, and snow) and has better penetration capabilities than lidar and cameras, making it particularly effective in complex road and environmental environments.

[0003] Despite their strong environmental perception capabilities, single-vehicle radar systems still have many limitations. Due to limitations in their field of view and detection range, single-vehicle radar often struggles to effectively detect objects far away or obscured targets. Furthermore, the perception results of single-vehicle radar are limited to the perspective of the single vehicle itself, failing to acquire environmental information about other vehicles. This makes it difficult for single-vehicle radar to accurately identify and locate targets in complex traffic environments, especially in dynamic traffic flow. More seriously, the accuracy of single-vehicle radar is often dependent on sensor quality and environmental conditions. In adverse weather conditions or complex environments with many obstacles, false detections and missed detections are common, severely impacting the safety and reliability of autonomous driving systems.

[0004] To overcome the limitations of single-vehicle radar systems, multi-vehicle cooperative systems have emerged. In a multi-vehicle cooperative environment, through real-time data sharing among vehicles, each vehicle can not only perceive its own surroundings but also fuse radar data from other vehicles, thereby obtaining broader and more comprehensive environmental perception information. This multi-vehicle cooperative approach can effectively improve target detection accuracy, reduce blind spots, and enhance system robustness. However, although multi-vehicle cooperative systems offer a new approach to addressing the limitations of single-vehicle radar, the processing of multi-vehicle cooperative radar data faces a series of challenges, particularly positional errors and time delays, as detailed below:

[0005] 1. Positional Error Issues: In multi-vehicle cooperative systems, radar data sharing relies on the fusion of radar data from different vehicles via onboard communication systems. However, due to differences in the position, speed, and orientation of each vehicle, and the frequent changes in their relative positions, accurately aligning the radar data of different vehicles to ensure spatial consistency has become a core technical challenge in multi-vehicle cooperative systems. Especially when vehicles are traveling at high speeds or making sharp turns, the relative positions of vehicles change rapidly. Failure to accurately synchronize the radar data of different vehicles may introduce significant positional errors, directly impacting the accuracy and robustness of target detection.

[0006] 2. Time Delay Issues: Time delay is another critical issue in multi-vehicle cooperative systems. In such environments, radar data between vehicles typically needs to be shared in real-time via onboard communication systems. This data transmission is affected by various factors, including network bandwidth, signal latency, and communication protocols. Especially in high-speed or complex traffic environments, data transmission delays are often unavoidable. This can lead to inaccurate data synchronization between different vehicles, thus affecting the entire system's target detection and judgment. For example, when a vehicle's radar detects an obstacle ahead, due to communication delays, other vehicles may need hundreds of milliseconds or even longer to receive this information. In dynamic traffic environments, especially in rapidly changing scenarios, this delay can cause other vehicles to experience lag in their environmental perception data, thereby affecting timely target location and avoidance decisions. Summary of the Invention

[0007] The main objective of this invention is to propose a multi-vehicle cooperative 3D target detection method and system based on radar data, which can improve the accuracy and robustness of target detection, enabling autonomous driving systems to identify targets in real time and accurately in dynamic and complex road conditions, thereby enhancing the safety of autonomous driving systems.

[0008] To achieve the above objectives, one aspect of this invention proposes a multi-vehicle cooperative 3D target detection method based on radar data, comprising the following steps:

[0009] Local environmental information of the main vehicle and each cooperating vehicle is acquired by radar sensors, and temporal and spatial noise is added to the acquired local environmental information to obtain radar data.

[0010] The radar data of the cooperating vehicle is shared to the main vehicle in real time through the vehicle communication system, and the radar data is converted to the coordinate system of the main vehicle through GPS technology;

[0011] The radar data of each vehicle is converted into pseudo-images using a PointPillar encoder, and features of the cooperating vehicle and the master vehicle are extracted using ResNet18.

[0012] The Kalman filter algorithm is used to remove temporal and spatial noise from the extracted features of the cooperative vehicles.

[0013] Through an attention mechanism, the features of the cooperating vehicle and the features of the master vehicle interact and merge to obtain fused features;

[0014] The fused features are input into the PointPillar decoder to generate a 3D target detection box of the surrounding environment, determining the spatial location and shape of the target;

[0015] By calculating the loss function, the obtained 3D target detection box is compared with the ground truth box, the gradient is calculated, and the gradient is back-propagated to update the network parameters, and the target detection model is iteratively optimized.

[0016] The optimized target detection model is used to perform multi-vehicle collaborative 3D target detection.

[0017] In some embodiments, sharing the radar data of the cooperating vehicle to the host vehicle in real time via an in-vehicle communication system, and converting the radar data to the coordinate system of the host vehicle using GPS technology, includes the following steps:

[0018] The local coordinate system is mapped to the world coordinate system, and the position and attitude information are obtained from the vehicle's GPS information. The attitude information includes pitch angle α, yaw angle β and roll angle γ.

[0019] Construct a rotation matrix R and a translation vector T of the vehicle in the world coordinate system to describe the rotation and translation of the local coordinate system relative to the world coordinate system;

[0020] The 3D point cloud features in the world coordinate system are calculated using the linear transformation formula. The specific calculation formula is: P world =R·P local +T, where P world A 3D point cloud feature in the world coordinate system; P local The three-dimensional point cloud features are in a local coordinate system;

[0021] The world coordinate system is normalized and transformed to the master vehicle coordinate system. The relative translation and the relative rotation matrix of the two vehicles are calculated using the position information of the master vehicle and the slave vehicle.

[0022] The point cloud data in the world coordinate system is projected to the vehicle coordinate system using the following formula: Among them, P main 3D point cloud features of the main vehicle coordinate system; R relative T1 is the relative rotation matrix between the two vehicles; T2 is the translation vector of the main vehicle in the world coordinate system.

[0023] In some embodiments, the step of converting the radar data of each vehicle into pseudo-images using a PointPillar encoder and extracting features of the cooperating vehicle and the master vehicle using ResNet18 includes the following steps:

[0024] The point cloud data is divided into regular cylindrical shapes according to the top view coordinate system;

[0025] The initial features of each point include the point's three-dimensional position and reflection intensity;

[0026] Calculate the relative position of each point with respect to the center of the column and its relative position with respect to the grid origin, forming a point feature vector; the point feature vector f point The calculation formula is: f point =[x,y,z,I,x rel ,y rel ,z rel ], where x, y, z represent the three-dimensional position of the point; I represents the reflection intensity; x rel ,y rel ,z rel It is the relative coordinate of a point with respect to the center of its column;

[0027] A point cloud feature extraction network is used to process the features of each point, and the features of points belonging to the same column are aggregated.

[0028] By using max pooling or average pooling, columnar features are generated, resulting in a fixed-dimensional feature vector f for each column. pillar .

[0029] In some embodiments, the extraction of features from the cooperative vehicles, using a Kalman filter algorithm to remove temporal and spatial noise, includes the following steps:

[0030] In the prediction of the target state at each time k, the prediction is first performed based on the system's state transition model and control input. The expression for the prediction process is: in, Let A be the predicted state at time k, and let A be the state transition matrix. It represents the updated state from the previous time step, B is the control input matrix, and uk-1 is the control input;

[0031] The prediction and update of the system's covariance is expressed as: P k|k-1 =AP k-1|k-1 A T +Q, where P k|k-1 Let Q represent the prediction covariance at time k, and let Q be the process noise covariance, representing the variation of the unmodeled part in the model.

[0032] At each time k, a new observation z is obtained. kThen, Kalman filtering fuses the predicted state and actual observations by calculating the Kalman gain, thereby obtaining an updated state estimate. The formula for calculating the Kalman gain is: K k =P k|k-1 H T HP k|k-1 H T +R) -1 , where K k Here, H is the Kalman gain, H is the observation matrix, and R is the observation noise covariance.

[0033] The updated target state estimate is expressed as: in, For the updated target state, z k The actual observed value at the current moment;

[0034] The covariance matrix is ​​updated based on the Kalman gain. The expression for the update process is: P k|k =(IK k H)P k|k-1 , where P k|k Let I be the updated covariance matrix, and let I be the identity matrix.

[0035] In some embodiments, the step of using an attention mechanism to interactively fuse the features of the cooperating vehicle with the features of the master vehicle to obtain fused features includes the following steps:

[0036] Input data X∈R n×d The linear transformation maps the matrix to a query matrix Q, a key matrix K, and a value matrix V:

[0037] Q = XW Q K = XW K V = XW V ;

[0038] in, The linear transformation matrices d represent the query, key, and value, respectively. k and d v Dimensions for queries, keys, and values;

[0039] The attention weights are calculated using softmax normalization. weights The calculation formula is:

[0040] The attention output after weighted summation is obtained, and the features of the cooperating vehicle are interactively fused with the features of the master vehicle to obtain fused features.

[0041] In some embodiments, the step of inputting fused features into the PointPillar decoder to generate a 3D target detection box of the surrounding environment and determine the spatial location and shape of the target includes the following steps:

[0042] The PointPillar decoder is used to convert the fused feature information into a 3D target detection box, which represents the position, size, and shape of the target object in the environment.

[0043] The three-dimensional coordinates and orientation information of the target are generated based on the three-dimensional target detection box;

[0044] When processing radar data, the PointPillar decoder maps the features obtained from multi-vehicle collaboration onto three-dimensional space to generate the target's three-dimensional bounding box and determine the target's spatial location and shape.

[0045] In some embodiments, the loss function is used to calculate the difference between the predicted 3D object detection box and the ground truth box, and to update the network parameters through a backpropagation mechanism to optimize the object detection accuracy.

[0046] The formula for the loss function L is:

[0047] L=∑(λ 1 *L 1 (x i ,x′ i )+λ 2 *L 2 (y i ,y′ i )+λ 3 *L 3 (z i ,z′ i ))

[0048] Among them, L 1 ,L 2 ,L 3 These represent the error calculations for the target's position, size, and orientation, respectively; x′ i ,y′ i ,z′ i For the location information of the predicted bounding box; x i ,y i ,z i The position information of the true bounding box; λ 1 ,λ 2 ,λ 3 These are weighting coefficients used to balance the losses of different categories.

[0049] Another aspect of this invention provides a multi-vehicle cooperative 3D target detection system based on radar data, comprising:

[0050] The first module is used to acquire local environmental information of the main vehicle and each cooperating vehicle through radar sensors, and add time and space noise to the acquired local environmental information to obtain radar data.

[0051] The second module is used to share the radar data of the cooperating vehicle to the main vehicle in real time through the vehicle communication system, and to convert the radar data to the coordinate system of the main vehicle through GPS technology.

[0052] The third module is used to convert the radar data of each vehicle into pseudo-images using the PointPillar encoder, and extract the features of the cooperating vehicle and the master vehicle using ResNet18.

[0053] The fourth module is used to remove temporal and spatial noise from the extracted features of the cooperative vehicles using the Kalman filter algorithm.

[0054] The fifth module is used to interact and fuse the features of the cooperating vehicle with the features of the master vehicle through an attention mechanism to obtain fused features;

[0055] The sixth module is used to input the fused features into the PointPillar decoder to generate a 3D target detection box of the surrounding environment and determine the spatial position and shape of the target.

[0056] The seventh module is used to compare the obtained 3D object detection box with the real box by calculating the loss function, calculate the gradient, and backfeed the gradient to update the network parameters, and iteratively optimize the object detection model.

[0057] The eighth module is used to complete multi-vehicle collaborative 3D target detection based on the optimized final target detection model.

[0058] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.

[0059] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0060] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0061] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a multi-vehicle cooperative 3D target detection method and system based on radar data. This scheme acquires local environmental information of the main vehicle and each cooperative vehicle through radar sensors, and adds temporal and spatial noise to the acquired local environmental information to obtain radar data; the radar data of the cooperative vehicles is shared to the main vehicle in real time through the vehicle communication system, and the radar data is converted to the coordinate system of the main vehicle through GPS technology; the radar data of each vehicle is converted into pseudo-images using a PointPillar encoder, and the features of the cooperative vehicles and the main vehicle are extracted using ResNet18; the extracted features of the cooperative vehicles are processed by a Kalman filter algorithm to remove temporal and spatial noise; through an attention mechanism, the features of the cooperative vehicles and the features of the main vehicle are interactively fused to obtain fused features; the fused features are input into a PointPillar decoder to generate a 3D target detection box of the surrounding environment to determine the spatial position and shape of the target; by calculating a loss function, the obtained 3D target detection box is compared with the ground truth box, the gradient is calculated, and the gradient is back propagated to update the network parameters, and the target detection model is iteratively optimized; the final target detection model obtained by optimization is used to complete the multi-vehicle cooperative 3D target detection. The embodiments of the present invention can improve the accuracy and robustness of target detection, enabling the autonomous driving system to identify targets in real time and accurately in dynamic and complex road conditions, thereby enhancing the safety of the autonomous driving system. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0063] Figure 2 This is a flowchart of the overall steps provided in the embodiments of the present invention;

[0064] Figure 3 This is an example diagram of a multi-vehicle cooperative 3D target detection architecture based on radar data provided in an embodiment of the present invention;

[0065] Figure 4 This is a flowchart illustrating the specific implementation process provided in the embodiments of the present invention;

[0066] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0068] It is understood that the terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0069] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0071] The multi-vehicle cooperative 3D target detection method and system based on radar data provided in this invention relates to the field of intelligent transportation technology. The multi-vehicle cooperative 3D target detection method based on radar data provided in this invention can be applied to a terminal, a server, or software running on a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the multi-vehicle cooperative 3D target detection method based on radar data, but is not limited to the above forms.

[0072] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0073] like Figure 1 The diagram shown is a schematic representation of an implementation environment provided by an embodiment of the present invention. (Refer to...) Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected via a network, either wirelessly or via a wired connection, to complete data transmission and exchange.

[0074] Server 101 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0075] Additionally, server 101 can also be a node server in a blockchain network. Blockchain is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms.

[0076] Terminal 102 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc. It can also be a vehicle-mounted terminal of the various device types described above, but is not limited to these. Terminal 102 and server 101 can be directly or indirectly connected via wired or wireless communication, and this embodiment of the invention does not impose any limitations.

[0077] Exemplary based on Figure 1 The implementation environment shown in this embodiment of the invention provides a multi-vehicle cooperative 3D target detection method based on radar data. The following description uses the application of this multi-vehicle cooperative 3D target detection method based on radar data in server 101 as an example. It can be understood that this method can also be applied to terminal 102.

[0078] Reference Figure 2 , Figure 2 The flowchart illustrates a multi-vehicle cooperative 3D target detection method based on radar data applied to a server, as provided in this embodiment of the invention. The execution subject of this method can be any of the aforementioned computer devices (including servers or terminals). Figure 3 This is an example of a multi-vehicle cooperative 3D target detection architecture based on radar data, see reference. Figure 2 and Figure 3 The method may include the following steps:

[0079] Step 1: Multiple vehicles acquire local environmental information through their respective radar sensors, and add temporal and spatial noise to the acquired data;

[0080] Step 2: Share the radar data of the cooperating vehicles to the master vehicle in real time through the vehicle communication system, and convert the radar data to the master vehicle coordinate system through GPS technology;

[0081] Step 3: Convert the radar data of each vehicle into pseudo-images using the PointPillar encoder, and extract the feature information of the cooperating vehicle and the master vehicle using ResNet18;

[0082] Step 4: Use the Kalman filter algorithm to remove temporal and spatial noise from the extracted cooperative vehicle features;

[0083] Step 5: Through the attention mechanism, the features of the cooperating vehicle and the features of the main vehicle are interacted and fused to obtain the fused features;

[0084] Step 6: Input the calculated fusion features into the PointPillar decoder to generate a 3D target detection box of the surrounding environment and determine the spatial position and shape of the target;

[0085] Step 7: By calculating the loss function, compare the obtained 3D object detection boxes with the ground truth boxes, calculate the gradient, and backpropagate the gradient to update the network parameters and optimize the object detection model;

[0086] Step 8: Repeat steps 4-7 until the model converges, and save the model parameters.

[0087] In some embodiments, step 1 involves multiple vehicles acquiring local environmental information via their respective radar sensors. To enhance system robustness, this embodiment incorporates temporal and spatial noise into the data collected by each vehicle. This noise injection effectively simulates the measurement uncertainties faced by sensors and the relative positional information deviations between vehicles during actual driving. In this way, the system can learn more robust features during training and better adapt to various changes in complex environments in subsequent target detection tasks.

[0088] In some embodiments, step 2 involves the following: In this invention, radar data between vehicles is shared in real time via an onboard communication system. Since each vehicle's radar system and sensor have different viewing angles, the data needs to be processed using GPS technology to convert the radar data of each vehicle to the coordinate system of the master vehicle. To ensure spatial consistency of radar data between vehicles, this embodiment employs GPS-based timestamp synchronization technology. This not only effectively synchronizes radar data from different vehicles but also ensures that the multi-vehicle collaborative system can accurately align the data of different vehicles in high-speed driving and dynamic environments, thereby reducing errors caused by time delays or spatial deviations. In step 2, in a multi-vehicle collaborative environment, vehicles need to share their respective radar data. To ensure the timeliness and accurate alignment of the data, this invention uses an onboard communication system (such as V2X technology) for data transmission and utilizes GPS technology to synchronize the radar data of different vehicles, ensuring that the data of all vehicles can be converted to the coordinate system of the master vehicle, avoiding errors caused by data inconsistency or time delays.

[0089] In some embodiments, step 3: During feature extraction, the present invention uses a PointPillar encoder as the core feature extraction module, which processes 3D point cloud data (or image data). Specifically, the PointPillar encoder maps the input point cloud data (or 2D image) into a regularized grid structure, grouping the data using columns, enabling subsequent network layers to process it effectively. This encoding method not only improves computational efficiency but also extracts meaningful features uniformly in both dense and sparse regions. To further improve the accuracy and reliability of target detection, this embodiment uses a ResNet18 network to extract feature information of the primary vehicle and cooperating vehicles. ResNet18, as a mature feature extraction network, can effectively extract target features at different scales and avoid missed detections due to insufficient feature extraction. In step 3, the PointPillar encoder converts radar data from different vehicles into pseudo-images, and the deep learning model ResNet18 is used to extract target feature information. The PointPillar encoder can preserve the spatial structure of radar data and transform it into a format that can be effectively processed by neural networks. ResNet18 extracts key features of the target through convolution operations, improving detection accuracy and feature extraction capabilities.

[0090] In some embodiments, step 4: For the cooperative features acquired from multiple vehicles, the present invention employs a Kalman filter algorithm to remove temporal and spatial noise from the features. The Kalman filter algorithm can effectively handle uncertainties in radar data. By dynamically filtering the radar data of each vehicle, it removes errors caused by sensor noise, changes in vehicle positions, and other factors, thereby significantly improving the stability and accuracy of the feature data. In the target state prediction at each time k, prediction is first performed based on the system's state transition model and control input. The specific formula is as follows:

[0091]

[0092] in, Let A be the predicted state at time k, and let A be the state transition matrix. It represents the updated state from the previous time step, B is the control input matrix, and uk-1 is the control input.

[0093] Next, the system's covariance will also be predicted and updated, using the following formula:

[0094] P k|k-1 =AP k-1|k-1 A T +Q

[0095] Among them, P k|k-1 Let represent the prediction covariance at time k, and Q be the process noise covariance, representing the variation of the unmodeled portion of the model. New observation z is obtained at each time k. k Then, Kalman filtering fuses the predicted state and actual observations by calculating the Kalman gain, thus obtaining an updated state estimate. The formula for calculating the Kalman gain is:

[0096]

[0097] Among them, K k Let H be the Kalman gain, H be the observation matrix, and R be the observation noise covariance, reflecting the uncertainty of the observations. Then, the updated target state estimate can be expressed as:

[0098]

[0099] in, For the updated target state, z k This represents the actual observed value at the current moment. Finally, the covariance matrix is ​​also updated based on the Kalman gain to reflect the uncertainty of the current estimate. The update formula is:

[0100] P k|k =(IK k H)P k|k-1

[0101] Among them, P k|k Let I be the updated covariance matrix, and let I be the identity matrix.

[0102] Through the Kalman filtering process described above, the system can accurately estimate the target's state by fusing data from multiple sensors and effectively cope with environmental noise and uncertainties, thereby improving the accuracy of target detection and localization. Furthermore, Kalman filtering can provide more precise target localization information in multi-vehicle cooperative environments, ensuring the system can perform accurate target detection in complex environments.

[0103] In step 4, since there may be changes in position and speed between vehicles during multi-vehicle cooperation, this invention uses the Kalman filter algorithm to remove noise from the features of the cooperating vehicles. Kalman filtering can effectively estimate the system state and remove noise introduced by factors such as relative motion between vehicles and sensor errors, thereby improving the accuracy of feature data and ensuring the stability of the target detection model in complex environments.

[0104] In some embodiments, step 5: To further enhance the feature information fusion between cooperative vehicles, this invention introduces an attention mechanism. Through the attention mechanism, the features of the master vehicle and cooperative vehicles can be weighted and interacted according to their importance, enabling the system to prioritize the features most critical for target detection. Unlike traditional convolutional neural networks, the attention mechanism can adaptively focus on key regions in an image or point cloud, suppressing interference from irrelevant information, thereby improving processing efficiency and prediction accuracy. This weighted interaction not only improves the effectiveness of cooperative vehicle information but also enhances the system's ability to identify key targets. Specifically, the attention mechanism calculation process is as follows:

[0105] Input data X∈R n×d The linear transformation maps the result to a query, key, and value matrix:

[0106] Q = XW Q K = XW K V = XW V

[0107] in, The linear transformation matrices d represent the query, key, and value, respectively. k and d v Dimensions for queries, keys, and values.

[0108] The formula for calculating scaled dot product attention is:

[0109]

[0110] Then, the attention weights are calculated using softmax normalization:

[0111]

[0112] Finally, we obtain the attention output after weighted summation:

[0113] Attention output =Attention weights ×V

[0114] In this way, the calculation of the attention mechanism can be adaptively adjusted in different feature spaces, enhancing the model's focus on key information through weight allocation. In this invention, this embodiment effectively fuses features from different vehicles through the attention mechanism, improving the system's target detection accuracy in dynamic environments.

[0115] In step 5, to enhance the target detection capability of the multi-vehicle cooperative system, this invention introduces an attention mechanism. This mechanism ensures that the system prioritizes the most critical features for target detection by weighted interaction of features from the master vehicle and cooperative vehicles, reducing unnecessary interference. Through weighted interaction, the system can effectively fuse information from different vehicles, enhancing its ability to identify and classify targets.

[0116] In some embodiments, step 6 involves inputting the fused features into the PointPilar decoder to generate a 3D target detection bounding box of the surrounding environment. This bounding box provides information on the spatial location, size, and shape of the target. When processing radar data, the PointPilar decoder can accurately map the features obtained through multi-vehicle collaboration to 3D space, accurately generating the 3D bounding box of the target. Based on this, the system can further perform target classification and localization, ensuring accurate identification of target objects in dynamic traffic environments.

[0117] In step 6, the fused features are input into the PointPilar decoder to generate accurate 3D target detection boxes. This decoder maps the fused features into 3D space, generating precise location, size, and shape information of the target, providing crucial environmental perception data for autonomous driving systems. This step ensures that targets are not only accurately identified in 2D space but also provides their 3D coordinates and orientation, improving the accuracy and applicability of target detection.

[0118] In some embodiments, step 7: the loss function is used to calculate the difference between the predicted 3D object detection box and the ground truth box, and to update the network parameters through backpropagation to optimize the object detection accuracy. The core of this step is to adjust the network parameters by calculating the errors in position, size, and orientation between the predicted and ground truth boxes, thereby improving the model's object detection accuracy. Specifically, the formula for the loss function is:

[0119] L=∑(λ 1 *L 1 (x i ,x′ i )+λ 2 *L 2 (y i ,y′ i )+λ 3 *L 3 (z i ,z′ i ))

[0120] Among them, L 1 ,L 2 ,L 3These represent the error calculations for the target's position, size, and orientation, respectively. i ,y′ i ,z′ i For the location information of the predicted bounding box, x i ,y i ,z i λ represents the position information of the true bounding box. 1 ,λ 2 ,λ 3 These are weighting coefficients used to balance the losses of different categories. This loss function optimizes object detection accuracy through backpropagation, improving the performance of 3D object detection. By introducing different weighting coefficients, the system can flexibly adjust the emphasis on position, size, and orientation errors according to the needs of different tasks, thereby improving the overall object detection performance.

[0121] In summary, for reference Figure 4 The specific implementation flow of the embodiments of the present invention shown demonstrates that this invention significantly improves the accuracy and robustness of target detection by combining multi-vehicle cooperative radar data sharing with deep learning technology. Through effective data fusion, temporal and spatial noise removal, and optimization of the deep learning model, this invention can accurately identify and locate targets in complex traffic scenarios, exhibiting high stability and adaptability, especially under dynamic traffic flow and adverse weather conditions. The designed loss function effectively optimizes the accuracy of 3D target detection by balancing position, size, and orientation errors. Furthermore, by combining Kalman filtering and attention mechanisms, this invention further enhances the system's detection capability in complex environments, reducing the risk of false detections and missed detections. Overall, this invention possesses strong robustness and generalization ability, can adapt to various complex scenarios, and has broad application prospects, particularly suitable for fields such as autonomous driving, intelligent transportation, and vehicle networking.

[0122] Another aspect of this invention provides a multi-vehicle cooperative 3D target detection system based on radar data, comprising:

[0123] The first module is used to acquire local environmental information of the main vehicle and each cooperating vehicle through radar sensors, and add time and space noise to the acquired local environmental information to obtain radar data.

[0124] The second module is used to share the radar data of the cooperating vehicle to the main vehicle in real time through the vehicle communication system, and to convert the radar data to the coordinate system of the main vehicle through GPS technology.

[0125] The third module is used to convert the radar data of each vehicle into pseudo-images using the PointPillar encoder, and extract the features of the cooperating vehicle and the master vehicle using ResNet18.

[0126] The fourth module is used to remove temporal and spatial noise from the extracted features of the cooperative vehicles using the Kalman filter algorithm.

[0127] The fifth module is used to interact and fuse the features of the cooperating vehicle with the features of the master vehicle through an attention mechanism to obtain fused features;

[0128] The sixth module is used to input the fused features into the PointPillar decoder to generate a 3D target detection box of the surrounding environment and determine the spatial position and shape of the target.

[0129] The seventh module is used to compare the obtained 3D object detection box with the real box by calculating the loss function, calculate the gradient, and backfeed the gradient to update the network parameters, and iteratively optimize the object detection model.

[0130] The eighth module is used to complete multi-vehicle collaborative 3D target detection based on the optimized final target detection model.

[0131] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0132] This invention also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned multi-vehicle cooperative 3D target detection method based on radar data. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0133] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0135] The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0136] The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and called by the processor 501 to execute the multi-vehicle cooperative 3D target detection method based on radar data according to the embodiments of this invention.

[0137] The input / output interface 503 is used to implement information input and output;

[0138] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0139] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0140] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0141] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned multi-vehicle cooperative 3D target detection method based on radar data.

[0142] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0143] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] It should be noted that in various specific embodiments of the present invention, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of the present invention require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data for the normal operation of the embodiments of the present invention acquired.

[0145] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0146] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0147] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0148] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0149] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0150] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0153] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A multi-vehicle cooperative 3D target detection method based on radar data, characterized in that, Includes the following steps: Local environmental information of the main vehicle and each cooperating vehicle is acquired by radar sensors, and temporal and spatial noise is added to the acquired local environmental information to obtain radar data. The radar data of the cooperating vehicle is shared to the main vehicle in real time through the vehicle communication system, and the radar data is converted to the coordinate system of the main vehicle through GPS technology; The radar data of each vehicle is converted into pseudo-images using a PointPillar encoder, and features of the cooperating vehicle and the master vehicle are extracted using ResNet18. The Kalman filter algorithm is used to remove temporal and spatial noise from the extracted features of the cooperative vehicles. Through an attention mechanism, the features of the cooperating vehicle and the features of the master vehicle interact and merge to obtain fused features; The fused features are input into the PointPillar decoder to generate a 3D target detection box of the surrounding environment, determining the spatial location and shape of the target; By calculating the loss function, the obtained 3D target detection box is compared with the ground truth box, the gradient is calculated, and the gradient is back-propagated to update the network parameters, and the target detection model is iteratively optimized. Based on the optimized final target detection model, perform multi-vehicle collaborative 3D target detection; The loss function is used to calculate the difference between the predicted 3D object detection box and the ground truth box, and to update the network parameters through the backpropagation mechanism to optimize the object detection accuracy. loss function The formula is: in, These represent the error calculations for the target's position, size, and orientation, respectively. This refers to the location information of the predicted bounding box; This refers to the position information of the actual bounding box; These are weighting coefficients used to balance the losses of different categories.

2. The multi-vehicle cooperative 3D target detection method based on radar data according to claim 1, characterized in that, The step of sharing radar data from the cooperating vehicle to the main vehicle in real time via the vehicle-to-vehicle communication system, and converting the radar data to the coordinate system of the main vehicle using GPS technology, includes the following steps: The local coordinate system is mapped to the world coordinate system, and position and attitude information are obtained from the vehicle's GPS information. The attitude information includes pitch angle. Yaw angle and roll angle ; Construct rotation matrix and the translation vector of the vehicle in the world coordinate system , to describe how the local coordinate system rotates and translates relative to the world coordinate system; The 3D point cloud features in the world coordinate system are calculated using the linear transformation formula. The specific calculation formula is as follows: ,in, The three-dimensional point cloud features in the world coordinate system; The three-dimensional point cloud features are in a local coordinate system; The world coordinate system is normalized and transformed to the master vehicle coordinate system. The relative translation and the relative rotation matrix of the two vehicles are calculated using the position information of the master vehicle and the slave vehicle. The point cloud data in the world coordinate system is projected to the vehicle coordinate system using the following formula: ,in, 3D point cloud features of the main vehicle coordinate system; Let be the relative rotation matrix between the two vehicles; This is the translation vector of the main vehicle in the world coordinate system.

3. The multi-vehicle cooperative 3D target detection method based on radar data according to claim 1, characterized in that, The process of converting radar data from each vehicle into pseudo-images using a PointPillar encoder and extracting features from the cooperating vehicle and the primary vehicle using ResNet18 includes the following steps: The point cloud data is divided into regular cylindrical shapes according to the top view coordinate system; The initial features of each point include the point's three-dimensional position and reflection intensity; Calculate the relative position of each point with respect to the center of the column and its relative position with respect to the grid origin, forming a point feature vector; point feature vector The calculation formula is: ,in, The three-dimensional position of the representative point; Represents the intensity of reflection; It is the relative coordinate of a point with respect to the center of its column; A point cloud feature extraction network is used to process the features of each point, and the features of points belonging to the same column are aggregated. By using max pooling or average pooling, columnar features are generated, resulting in a fixed-dimensional feature vector corresponding to each column. .

4. The multi-vehicle cooperative 3D target detection method based on radar data according to claim 1, characterized in that, The extracted features of the cooperative vehicles are processed using a Kalman filter algorithm to remove temporal and spatial noise, including the following steps: At every moment In the target state prediction, the prediction is first performed based on the system's state transition model and control input. The expression for the prediction process is as follows: ,in, It is a moment The predicted state, It is the state transition matrix. This is the status update from the previous moment. It is a control input matrix. It is a control input; The covariance of the system is predicted and updated; the expression for this process is: ,in, Indicates time The predicted covariance, Let be the process noise covariance, representing the variation of the unmodeled portion of the model; At every moment New observations were obtained Then, Kalman filtering fuses the predicted state and the actual observation by calculating the Kalman gain, thereby obtaining an updated state estimate. The formula for calculating the Kalman gain is as follows: ,in, For Kalman gain, For the observation matrix, To observe the noise covariance; The updated target state estimate is expressed as: ,in, For the updated target state, The actual observed value at the current moment; The covariance matrix is ​​updated based on the Kalman gain. The expression for the update process is as follows: ,in, The updated covariance matrix, It is an identity matrix.

5. The multi-vehicle cooperative 3D target detection method based on radar data according to claim 1, characterized in that, The process of using an attention mechanism to interact and fuse the features of the cooperating vehicle with those of the master vehicle to obtain fused features includes the following steps: Input data Mapped to a query matrix through linear transformation Key matrix Value matrix : ； in, , , These are linear transformation matrices for the query, key, and value, respectively. and Dimensions for queries, keys, and values; The attention weights are calculated using softmax normalization. The calculation formula is: ; The attention output after weighted summation is obtained, and the features of the cooperating vehicle are interactively fused with the features of the master vehicle to obtain fused features.

6. The multi-vehicle cooperative 3D target detection method based on radar data according to claim 1, characterized in that, The process of inputting fused features into the PointPillar decoder to generate a 3D target detection box of the surrounding environment and determine the spatial location and shape of the target includes the following steps: The PointPillar decoder is used to convert the fused feature information into a 3D target detection box, which represents the position, size, and shape of the target object in the environment. The three-dimensional coordinates and orientation information of the target are generated based on the three-dimensional target detection box; When processing radar data, the PointPillar decoder maps the features obtained from multi-vehicle collaboration onto three-dimensional space to generate the target's three-dimensional bounding box and determine the target's spatial location and shape.

7. A system for implementing the multi-vehicle cooperative 3D target detection method based on radar data as described in any one of claims 1-6, characterized in that, include: The first module is used to acquire local environmental information of the main vehicle and each cooperating vehicle through radar sensors, and add time and space noise to the acquired local environmental information to obtain radar data. The second module is used to share the radar data of the cooperating vehicle to the main vehicle in real time through the vehicle communication system, and to convert the radar data to the coordinate system of the main vehicle through GPS technology. The third module is used to convert the radar data of each vehicle into pseudo-images using the PointPillar encoder, and extract the features of the cooperating vehicle and the master vehicle using ResNet18. The fourth module is used to remove temporal and spatial noise from the extracted features of the cooperative vehicles using the Kalman filter algorithm. The fifth module is used to interact and fuse the features of the cooperating vehicle with the features of the master vehicle through an attention mechanism to obtain fused features; The sixth module is used to input the fused features into the PointPillar decoder to generate a 3D target detection box of the surrounding environment and determine the spatial location and shape of the target. The seventh module is used to compare the obtained 3D object detection box with the real box by calculating the loss function, calculate the gradient, and backfeed the gradient to update the network parameters, and iteratively optimize the object detection model. The eighth module is used to complete multi-vehicle collaborative 3D target detection based on the optimized final target detection model.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 6.

Images