Multi-vehicle cooperative 3D target detection method and system based on radar data

Abstract

The invention discloses a multi-vehicle cooperative 3D target detection method and system based on radar data, and the method comprises the steps: obtaining local environment information of a main vehicle and all cooperative vehicles, and adding noise to obtain radar data; sharing the radar data of the cooperative vehicle to the host vehicle, and converting the radar data to a coordinate system of the host vehicle; converting the radar data into a pseudo image, and extracting features of the cooperative vehicle and the host vehicle; removing time and space noise from the extracted features of the cooperative vehicles; performing interactive fusion on the characteristics of the cooperative vehicle and the main vehicle; generating a three-dimensional target detection frame of the surrounding environment; and by calculating a loss function, circularly optimizing the target detection model, and completing multi-vehicle collaborative 3D target detection. According to the embodiment of the invention, the precision and robustness of target detection can be improved, the automatic driving system can accurately recognize the target in real time in a dynamic and complex road condition, the safety of the automatic driving system is enhanced, and the method can be widely applied to the technical field of intelligent traffic.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to a multi-vehicle collaborative 3D target detection method and system based on radar data. Background Art

[0002] As autonomous driving technology matures, the vehicle's environmental perception capability has become one of the key factors in ensuring autonomous driving safety and improving driving efficiency. Radar, especially millimeter-wave radar, is widely used in autonomous driving systems due to its stable working ability all day and all night. Radar can work stably in adverse weather conditions (such as haze, rain and snow, etc.) and has better penetration ability than lidar and cameras. Therefore, it is particularly outstanding in applications on complex roads and environments.

[0003] Although the single-vehicle radar system has strong environmental perception capabilities, it still has many limitations. Due to the limitations of its field of view and detection distance, single-vehicle radars often have difficulty effectively perceiving objects that are far away from themselves or obscured targets. In addition, the perception results of single-vehicle radars are limited to the perspective of the single vehicle itself and cannot obtain environmental information of other vehicles. This makes it difficult for single-vehicle radars to accurately identify and locate targets in complex traffic environments, especially in dynamic traffic flows. What's more serious is that the accuracy of single-vehicle radars usually depends on the quality of the sensor and environmental conditions. In complex environments with bad weather or multiple obstacles, false detections and missed detections are common, which seriously affects the safety and reliability of the autonomous driving system.

[0004] In order to overcome the limitations of single-vehicle radar systems, multi-vehicle collaborative systems came into being. In a multi-vehicle collaborative environment, through real-time data sharing between vehicles, each vehicle can not only perceive the environment around itself, but also integrate the radar data of other vehicles to obtain more extensive and comprehensive environmental perception information. This multi-vehicle collaborative approach can effectively improve the accuracy of target detection, reduce blind spots, and enhance the robustness of the system. However, although the multi-vehicle collaborative system provides a new idea for solving the limitations of single-vehicle radars, the processing of multi-vehicle collaborative radar data faces a series of challenges, especially position error and time delay problems. The specific problems are as follows:

[0005] 1. Position error problem. In a multi-vehicle collaborative system, the sharing of radar data needs to rely on the vehicle communication system to fuse the radar data from different vehicles. However, since each vehicle has a different position, speed, and orientation, and the relative position between vehicles often changes, how to accurately align the radar data of different vehicles and ensure the spatial consistency of the data has become a core technical problem in the multi-vehicle collaborative system. Especially when the vehicle is driving at high speed or turning sharply, the relative position between vehicles changes rapidly. If the radar data of different vehicles cannot be accurately synchronized, large position errors may be introduced, which will directly affect the accuracy and robustness of target detection.

[0006] 2. Time delay problem. Time delay is another key issue in multi-vehicle cooperative systems. In a multi-vehicle cooperative environment, radar data between vehicles usually needs to be shared in real time through the vehicle communication system. This data transmission will be affected by many factors such as network bandwidth, signal delay, and communication protocol. Especially in high-speed driving or complex traffic environments, data transmission delays are often difficult to avoid, which will lead to inaccurate data synchronization between different vehicles, thereby affecting the detection and judgment of the target by the entire system. For example, when the radar of a vehicle detects an obstacle ahead, due to communication delays, it may take hundreds of milliseconds or even longer for other vehicles to receive the information. This delay in a dynamic traffic environment, especially in rapidly changing scenarios, may cause the environmental perception data of other vehicles to lag, thereby affecting the timely positioning and avoidance decisions of the target. Summary of the invention

[0007] The main purpose of the embodiments of the present invention is to propose a multi-vehicle collaborative 3D target detection method and system based on radar data, which can improve the accuracy and robustness of target detection, enable the autonomous driving system to identify targets in real time and accurately in dynamic and complex road conditions, and enhance the safety of the autonomous driving system.

[0008] To achieve the above object, an embodiment of the present invention provides a multi-vehicle collaborative 3D target detection method based on radar data, comprising the following steps:

[0009] The local environment information of the main vehicle and each cooperative vehicle is obtained through the radar sensor, and time and space noise are added to the obtained local environment information to obtain radar data;

[0010] Sharing the radar data of the cooperative vehicle to the host vehicle in real time through the vehicle communication system, and converting the radar data to the coordinate system of the host vehicle through GPS technology;

[0011] The radar data of each vehicle is converted into a pseudo image using the PointPillar encoder, and the features of the cooperative vehicle and the main vehicle are extracted through ResNet18;

[0012] The Kalman filter algorithm is used to remove the temporal and spatial noises from the extracted features of cooperative vehicles;

[0013] Through the attention mechanism, the features of the cooperative vehicle and the main vehicle are interactively fused to obtain fused features;

[0014] The fused features are input into the PointPillar decoder to generate a 3D target detection frame of the surrounding environment and determine the spatial position and shape of the target;

[0015] By calculating the loss function, the obtained three-dimensional target detection frame is compared with the real frame, the gradient is calculated, and the gradient is returned to update the network parameters, and the target detection model is cyclically optimized;

[0016] The multi-vehicle collaborative 3D target detection is completed based on the optimized final target detection model.

[0017] In some embodiments, the step of sharing the radar data of the cooperative vehicle to the host vehicle in real time through the vehicle communication system and converting the radar data to the coordinate system of the host vehicle through GPS technology includes the following steps:

[0018] Map the local coordinate system to the world coordinate system, and obtain position information and attitude information according to the GPS information of the slave vehicle, wherein the attitude information includes a pitch angle α, a yaw angle β, and a roll angle γ;

[0019] Construct the rotation matrix R and the translation vector T of the vehicle in the world coordinate system to describe the rotation and translation of the local coordinate system relative to the world coordinate system;

[0020] The three-dimensional point cloud features of the world coordinate system are calculated using the linear transformation formula. The specific calculation formula is: world =R·P local +T, where P world is the three-dimensional point cloud feature of the world coordinate system; P local is the three-dimensional point cloud feature of the local coordinate system;

[0021] The world coordinate system is normalized and transformed into the master vehicle coordinate system, and the relative translation and the relative rotation matrix of the two vehicles are calculated based on the position information of the master vehicle and the slave vehicle;

[0022] Project the point cloud data in the world coordinate system to the main vehicle coordinate system. The specific formula is: Among them, P main 3D point cloud features of the main vehicle coordinate system; R relative is the relative rotation matrix of the two vehicles; T2 is the translation vector of the main vehicle in the world coordinate system.

[0023] In some embodiments, the step of converting the radar data of each vehicle into a pseudo image using a PointPillar encoder and extracting the features of the cooperative vehicle and the host vehicle through ResNet18 includes the following steps:

[0024] Divide the point cloud data into regular columns according to the top view coordinate system;

[0025] The initial features of each point include the three-dimensional position and reflection intensity of the point;

[0026] Calculate the relative position of each point relative to the center of the column and the relative position relative to the origin of the grid to form a point feature vector; the point feature vector f point The calculation formula is: point =[x,y,z,I,x rel ,y rel ,z rel ], where x, y, z represent the three-dimensional position of the point; I represents the reflection intensity; x rel ,y rel ,z rel is the relative coordinate of the point with respect to the center of its cylinder;

[0027] A point cloud feature extraction network is used to process each point feature and aggregate the point features belonging to the same column;

[0028] Generate column-level features through maximum pooling or average pooling, and obtain a fixed-dimensional feature vector f corresponding to each column pillar .

[0029] In some embodiments, the extracting the cooperative vehicle features and removing the temporal and spatial noises by using a Kalman filter algorithm comprises the following steps:

[0030] In the target state prediction at each time k, the prediction is first performed based on the system state transition model and control input. The expression of the prediction process is: in, is the predicted state at time k, A is the state transfer matrix, is the updated state at the previous moment, B is the control input matrix, and uk-1 is the control input;

[0031] The covariance of the system is predicted and updated. The expression of this process is: k|k-1 =AP k-1|k-1 A T +Q, where P k|k-1 represents the prediction covariance at time k, Q is the process noise covariance, which represents the change in the unmodeled part of the model;

[0032] At each time point k, a new observation z is obtained kAfter that, the Kalman filter fuses the predicted state and the actual observation by calculating the Kalman gain to obtain the updated state estimate. The calculation formula of the Kalman gain is: k =P k|k-1 H T (HP k|k-1 H T +R) -1 , where K k is the Kalman gain, H is the observation matrix, and R is the observation noise covariance;

[0033] The updated target state estimate is expressed as: in, is the updated target state, z k is the actual observed value at the current moment;

[0034] The covariance matrix is ​​updated according to the Kalman gain. The expression of the updating process is: k|k =(IK k H)P k|k-1 , where P k|k is the updated covariance matrix, and I is the identity matrix.

[0035] In some embodiments, the interactive fusion of the features of the cooperative vehicle and the features of the host vehicle through the attention mechanism to obtain the fused features includes the following steps:

[0036] Input data X∈R n×d Mapped into query matrix Q, key matrix K, and value matrix V through linear transformation:

[0037] Q=XW Q ,K=XW K ,V=XW V ;

[0038] in, are the linear transformation matrices of query, key and value respectively, d k and d v are the dimensions of query, key, and value;

[0039] Use softmax normalization operation to calculate attention weight, attention weight Attention weights The calculation formula is:

[0040] The weighted summed attention output is obtained, so that the features of the cooperative vehicle and the main vehicle are interactively fused to obtain the fused features.

[0041] In some embodiments, the step of inputting the fused features into a PointPillar decoder, generating a three-dimensional target detection frame of the surrounding environment, and determining the spatial position and shape of the target includes the following steps:

[0042] The PointPillar decoder is used to convert the fused feature information into a three-dimensional object detection frame, which represents the position, size, and shape of the target object in the environment;

[0043] Generating three-dimensional coordinates and direction information of the target according to the three-dimensional target detection frame;

[0044] When processing radar data, the PointPillar decoder maps the features obtained by multi-vehicle collaboration into three-dimensional space, generates a three-dimensional bounding box of the target, and determines the spatial position and shape of the target.

[0045] In some embodiments, the loss function is used to calculate the difference between the predicted 3D object detection box and the real box, and to update the network parameters through a back-propagation mechanism to optimize the object detection accuracy;

[0046] The formula of the loss function L is:

[0047] L = ∑(λ 1 *L 1 (x i ,x′ i )+λ 2 *L 2 (y i ,y′ i )+λ 3 *L 3 (z i ,z′ i ))

[0048] Among them, L 1 ,L 2 ,L 3 Respectively represent the error calculation of the target's position, size and direction; x' i ,y′ i ,z′ i is the location information of the prediction box; x i ,y i ,z i is the location information of the real frame; 1 ,λ 2 ,λ 3 is the weight coefficient used to balance the losses of different categories.

[0049] Another aspect of the embodiment of the present invention further provides a multi-vehicle collaborative 3D target detection system based on radar data, including:

[0050] The first module is used to obtain local environmental information of the host vehicle and each cooperative vehicle through a radar sensor, and add temporal and spatial noise to the obtained local environmental information to obtain radar data;

[0051] The second module is used to share the radar data of the cooperative vehicle to the main vehicle in real time through the vehicle communication system, and convert the radar data to the coordinate system of the main vehicle through GPS technology;

[0052] The third module is used to convert the radar data of each vehicle into a pseudo image using the PointPill ar encoder, and extract the features of the cooperative vehicle and the main vehicle through ResNet18;

[0053] The fourth module is used to remove temporal and spatial noises from the extracted cooperative vehicle features using a Kalman filter algorithm;

[0054] The fifth module is used to interactively fuse the features of the cooperative vehicle and the features of the main vehicle through the attention mechanism to obtain the fused features;

[0055] The sixth module is used to input the fused features into the PointPi ll ar decoder to generate a three-dimensional target detection box of the surrounding environment and determine the spatial position and shape of the target;

[0056] The seventh module is used to compare the obtained three-dimensional target detection frame with the real frame by calculating the loss function, calculate the gradient, and return the gradient to update the network parameters, and cyclically optimize the target detection model;

[0057] The eighth module is used to complete multi-vehicle collaborative 3D target detection based on the optimized final target detection model.

[0058] To achieve the above objective, another aspect of an embodiment of the present invention provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0059] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0060] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0061] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a multi-vehicle collaborative 3D target detection method and system based on radar data, which obtains local environmental information of the main vehicle and each collaborative vehicle through a radar sensor, and adds time and space noise to the acquired local environmental information to obtain radar data; the radar data of the collaborative vehicle is shared to the main vehicle in real time through the vehicle-mounted communication system, and the radar data is converted to the coordinate system of the main vehicle through GPS technology; the radar data of each vehicle is converted into a pseudo image using a PointPill ar encoder, and the features of the collaborative vehicle and the main vehicle are extracted through ResNet18; the Kalman filter algorithm is used to remove time and space noise from the extracted features of the collaborative vehicle; through the attention mechanism, the features of the collaborative vehicle are interactively fused with the features of the main vehicle to obtain fused features; the fused features are input into the PointPill ar decoder to generate a three-dimensional target detection frame of the surrounding environment and determine the spatial position and shape of the target; by calculating the loss function, the obtained three-dimensional target detection frame is compared with the real frame, the gradient is calculated, and the gradient is returned to update the network parameters, and the target detection model is cyclically optimized; and the multi-vehicle collaborative 3D target detection is completed according to the final target detection model obtained by optimization. The embodiments of the present invention can improve the accuracy and robustness of target detection, so that the autonomous driving system can identify targets in real time and accurately in dynamic and complex road conditions, thereby enhancing the safety of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0063] Figure 2 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0064] Figure 3 is an example diagram of a multi-vehicle collaborative 3D target detection architecture based on radar data provided by an embodiment of the present invention;

[0065] Figure 4 It is a flowchart of a specific implementation process provided by an embodiment of the present invention;

[0066] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0068] It is understood that the terms "first", "second", "third", "fourth", etc. (if any) in the description of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0069] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0071] The multi-vehicle collaborative 3D target detection method and system based on radar data provided by the embodiment of the present invention relate to the field of intelligent transportation technology. The multi-vehicle collaborative 3D target detection method based on radar data provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a vehicle-mounted terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a multi-vehicle collaborative 3D target detection method based on radar data, etc., but is not limited to the above forms.

[0072] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0073] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0074] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0075] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0076] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0077] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a multi-vehicle collaborative 3D target detection method based on radar data. The following is explained using the multi-vehicle collaborative 3D target detection method based on radar data applied in the server 101 as an example. It can be understood that the method can also be applied to the terminal 102.

[0078] Reference Figure 2 , Figure 2 A flowchart of a multi-vehicle collaborative 3D target detection method based on radar data applied to a server is provided in an embodiment of the present invention. The executor of the method can be any of the aforementioned computer devices (including a server or a terminal). Figure 3 This is an example of a multi-vehicle collaborative 3D target detection architecture based on radar data. Figure 2 and Figure 3 , the method may include the following steps:

[0079] Step 1: Multiple vehicles acquire local environment information through their respective radar sensors and add temporal and spatial noise to the acquired data;

[0080] Step 2: Share the radar data of the cooperative vehicle to the main vehicle in real time through the vehicle communication system, and convert the radar data to the main vehicle coordinate system through GPS technology;

[0081] Step 3: Convert the radar data of each vehicle into a pseudo image using the PointPill ar encoder, and extract the feature information of the cooperative vehicle and the main vehicle through ResNet18;

[0082] Step 4: Use Kalman filtering algorithm to remove temporal and spatial noise from the extracted cooperative vehicle features;

[0083] Step 5: Through the attention mechanism, the cooperative vehicle features are interactively fused with the main vehicle features to obtain fused features;

[0084] Step 6: Input the calculated fusion features into the PointPill ar decoder to generate a three-dimensional target detection box of the surrounding environment and determine the spatial position and shape of the target;

[0085] Step 7: By calculating the loss function, the obtained 3D object detection box is compared with the real box, the gradient is calculated, and the gradient is returned to update the network parameters to optimize the object detection model;

[0086] Step 8: Repeat steps 4-7 until the model converges and save the model parameters.

[0087] In some embodiments, step 1: In the present invention, multiple vehicles obtain local environmental information through their own radar sensors. In order to enhance the robustness of the system, this embodiment adds temporal and spatial noise to the data collected by each vehicle. This noise injection can effectively simulate the measurement uncertainty and relative position information deviation between vehicles faced by sensors during actual driving. In this way, the system can learn more robust features during the training process and can better adapt to various changes in complex environments in subsequent target detection tasks.

[0088] In some embodiments, step 2: In the present invention, the radar data between vehicles is shared in real time through the vehicle-mounted communication system. Since the perspectives of the radar system and sensors of each vehicle are different, the data needs to be processed by GPS technology to convert the radar data of each vehicle to the coordinate system of the main vehicle. In order to ensure the spatial consistency of the radar data of each workshop, this embodiment adopts a timestamp synchronization technology based on GPS, which can not only effectively synchronize the radar data of different vehicles, but also ensure that the multi-vehicle cooperative system can accurately align the data of different vehicles in high-speed driving and dynamic environments, thereby reducing errors caused by time delays or spatial deviations. In step 2, in a multi-vehicle cooperative environment, the vehicles need to share their respective radar data. In order to ensure the timeliness and precise alignment of the data, the present invention uses a vehicle-mounted communication system (such as V2X technology) for data transmission, and uses GPS technology to synchronize the radar data of different vehicles, ensuring that the data of all vehicles can be converted to the coordinate system of the main vehicle, avoiding errors caused by inconsistent data or delays.

[0089] In some embodiments, step 3: In the feature extraction process, the present invention uses a PointPill encoder as a core feature extraction module, which processes three-dimensional point cloud data (or image data). Specifically, the PointPill encoder maps the input point cloud data (or two-dimensional image) to a regularized grid structure, and groups the data through pillars (pills), so that subsequent network layers can process effectively. This encoding method not only improves the computational efficiency, but also can evenly extract meaningful features in dense and sparse areas. In order to further improve the accuracy and reliability of target detection, this embodiment uses the ResNet18 network to extract feature information of the main vehicle and the cooperative vehicle. As a mature feature extraction network, ResNet18 can effectively extract target features at different scales and avoid missed detection problems caused by insufficient feature extraction. In step 3, the PointPill encoder is used to convert radar data of different vehicles into pseudo images, and the deep learning model ResNet18 is used to extract target feature information. The PointPill ar encoder can retain the spatial structure of radar data and convert it into a format that can be effectively processed by neural networks. ResNet18 extracts the key features of the target through convolution operations, improving detection accuracy and feature extraction capabilities.

[0090] In some embodiments, step 4: For collaborative features obtained from multiple vehicles, the present invention uses a Kalman filter algorithm to remove temporal and spatial noise from the features. The Kalman filter algorithm can effectively handle the uncertainty in radar data. By dynamically filtering the radar data of each vehicle, it removes errors caused by factors such as sensor noise and position changes between vehicles, thereby significantly improving the stability and accuracy of the feature data. In the target state prediction at each time k, a prediction is first made based on the system's state transition model and control input. The specific formula is as follows:

[0091]

[0092] in, is the predicted state at time k, A is the state transfer matrix, is the updated state at the previous moment, B is the control input matrix, and uk-1 is the control input.

[0093] Next, the system covariance is also updated by prediction, the formula is:

[0094] P k|k-1 =AP k-1|k-1 A T +Q

[0095] Among them, P k|k-1 represents the prediction covariance at time k, Q is the process noise covariance, which represents the change in the unmodeled part of the model. At each time k, a new observation z is obtained. k After that, the Kalman filter fuses the predicted state and the actual observation by calculating the Kalman gain to obtain the updated state estimate. The calculation formula of the Kalman gain is:

[0096]

[0097] Among them, K k is the Kalman gain, H is the observation matrix, and R is the observation noise covariance, which reflects the uncertainty of the observation. Then, the updated target state estimate can be expressed as:

[0098]

[0099] in, is the updated target state, z k is the actual observation value at the current moment. Finally, the covariance matrix is ​​also updated according to the Kalman gain to reflect the uncertainty of the current estimate. The update formula is:

[0100] P k|k =(IK k H)P k|k-1

[0101] Among them, P k|k is the updated covariance matrix, and I is the identity matrix.

[0102] Through the above Kalman filtering process, the system can accurately estimate the state of the target by fusing multiple sensor data, and effectively deal with noise and uncertainty in the environment, thereby improving the accuracy of target detection and positioning. In addition, Kalman filtering can also provide more accurate target positioning information in a multi-vehicle collaborative environment, ensuring that the system can make accurate target detection in complex environments.

[0103] In step 4, since there may be changes in position and speed between vehicles during the multi-vehicle collaboration process, the present invention uses the Kalman filter algorithm to remove noise from the features of the collaborative vehicles. Kalman filtering can effectively estimate the state of the system and remove noise introduced by factors such as relative motion between vehicles and sensor errors, thereby improving the accuracy of feature data and ensuring the stability of the target detection model in complex environments.

[0104] In some embodiments, step 5: In order to further enhance the fusion of feature information between cooperative vehicles, the present invention introduces an attention mechanism. Through the attention mechanism, the features of the main vehicle and the cooperative vehicle can be weighted and interacted according to their importance, so that the system can give priority to the features that are most critical to target detection. Unlike traditional convolutional neural networks, the attention mechanism can adaptively focus on key areas in images or point clouds and suppress interference from irrelevant information, thereby improving processing efficiency and prediction accuracy. This weighted interaction method not only improves the effectiveness of cooperative vehicle information, but also enhances the system's ability to identify key targets. Specifically, the attention mechanism calculation process is as follows:

[0105] Input data X∈R n×d Mapped into query, key, and value matrices through linear transformation:

[0106] Q=XW Q ,K=XW K ,V=XW V

[0107] in, are the linear transformation matrices of query, key and value respectively, d k and d v The dimensions are query, key, and value.

[0108] The calculation formula for scaled dot product attention is:

[0109]

[0110] Then, the softmax normalization operation is used to calculate the attention weight:

[0111]

[0112] Finally, we get the weighted summed attention output:

[0113] Attention output =Attention weights ×V

[0114] In this way, the calculation of the attention mechanism can be adaptively adjusted in different feature spaces, and the model's attention to key information can be enhanced by weight distribution. In the present invention, this embodiment effectively integrates the features between different vehicles through the attention mechanism, thereby improving the target detection accuracy of the system in a dynamic environment.

[0115] In step 5, in order to improve the target detection capability in the multi-vehicle cooperative system, the present invention introduces an attention mechanism. This mechanism ensures that the system can give priority to the most critical features for target detection and reduce unnecessary interference by performing weighted interaction on the features of the host vehicle and the cooperative vehicles. Through weighted interaction, the system can effectively integrate information from different vehicles and enhance the ability to identify and classify targets.

[0116] In some embodiments, step 6: input the fused features into the PointPi l lar decoder to generate a three-dimensional target detection frame of the surrounding environment. The detection frame can provide the spatial position, size and shape information of the target. When processing radar data, the PointPi l lar decoder can accurately map the features obtained by multi-vehicle collaboration to three-dimensional space and accurately generate a three-dimensional bounding box of the target. On this basis, the system can further perform target classification and positioning to ensure that the target object can be accurately identified in a dynamic traffic environment.

[0117] In step 6, the fused features are input into the PointPi l lar decoder to generate an accurate 3D object detection box. The decoder can map the fused features into 3D space, generate the precise position, size and shape information of the target, and provide key environmental perception data for the autonomous driving system. This step ensures that the target is not only accurately identified in 2D space, but also provides the 3D coordinates and direction of the target, improving the accuracy and applicability of target detection.

[0118] In some embodiments, the step 7: loss function is used to calculate the difference between the predicted 3D target detection box and the real box, and update the network parameters through the back propagation mechanism to optimize the target detection accuracy. The core of this step is to adjust the network parameters by calculating the error between the predicted box and the real box in position, size and direction, thereby improving the model's detection accuracy of the target. Specifically, the formula of the loss function is:

[0119] L = ∑(λ 1 *L 1 (x i ,x′ i )+λ 2 *L 2 (y i ,y′ i )+λ 3 *L 3 (z i ,z′ i ))

[0120] Among them, L 1 ,L 2 ,L 3Respectively represent the error calculation of the target position, size and direction, x′ i ,y′ i ,z′ i is the location information of the prediction box, x i ,y i ,z i is the location information of the real box, λ 1 ,λ 2 ,λ 3 is a weight coefficient used to balance the losses of different categories. This loss function optimizes the target detection accuracy through the back-propagation mechanism and improves the effect of 3D target detection. By introducing different weight coefficients, the system can flexibly adjust the emphasis on position, size and direction errors according to the needs of different tasks, thereby improving the overall target detection performance.

[0121] In summary, reference Figure 4 The specific implementation process of the embodiment of the present invention is shown. The present invention significantly improves the accuracy and robustness of target detection by combining multi-vehicle collaborative radar data sharing with deep learning technology. Through effective data fusion, temporal and spatial noise removal, and optimization of deep learning models, the present invention can accurately identify and locate targets in complex traffic scenes, especially in dynamic traffic flows and severe weather conditions, showing high stability and adaptability. The designed loss function effectively optimizes the accuracy of 3D target detection by balancing position, size, and direction errors. In addition, combined with Kalman filtering and attention mechanisms, the present invention further enhances the detection capability of the system in complex environments and reduces the risk of false detection and missed detection. Overall, the present invention has strong robustness and generalization capabilities, can adapt to a variety of complex scenarios, has broad application prospects, and is particularly suitable for fields such as autonomous driving, intelligent transportation, and vehicle networking.

[0122] Another aspect of the embodiment of the present invention further provides a multi-vehicle collaborative 3D target detection system based on radar data, including:

[0123] The first module is used to obtain local environmental information of the host vehicle and each cooperative vehicle through a radar sensor, and add temporal and spatial noise to the obtained local environmental information to obtain radar data;

[0124] The second module is used to share the radar data of the cooperative vehicle to the main vehicle in real time through the vehicle communication system, and convert the radar data to the coordinate system of the main vehicle through GPS technology;

[0125] The third module is used to convert the radar data of each vehicle into a pseudo image using the PointPill ar encoder, and extract the features of the cooperative vehicle and the main vehicle through ResNet18;

[0126] The fourth module is used to remove temporal and spatial noises from the extracted cooperative vehicle features using a Kalman filter algorithm;

[0127] The fifth module is used to interactively fuse the features of the cooperative vehicle and the features of the main vehicle through the attention mechanism to obtain the fused features;

[0128] The sixth module is used to input the fused features into the PointPi ll ar decoder to generate a three-dimensional target detection box of the surrounding environment and determine the spatial position and shape of the target;

[0129] The seventh module is used to compare the obtained three-dimensional target detection frame with the real frame by calculating the loss function, calculate the gradient, and return the gradient to update the network parameters, and cyclically optimize the target detection model;

[0130] The eighth module is used to complete multi-vehicle collaborative 3D target detection based on the optimized final target detection model.

[0131] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0132] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned multi-vehicle collaborative 3D target detection method based on radar data when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0133] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] See also Figure 5 , Figure 5 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0135] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0136] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes the multi-vehicle collaborative 3D target detection method based on radar data of the embodiment of the present invention;

[0137] Input / output interface 503, used to implement information input and output;

[0138] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0139] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0140] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .

[0141] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multi-vehicle collaborative 3D target detection method based on radar data.

[0142] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0145] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0146] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0147] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0149] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0151] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0152] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0153] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A multi-vehicle collaborative 3D target detection method based on radar data, characterized in that: The following steps are involved: The local environment information of the main vehicle and each cooperative vehicle is obtained through the radar sensor, and time and space noise are added to the obtained local environment information to obtain radar data; Sharing the radar data of the cooperative vehicle to the host vehicle in real time through the vehicle communication system, and converting the radar data to the coordinate system of the host vehicle through GPS technology; The radar data of each vehicle is converted into a pseudo image using the PointPillar encoder, and the features of the cooperative vehicle and the main vehicle are extracted through ResNet18; The Kalman filter algorithm is used to remove the temporal and spatial noises from the extracted features of cooperative vehicles; Through the attention mechanism, the features of the cooperative vehicle and the main vehicle are interactively fused to obtain fused features; The fused features are input into the PointPillar decoder to generate a 3D target detection frame of the surrounding environment and determine the spatial position and shape of the target; By calculating the loss function, the obtained three-dimensional target detection frame is compared with the real frame, the gradient is calculated, and the gradient is returned to update the network parameters, and the target detection model is cyclically optimized; The multi-vehicle collaborative 3D target detection is completed based on the optimized final target detection model.

2. The multi-vehicle collaborative 3D target detection method based on radar data according to claim 1, characterized in that: The step of sharing the radar data of the cooperative vehicle to the host vehicle in real time through the vehicle-mounted communication system and converting the radar data to the coordinate system of the host vehicle through GPS technology includes the following steps: Map the local coordinate system to the world coordinate system, and obtain position information and attitude information according to the GPS information of the slave vehicle, wherein the attitude information includes a pitch angle α, a yaw angle β, and a roll angle γ; Construct the rotation matrix R and the translation vector T of the vehicle in the world coordinate system to describe the rotation and translation of the local coordinate system relative to the world coordinate system; The three-dimensional point cloud features of the world coordinate system are calculated using the linear transformation formula. The specific calculation formula is: world =R·P local +T, where P world is the three-dimensional point cloud feature of the world coordinate system; P local is the three-dimensional point cloud feature of the local coordinate system; The world coordinate system is normalized and transformed into the master vehicle coordinate system, and the relative translation and the relative rotation matrix of the two vehicles are calculated based on the position information of the master vehicle and the slave vehicle; Project the point cloud data in the world coordinate system to the main vehicle coordinate system. The specific formula is: Among them, P main 3D point cloud features of the main vehicle coordinate system; R relative is the relative rotation matrix of the two vehicles; T2 is the translation vector of the main vehicle in the world coordinate system.

3. The multi-vehicle collaborative 3D target detection method based on radar data according to claim 1, characterized in that: The method of converting the radar data of each vehicle into a pseudo image using a PointPillar encoder and extracting the features of the cooperative vehicle and the main vehicle through ResNet18 includes the following steps: Divide the point cloud data into regular columns according to the top view coordinate system; The initial features of each point include the three-dimensional position and reflection intensity of the point; Calculate the relative position of each point relative to the center of the column and the relative position relative to the origin of the grid to form a point feature vector; the point feature vector f point The calculation formula is: point =[x,y,z,I,x rel ,y rel ,z rel ], where x, y, z represent the three-dimensional position of the point; I represents the reflection intensity; x rel ,y rel ,z rel is the relative coordinate of the point with respect to the center of its cylinder; A point cloud feature extraction network is used to process each point feature and aggregate the point features belonging to the same column; Generate column-level features through maximum pooling or average pooling, and obtain a fixed-dimensional feature vector f corresponding to each column pillar .

4. The multi-vehicle collaborative 3D target detection method based on radar data according to claim 1, characterized in that: The extracted cooperative vehicle features are subjected to a Kalman filter algorithm to remove temporal and spatial noise, comprising the following steps: In the target state prediction at each time k, the prediction is first performed based on the system state transition model and control input. The expression of the prediction process is: in, is the predicted state at time k, A is the state transfer matrix, is the updated state at the previous moment, B is the control input matrix, and uk-1 is the control input; The covariance of the system is predicted and updated. The expression of this process is: k|k-1 =AP k-1|k-1 A T +Q, where P k|k-1 represents the prediction covariance at time k, Q is the process noise covariance, which represents the change in the unmodeled part of the model; At each time point k, a new observation z is obtained k After that, the Kalman filter fuses the predicted state and the actual observation by calculating the Kalman gain to obtain the updated state estimate. The calculation formula of the Kalman gain is: k =P k|k-1 H T (HP k|k-1 H T +R) -1 , where K k is the Kalman gain, H is the observation matrix, and R is the observation noise covariance; The updated target state estimate is expressed as: in, is the updated target state, z k is the actual observed value at the current moment; The covariance matrix is ​​updated according to the Kalman gain. The expression of the updating process is: k|k =(IK k H)P k|k-1 , where P k|k is the updated covariance matrix, and I is the identity matrix.

5. The multi-vehicle collaborative 3D target detection method based on radar data according to claim 1, characterized in that: The method of interactively fusing the features of the cooperative vehicle and the features of the main vehicle through the attention mechanism to obtain the fused features includes the following steps: Input data X∈R n×d Mapped into query matrix Q, key matrix K, and value matrix V through linear transformation: Q=XW Q ,K=XW K ,V=XW V ; in, are the linear transformation matrices of query, key and value respectively, d k and d v are the dimensions of query, key, and value; Use softmax normalization operation to calculate attention weight, attention weight Sttention weights The calculation formula is: The weighted summed attention output is obtained, so that the features of the cooperative vehicle and the main vehicle are interactively fused to obtain the fused features.

6. The multi-vehicle collaborative 3D target detection method based on radar data according to claim 1 is characterized in that: The step of inputting the fused features into the PointPillar decoder, generating a three-dimensional target detection frame of the surrounding environment, and determining the spatial position and shape of the target includes the following steps: The PointPillar decoder is used to convert the fused feature information into a three-dimensional object detection frame, which represents the position, size, and shape of the target object in the environment; Generating three-dimensional coordinates and direction information of the target according to the three-dimensional target detection frame; When processing radar data, the PointPillar decoder maps the features obtained by multi-vehicle collaboration into three-dimensional space, generates a three-dimensional bounding box of the target, and determines the spatial position and shape of the target.

7. The multi-vehicle collaborative 3D target detection method based on radar data according to claim 1, characterized in that: The loss function is used to calculate the difference between the predicted 3D object detection box and the real box, and to update the network parameters through the back-propagation mechanism to optimize the object detection accuracy; The formula of the loss function L is: L=∑(λ 1 *L 1 (x i ,x′ i )+λ 2 *L 2 (y i ,y′ i )+λ 3 *L 3 (z i ,z′ i )) Among them, L 1 ,L 2 ,L 3 Respectively represent the error calculation of the target's position, size and direction; x' i ,y′ i ,z′ i is the location information of the prediction box; x i ,y i ,z i is the location information of the real frame; 1 ,λ 2 ,λ 3 is the weight coefficient used to balance the losses of different categories.

8. A multi-vehicle collaborative 3D target detection system based on radar data, characterized in that: include: The first module is used to obtain local environmental information of the host vehicle and each cooperative vehicle through a radar sensor, and add temporal and spatial noise to the obtained local environmental information to obtain radar data; The second module is used to share the radar data of the cooperative vehicle to the main vehicle in real time through the vehicle communication system, and convert the radar data to the coordinate system of the main vehicle through GPS technology; The third module is used to convert the radar data of each vehicle into a pseudo image using the PointPillar encoder and extract the features of the cooperative vehicle and the main vehicle through ResNet18; The fourth module is used to remove temporal and spatial noises from the extracted cooperative vehicle features using a Kalman filter algorithm; The fifth module is used to interactively fuse the features of the cooperative vehicle and the features of the main vehicle through the attention mechanism to obtain the fused features; The sixth module is used to input the fused features into the PointPillar decoder to generate a 3D target detection box of the surrounding environment and determine the spatial position and shape of the target; The seventh module is used to compare the obtained three-dimensional target detection frame with the real frame by calculating the loss function, calculate the gradient, and return the gradient to update the network parameters, and cyclically optimize the target detection model; The eighth module is used to complete multi-vehicle collaborative 3D target detection based on the optimized final target detection model.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images