A multi-vehicle cooperative perception post-fusion method, device, equipment and storage medium

Abstract

This invention relates to the field of connected vehicle technology, specifically to a post-fusion method, apparatus, device, and storage medium for multi-vehicle cooperative perception. The post-fusion method includes the following steps: generating a first target trajectory and other vehicle trajectories based on the first target state information sensed by the vehicle itself and the received state information of other vehicles; when a first target trajectory similar to the other vehicle's trajectory exists, correcting all remaining first target state information based on the deviation between the other vehicle's trajectory and the similar first target trajectory, and then fusing it with all second target state information sensed by the other vehicles. This addresses the problem of poor accuracy after fusion in traditional post-fusion methods due to insufficient performance of certain sensors in specific scenarios and limitations of perception algorithms.

Description

Technical Field

[0001] This invention relates to the field of connected vehicle technology, specifically to a post-fusion method, apparatus, device, and storage medium for multi-vehicle collaborative perception. Background Technology

[0002] Intelligent vehicles are a crucial component of intelligent transportation systems. They possess sensors that can identify their surroundings and analyze sensor information to determine the type, location, and motion status of objects in the vicinity. However, single-vehicle perception often has blind spots, leading to inaccurate perception results. To overcome the limitations of single-vehicle perception, multi-vehicle cooperative perception fusion technology has significant application value in connected vehicle systems.

[0003] Multi-vehicle cooperative perception refers to the process where a vehicle (also known as the master vehicle or fusion vehicle) and multiple other vehicles with sensing and network communication capabilities (also known as sensing vehicles) transmit the perception information of the other vehicles to the vehicle in a timely manner through vehicle-to-vehicle (V2V) communication. The vehicle then performs information fusion to expand its field of vision. Multi-vehicle cooperative perception improves the efficiency and accuracy of the perception system by fusing data from different vehicles. This cooperative perception can significantly enhance the vehicle's ability to perceive its surroundings, significantly improve the positional accuracy, motion accuracy, and system stability of perceived targets, and help the driving system make safer and more reliable decisions. Depending on the stage of vehicle perception data sharing and cooperation, it is generally divided into pre-fusion and post-fusion methods. Pre-fusion is the most direct cooperative perception method, which refers to the fusion of raw (or lightly processed) sensor information shared by each vehicle. However, this places high demands on bandwidth and makes it difficult to meet the real-time requirements of driving decisions. In addition, fusing data received from various sensors of other vehicles and processing large amounts of raw data is challenging for the automotive electronic control unit (ECU). Therefore, in practice, a more common approach is post-fusion, where each vehicle shares processed sensor information, such as displaying the position, attitude, and predicted trajectory of surrounding objects, vehicles, and pedestrians.

[0004] However, traditional post-fusion methods may suffer from poor fusion results due to limitations in the performance of certain sensors in specific scenarios and the limitations of perception algorithms. For example, radar has weak detection capabilities for near targets, and cameras exhibit lateral drift in some scenarios. Without proper processing, directly fusing the detected data will limit the accuracy of multi-vehicle perception fusion in real-world applications. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a post-fusion method, apparatus, device, and storage medium for multi-vehicle collaborative perception, which can solve the problem that the accuracy of fusion is poor due to the insufficient performance of certain sensors in specific scenarios and the limitations of perception algorithms in traditional post-fusion methods.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the invention provides a post-fusion method for multi-vehicle collaborative perception, comprising the following steps:

[0008] Based on the first target state information sensed by the vehicle and the received state information of other vehicles, the trajectory of the first target and the trajectory of other vehicles are generated.

[0009] When there is a first target trajectory similar to that of another vehicle, the state information of all other first targets is corrected based on the deviation between the trajectory of the other vehicle and the similar first target trajectory, and then fused with the state information of all second targets perceived by the other vehicle.

[0010] In some alternative schemes, after generating the first target trajectory and the trajectory of other vehicles, it is determined whether there is a first target trajectory similar to the trajectory of other vehicles based on the spatial distance between each first target trajectory and the trajectory of other vehicles.

[0011] In some alternative solutions, determining whether there exists a first target trajectory similar to the trajectory of another vehicle based on the spatial distance between each first target trajectory and the trajectory of another vehicle includes:

[0012] For each first target trajectory and each other vehicle trajectory, the first maximum spatial distance value is obtained based on the minimum spatial distance from each trajectory point in the first target trajectory to the other vehicle trajectory, and the second maximum spatial distance value is obtained based on the minimum spatial distance from each trajectory point in the other vehicle trajectory to the first target trajectory.

[0013] If both the first maximum spatial distance value and the second maximum spatial distance value are less than a set threshold, then the trajectory of the first target is determined to be similar to the trajectory of the corresponding other vehicle.

[0014] In some alternative solutions, the correction of the state information of all other first targets based on the deviation between the trajectory of another vehicle and the trajectory of a similar first target includes:

[0015] Based on the trajectory points at sampling time t in the similar first target trajectory, find the trajectory points at two times adjacent to the sampling time t in the trajectory of the corresponding other vehicle;

[0016] The state information of the trajectory point at time t in the trajectory of another vehicle is obtained by using the difference method or taking the average of the two trajectory points adjacent to the sampling time t.

[0017] Based on the state deviation between the trajectory points at each sampling time t in the similar first target trajectory and the corresponding trajectory points at the corresponding time t in the trajectory of other vehicles, the state information of all other first targets is corrected.

[0018] In some alternative schemes, if multiple other vehicle trajectories have corresponding similar first target trajectories, the deviation value between each similar first target trajectory and the trajectory point at the same time in the corresponding other vehicle trajectory is obtained. The deviation value used for correction is the minimum, average, or weighted average of multiple deviation values.

[0019] In some alternative solutions, when there is no first target trajectory similar to that of other vehicles, the state information of all first targets, the state information of other vehicles, and the state information of second targets are directly fused.

[0020] In some alternative solutions, the characteristic is that a temporal recurrent attention network model is used for fusion, and the input data is normalized and time-aligned.

[0021] Secondly, the present invention also provides a post-fusion device for multi-vehicle cooperative perception, comprising:

[0022] The trajectory generation module is used to generate the trajectory of the first target and the trajectory of the other vehicle based on the first target state information perceived by the vehicle and the received state information of the other vehicle.

[0023] The fusion module is used to, when there is a first target trajectory similar to that of another vehicle, correct the state information of all other first targets based on the deviation value between the trajectory of the other vehicle and the similar first target trajectory, and then fuse it with the state information of all second targets perceived by the other vehicle.

[0024] Thirdly, the present invention also provides a computer device, characterized in that the computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the post-fusion method for multi-vehicle cooperative perception described above.

[0025] Fourthly, the present invention also provides a computer-readable storage medium storing a multi-vehicle cooperative perception post-fusion program, wherein when the multi-vehicle cooperative perception post-fusion program is executed by a processor, it implements the steps of the multi-vehicle cooperative perception post-fusion method described in any of the preceding claims.

[0026] Compared with existing technologies, the advantages of this invention are as follows: Since the accuracy of the received state information of other vehicles is much higher than the accuracy of the position and motion state of surrounding targets perceived by vehicle sensors, this solution generates the first target trajectory and the trajectory of other vehicles based on the first target state information perceived by the vehicle itself and the received state information of other vehicles. When there is a first target trajectory similar to the trajectory of other vehicles, the state information of all other first targets is corrected based on the deviation value between the trajectory of other vehicles and the similar first target trajectory. Then, the second target state information perceived by other vehicles is fused with the corrected first target state information. This can significantly improve the perception accuracy in many scenarios, reduce the perception deviation caused by sensor perception errors or environment at a lower cost, and significantly improve the target perception accuracy through collaboration based on existing vehicle perception hardware and algorithms, thereby improving the perception capability of a single vehicle and thus improving the safety of the driving system. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of the post-fusion method for multi-vehicle cooperative perception in an embodiment of the present invention;

[0029] Figure 2 This is a schematic diagram illustrating data interaction between the vehicle and other vehicles in an embodiment of the present invention;

[0030] Figure 3 This is a flowchart of another embodiment of the post-fusion method in this invention; Figure 4 This is a flowchart of trajectory generation in an embodiment of the present invention;

[0031] Figure 5 This is a flowchart illustrating the trajectory similarity comparison in an embodiment of the present invention;

[0032] Figure 6 This is a flowchart of the self-vehicle sensing target state correction process in an embodiment of the present invention;

[0033] Figure 7 This is a schematic diagram of the vehicle sensing target state correction in an embodiment of the present invention;

[0034] Figure 8 This is a schematic diagram of training data acquisition in an embodiment of the present invention;

[0035] Figure 9 This is a diagram of the deep neural network model architecture in an embodiment of the present invention;

[0036] Figure 10 This is a schematic block diagram of the structure of a computer device in an embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, in a first aspect, the present invention provides a post-fusion method for multi-vehicle cooperative perception, comprising the following steps:

[0040] S100: Generate the trajectory of the first target and the trajectory of the other vehicle based on the first target state information perceived by the vehicle and the state information of the other vehicle received.

[0041] like Figure 2 As shown, each vehicle equipped with sensing and network communication capabilities can collect its own vehicle status information and send it to other vehicles, i.e., BSM information X. bsm ={x b1 ,x b2 ,…x bN} represents the vehicle's status information sent out by the vehicle over a period of time, where x b1 This represents the vehicle's status information at time t1, x b1 ={t1,lat b1 ,lon b1 ,v b1 ,h ab1 ,a xb1 ,a yb1 ,a zb1 ,r yawb1}, N represents the Nth time point; it can also sense the target status information around the vehicle and send it to other vehicles, i.e., SSM information X. ssm ={x s1 ,x s2 ,…x sN}, SSM represents the perceived target state information transmitted by the vehicle over a period of time, where x s1 x represents the perceived target state information at time t1. s1 ={t1,lat 11 ,lon 11 ,v11 ,h a11 ,a x11 ,a y11 ,a z11 ,r yaw11 ;...;lat j1 ,lon j1 ,v j1 ,h aj1 ,a xj1 ,a yj1 ,a zj1 ,r yawj1 ;...;lat n1 ,lon n1 ,v n1 ,h an1 ,a xn1 ,a xn1 ,a n1 ,r yawn1}, where j represents the target sequence, n is the total number, and t,lat,lon,v,h a ,a x ,a y ,a z ,r yaw These represent the object's timestamp, latitude, longitude, velocity, heading angle, lateral acceleration, longitudinal acceleration, vertical acceleration, and yaw rate, respectively. All values ​​are represented by 's'. i To represent x bsm and x ssm The target state information perceived in the image, for BSM information, one piece of information includes only one object, corresponding to one s. i For SSM information, one piece of information includes n perceived objects, corresponding to n s i s from the same SSM i t i If the values ​​are the same, then:

[0042] s i ={t i ,lat i ,lon i ,v i ,h ai ,a xi ,a yi ,a zi ,c yawi}

[0043] S = {s1, s2, ... s} M}

[0044] i represents the i-th target status information, M is the total number of targets in all BSM and SSM information received by the vehicle during the collection period, and S is the set of statuses of these targets.

[0045] In this example, multi-target trajectory generation mainly consists of two parts: association between perceived data and historical trajectory data, and trajectory management. The first target trajectory is generated based on the first target's state information sensed by the vehicle. In this example, the collected state information of surrounding targets, i.e., SSM information, is recorded as the target state information at time k+1.

[0046] like Figure 4 As shown, in this embodiment, S100A: Based on the first target state information perceived by the vehicle, a first target trajectory is generated, including the following steps:

[0047] S101A: Collect the latest trajectory point status at time K from the existing generated trajectory data.

[0048] S102A: Perform data association between the state values ​​of all existing generated trajectory data at time k and the state values ​​of all newly sensed targets at time k+1.

[0049] In this example, the Hungarian algorithm is used for association. Three results are obtained: matched sensing targets, unmatched sensing targets, and existing generated trajectory data containing state information of unmatched sensing targets.

[0050] S1031A: For a matched sensing target, the sensing state at time k+1 is added to the existing generated trajectory data.

[0051] S1032A: For a sensing target that is not matched, the sensing target is considered a new trajectory point, which needs to be created as a new trajectory and added to the existing trajectory set.

[0052] S1033A: For existing generated trajectory data that does not match the state information of a sensed target, if no sensed target matches it for several consecutive times, delete the existing generated trajectory data. It is assumed that the trajectory target has left the sense range.

[0053] The above is an iterative process for trajectory generation. The initial trajectory is the trajectory point of the perceived target at the earliest time t. By iterating the above process repeatedly based on the initial trajectory, the set of perceived target trajectories of the vehicle in continuous time can be obtained, including at least one first target trajectory perceived by the vehicle. If multiple targets are perceived, multiple first target trajectories will be generated.

[0054] S100B: For BSM information received by the vehicle from other vehicles, the vehicle generates the trajectory of the other vehicle based on the received state information of the other vehicle itself. Specifically, the spatiotemporal state points of the vehicle can be obtained from the BSM information sent by the other vehicle within a certain period of time, and the driving trajectory of the other vehicle within that period of time can be generated based on this state information of the other vehicle.

[0055] In this example, if there are multiple vehicles around the vehicle that are sending out BSM information, and the vehicle receives the BSM information, the vehicle can generate the trajectory of the other vehicle based on the BSM information received from the multiple vehicles.

[0056] Both the vehicle and other vehicles possess positioning, sensing, communication, and computing functions. Positioning is generally provided by a Global Navigation Satellite System (GNSS) receiver, sensing by radar, cameras, etc., communication by an On-Board Unit (OBU), and computing power by an electronic control unit to run a fusion model. Other vehicles (including but not limited to those broadcasting their own position, size, and motion status via Basic Safety Messages (BSMs) according to T / CSAE 53-2020 standards) and broadcast (including but not limited to those broadcasting the position, size, and motion status of sensed targets via Sensor Sharing Messages (SSMs) according to T / CSAE 157-2020 standards).

[0057] like Figure 3 As shown, S200: After the step of generating the first target trajectory and the trajectory of other vehicles, it further includes: determining whether there is a first target trajectory similar to the trajectory of other vehicles based on the spatial distance between each first target trajectory and the trajectory of other vehicles.

[0058] Since BSM (Balanced Sense Monitoring) measures a vehicle's own position and state, its accuracy is typically much higher than that of SSM (Side Sense Monitoring) information providing the target's position and state. Therefore, the SSM information perceived by the vehicle can be corrected based on the trajectories of other vehicles obtained from BSM information.

[0059] In this example, Hausdorff distance is used to compare the similarity between the target trajectory perceived by the vehicle and the BSM trajectory of another vehicle. This does not mean that the invention is limited to using Hausdorff distance; all appropriate trajectory similarity comparison methods can be used. For example, Euclidean distance can be used: calculating the Euclidean distance between corresponding points on two trajectories, and then summing these distances (e.g., summing or averaging) to obtain a similarity measure. Alternatively, Hausdorff distance, edit distance, etc., can also be used.

[0060] In addition, other similarity comparison methods are used in other embodiments to determine whether there is a first target trajectory similar to the trajectory of another vehicle. For example, Dynamic Time Warping (DTW): the optimal alignment path between two trajectories is found through dynamic programming algorithm, thereby calculating the minimum cumulative distance between them.

[0061] like Figure 5 As shown, in some optional embodiments, determining whether there is a first target trajectory similar to the trajectory of another vehicle based on the spatial distance between each first target trajectory and the trajectory of another vehicle includes:

[0062] S201: For each first target trajectory and each other vehicle trajectory, obtain the first maximum spatial distance value based on the minimum spatial distance from each trajectory point in the first target trajectory to the other vehicle trajectory, and obtain the second maximum spatial distance value based on the minimum spatial distance from each trajectory point in the other vehicle trajectory to the first target trajectory.

[0063] Specifically, for any first target trajectory in the first target trajectory set generated based on the received SSM information, the minimum spatial distance from each trajectory point in the first target trajectory to the trajectory of another vehicle is calculated, and the set of minimum spatial distances from each trajectory point in the first target trajectory to the trajectory of another vehicle is obtained. The maximum value Max_1 is obtained by sorting the sets of minimum spatial distances from each trajectory point of the other vehicle to the first target trajectory. Similarly, the minimum spatial distance from each point of the other vehicle trajectory to the first target trajectory is calculated, and the set of minimum spatial distances from each point of the other vehicle trajectory to the first target trajectory is obtained by sorting the sets of minimum spatial distances from the other vehicle trajectory to the first target trajectory. The maximum value Max_2 is obtained by sorting the sets of minimum spatial distances from each trajectory point of the other vehicle to the first target trajectory.

[0064] S202: If both the first maximum spatial distance value and the second maximum spatial distance value are less than the set threshold, then the trajectory of the first target is determined to be similar to the trajectory of the corresponding other vehicle.

[0065] In this embodiment, the values ​​of Max_1 and Max_2 are compared, and the larger one is the Hausdorff distance between the first target trajectory and the trajectory of other vehicles. If the first target trajectory is concentrated in a group with a Hausdorff distance less than a set threshold d, the distance between the target trajectory and the trajectory of other vehicles is determined. thred If the trajectory information is obtained, it is determined that the trajectory of the first target is similar to the trajectory of another vehicle, indicating that the object corresponding to the trajectory of the first target and the vehicle corresponding to the trajectory of another vehicle are the same target.

[0066] In this example, the first target status information is the target status information perceived by the vehicle itself, and the second target status information is the target status information perceived by other vehicles and sent to the vehicle itself.

[0067] In other embodiments, the first maximum spatial distance value and the second maximum spatial distance can also be directly compared with a set threshold. When both the first maximum spatial distance value and the second maximum spatial distance value are less than the set threshold, the first target trajectory is considered to be similar to the corresponding trajectory of another vehicle.

[0068] Iterate through all first target trajectories in the first target trajectory set generated by SSM information, and perform similarity comparisons between all first target trajectories and other vehicle trajectories.

[0069] If the vehicle receives BSM information from multiple vehicles, it will generate multiple trajectories of other vehicles. For each other vehicle trajectory, the spatial distance between the other vehicle trajectory and all first target trajectories will be calculated in the same way as described above, and a similarity comparison will be completed.

[0070] S300: When there is a first target trajectory similar to that of another vehicle, based on the deviation between the trajectory of another vehicle and the similar first target trajectory, the state information of all other first targets is corrected and then fused with the state information of all second targets perceived by the other vehicle.

[0071] like Figure 6 and Figure 7 As shown, in some optional embodiments, the state information of all other first targets is corrected based on the deviation between the trajectory of another vehicle and its similar first target trajectory, including:

[0072] S311: Based on the trajectory points at sampling time t in the similar first target trajectory, find the trajectory points at two times adjacent to sampling time t in the corresponding other vehicle trajectory.

[0073] In this example, sampling time t is the time with timestamp t, that is, the time when the sampled data is recorded. Specifically, for the trajectory points with timestamp t in the similar first target trajectory, find two trajectory points in the trajectory of the other vehicle with the smallest interval time interval including time t. Their timestamps are t1 and t2, which are the two times in the trajectory of the other vehicle that are adjacent to sampling time t.

[0074] S312: The state information of the trajectory point at time t in the trajectory of another vehicle is obtained by using the difference method or taking the average of the state information of the trajectory points at two adjacent sampling times of sampling time t.

[0075] See you again Figure 7 In this embodiment, based on the difference between times t1 and t2 and time t, the difference method is used to obtain the state information of the trajectory point at time t in the trajectory of another vehicle based on the state information of the trajectory points corresponding to the trajectory of another vehicle at times t1 and t2. Alternatively, the average value of the state information of the trajectory points corresponding to the trajectory of another vehicle at times t1 and t2 can be directly used as the state information of the trajectory point at time t in the trajectory of another vehicle.

[0076] For each sampling time in the similar first target trajectory, the state information of the trajectory point at the corresponding time in the trajectory of the other vehicle is found.

[0077] S313: Based on the state deviation between the trajectory points at each sampling time t in the similar first target trajectory and the corresponding trajectory points at time t in the trajectory of other vehicles, correct the state information of all other first targets.

[0078] Specifically, the original state information at time t in the similar first target trajectory is compared with the corresponding trajectory point state information at time t in the obtained trajectory of other vehicles to obtain the deviation value of each state information, including position, size, and motion state. This deviation value is applied to the SSM information perceived by the vehicle, and the state information of all other objects perceived by the vehicle is completed, corrected and calibrated by the deviation value.

[0079] In some optional embodiments, if multiple other vehicle trajectories have corresponding similar first target trajectories, the deviation value between each similar first target trajectory and the trajectory point at the same time in the corresponding other vehicle trajectory is obtained. The deviation value used for correction is the minimum value, average value or weighted average value among multiple deviation values.

[0080] If a vehicle receives BSM information from multiple vehicles, it will generate multiple trajectories for other vehicles, and multiple of these trajectories will have corresponding similar first target trajectories. First, the deviation value between each similar first target trajectories and the corresponding trajectory points at the same time in the other vehicle trajectories will be obtained using the aforementioned deviation value calculation method. The final deviation value used to correct other first target state information is taken as the minimum of multiple deviation values, or the average of multiple deviation values ​​can be used as the final deviation value. Alternatively, multiple deviation values ​​can be weighted according to the similarity between the corresponding other vehicle trajectory and the corresponding similar first target trajectory, and the weighted average value can be used as the final deviation value to correct other first target state information. Of course, other methods can also be used to process the deviation values ​​between multiple similar first target trajectories and the corresponding trajectory points at the same time in the other vehicle trajectories to obtain the final deviation value to correct the remaining first target state information.

[0081] In this embodiment, if the trajectories of all the first targets perceived by the vehicle are not similar to the trajectories of all other vehicles, it is assumed that there are no other vehicles participating in the perception of the targets currently perceived by the vehicle, or that there is no trajectory information of other vehicles during this period. The original single-vehicle perception information is retained for subsequent steps.

[0082] S400: When there is no first target trajectory similar to that of other vehicles, directly fuse all first target state information, other vehicles' own state information, and second target state information.

[0083] In some optional embodiments, a temporal recurrent attention network model is used for fusion, and the input data is normalized and time-aligned.

[0084] In this example, after correcting all other first target state information, the system merges all second target state information perceived by other vehicles, or directly merges all first target state information, other vehicle's own state information, and second target state information. Both methods employ a temporal recurrent attention network model.

[0085] Before using a temporal recurrent attention network model for fusion, it is necessary to filter, clean, and normalize all the first target state information, the other vehicle's own state information, and the second target state information. This involves processing all target states ordered chronologically. Where k represents the identification number of the sensing vehicle, and n represents the number of state information entries for the sensed object, including the state information of the sensing vehicle itself. After obtaining the cleaned data, time alignment is performed based on the vehicle's time, and the results are categorized according to the corresponding vehicle information. The vehicle's time is divided into durations T. For all vehicles, a standardized tensor is constructed for the sensing information within time T, and a reasonable time series length, sequencelen, is designed. When the length of the sensing information within time T exceeds sequencelen, the spatiotemporal sequence information closest to the current time is selected, and the remaining information is truncated. When the length of the sensing information within time T is less than sequencelen, positions with insufficient sequence length are padded with 0s to ensure uniform tensor length.

[0086] The input data for the temporal recurrent attention network model is of size batchsize * vsize * sequencelen * 9, where batchsize is the batch size for model training, vsize represents the number of participating perception vehicles, sequencelen is the length of the time series, and 9 is the feature dimension selected for each time step, i.e., the spatiotemporal information s of the perceived target state. i ={t i ,lat i ,lon i ,v i ,h ai ,a xi ,a yi ,a zi ,r yawi This includes information such as time, object position, velocity, acceleration, and heading angle.

[0087] The output data of the temporal recurrent attention network model is as follows: For each time step, the temporal recurrent attention network model predicts information in 9 dimensions. The size of the model output is batchsize * itemsize * 9, where itemsize is the number of objects divided in that scene.

[0088] The standardized tensor is fed into the temporal recurrent attention network F(θ) for prediction. The prediction results are then used for state recovery and post-processing, such as removing duplicate targets, to obtain the refined position, size, and motion state of all targets perceived by all vehicles. The refined position, size, and motion state of all targets perceived by all vehicles after removing duplicate targets can be displayed on the HMI interface or used as input parameters for driver assistance or intelligent driving control modules.

[0089] The HMI (Human-Machine Interface) unit is responsible for displaying the driving status and providing reminders to the driver through images, sounds, and other means. The terms "self vehicle" and "other vehicles (sensing vehicles)" mentioned above are for ease of description only. All vehicles with sensing and communication capabilities can be considered as both self and other vehicles; that is, when a vehicle is considered as its own vehicle, the surrounding vehicles are considered as other vehicles (sensing vehicles).

[0090] In summary, the accuracy of the vehicle's own position and motion state obtained through its own sensors (i.e., the received state information of other vehicles) is significantly higher than the accuracy of the position and motion state perceived by other vehicles through sensors such as cameras and radar. By generating the first target trajectory and other vehicle trajectories based on the first target state information perceived by the vehicle itself and the received state information of other vehicles, the system detects whether surrounding vehicles belong to targets already perceived by the vehicle in multi-vehicle collaborative perception. When a first target trajectory similar to that of other vehicles exists, it is considered that a vehicle that has been perceived by the vehicle and can generate its own state information exists. Based on the deviation between the trajectory of other vehicles and the similar first target trajectory, the state information of all other first targets is corrected. The deviation between the vehicle's perception data and the shared self-sense (BSM) data of other vehicles is used to calibrate the vehicle's perception data, which can significantly improve perception accuracy in many scenarios. It can reduce perception deviations caused by sensor perception errors or environmental factors at a lower cost, and can significantly improve target perception accuracy through collaboration based on existing vehicle perception hardware and algorithms, thereby improving single-vehicle perception capabilities and ultimately enhancing the safety of the driving system.

[0091] In addition, training the temporal recurrent attention network model includes the following steps:

[0092] First, a temporal recurrent attention network model is established, and training and validation datasets are collected. The temporal recurrent attention network model is trained using the training dataset, and the trained temporal recurrent attention network model is validated using the validation dataset. If the output of the temporal recurrent attention network model does not meet the accuracy requirements, the training dataset is collected again, and the temporal recurrent attention network model is trained again until the output of the temporal recurrent attention network model meets the accuracy requirements.

[0093] like Figure 8As shown, during data collection, ensure that at least three vehicles with communication, sensing, and computing capabilities are present. All vehicles must be operating and communicating normally, with their BSM and SSM information transmission and reception, as well as their position and motion status data, functioning correctly. Global time synchronization must be performed between the vehicle and other vehicles (via, but not limited to, GNSS). The data collection scenario is divided into an analysis and evaluation scenario (Scenario 1) and a verification scenario (Scenario 2).

[0094] Scenario 1 is used for training data collection. In the absence of other vehicles, one vehicle is in front (as the ground truth vehicle) to ensure that the other two vehicles (as perception fusion vehicles) can perceive the aforementioned ground truth vehicle and collect data.

[0095] Scenario 2 is a real-world driving scenario used for model validation. This requires all vehicles involved in the perception process to be in normal driving condition and able to communicate with each other. In this scenario, the BSM message from the ground truth vehicle serves as the actual value of the perceived target (i.e., the true annotation for model training), while the BSM and SSM messages from all perceiving vehicles are used as model inputs for supervised training.

[0096] During data collection, each vehicle simulates actual driving conditions in an open area, senses each other, and records the BSM and SSM data sent and received, along with their timestamp information.

[0097] During training, the input data size of the temporal recurrent attention network model is the same as the input size during inference, which is batchsize * vsize * sequencelen * 9. The input data is format-converted and fed into the deep fusion network for forward propagation to obtain the predicted spatiotemporal state of each object in the next time step.

[0098] The deep learning-based fusion model architecture and training loss function are as follows:

[0099] like Figure 9 As shown, the vehicle performing the fusion operation needs to deploy a time-series recurrent attention network model on the ECU. Its architecture mainly consists of two parts: a recurrent neural network (RNN) suitable for processing sequential data, capable of capturing dynamic features in time series; and an attention mechanism, a technique that enables the model to focus on important parts of the input sequence when processing information. It is commonly used for sequence-to-sequence tasks. Combining RNN and attention mechanisms allows for the construction of a more powerful model for handling regression tasks with time-series characteristics.

[0100] The MAE loss function is mathematically defined as the average of the absolute values ​​of the errors of all samples. It is a loss function that measures the performance of a prediction model and is widely used in regression problems. It provides a measure of the difference between predicted and observed values. The specific formula is as follows:

[0101]

[0102] Where N is the total number of samples in the dataset, i is the sample index, and y is the number of samples in the dataset. i It is the real spatiotemporal information of the perceived object. It is the information of the perceived object predicted by the model, Loss mae This is a loss.

[0103] The absolute error loss is calculated by comparing the perceived prediction result with the position of surrounding real objects. Backpropagation is then used to update the model parameters, improving prediction accuracy. After obtaining the loss, backpropagation is performed to update the model F(θ) and iterative training is conducted. Multiple batches of training samples are sampled, with each batch containing a small amount of training data. The length of each batch is called the batchsize. By continuously adjusting the model parameters, the model fully learns the features of the dataset, gradually approaching its optimal state to improve performance and generalization ability. The above training steps are repeated for a pre-set number of training rounds until the model achieves good convergence.

[0104] This solution utilizes an end-to-end deep learning temporal recurrent attention network model, especially by combining recurrent neural networks and attention mechanisms, to more effectively process and fuse perception data from multiple vehicles, thereby obtaining more accurate target position, size, and motion state.

[0105] The real-time nature of this method ensures that the system can respond quickly to environmental changes. These advancements have promoted the development of vehicle-to-everything (V2X) and collaborative sensing technologies, laying the foundation for a wider range of intelligent transportation systems. The implementation of this method will help advance intelligent transportation technologies, providing a safer, more efficient, intelligent, and low-carbon solution for future transportation.

[0106] Secondly, the present invention provides a post-fusion device for multi-vehicle collaborative perception, comprising: a trajectory generation module and a fusion module.

[0107] The trajectory generation module is used to generate the first target trajectory and the trajectory of other vehicles based on the first target state information perceived by the vehicle itself and the state information of other vehicles received. The fusion module is used to, when there is a first target trajectory similar to the trajectory of other vehicles, correct all other first target state information based on the deviation value between the trajectory of other vehicles and the similar first target trajectory, and then fuse all second target state information perceived by other vehicles.

[0108] The fusion module is also used to directly fuse all first target state information, other vehicle's own state information, and second target state information when there is no first target trajectory similar to that of other vehicles.

[0109] It also includes a judgment module, which is used to determine whether there is a first target trajectory similar to the trajectory of another vehicle based on the spatial distance between each first target trajectory and the trajectory of another vehicle.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the functional implementation of each module in the above-mentioned multi-vehicle cooperative perception post-fusion device corresponds to each step in the above-mentioned multi-vehicle cooperative perception post-fusion method embodiment. The corresponding process in the aforementioned embodiment can be referred to, and its functions and implementation process will not be described in detail here.

[0111] Thirdly, the present invention also provides a computer device, characterized in that the computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the post-fusion method of multi-vehicle cooperative perception as described above.

[0112] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 10 It runs on the computer device shown.

[0113] Please see Figure 10 , Figure 10 This is a schematic block diagram illustrating the structure of a computer device provided in an embodiment of this application. The computer device can be a terminal.

[0114] like Figure 10 As shown, the computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0115] The non-volatile storage medium can store an operating system and a computer program. This computer program includes program instructions that, when executed, cause the processor to perform any post-fusion method for multi-vehicle cooperative perception.

[0116] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0117] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any post-fusion method of multi-vehicle cooperative perception.

[0118] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0119] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0120] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the method implemented when the program instructions are executed can refer to various embodiments of the post-fusion method for multi-vehicle cooperative perception of this application.

[0121] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0122] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0123] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. The above descriptions are merely specific implementations of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0124] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

Claims

1. A post-fusion method for multi-vehicle cooperative perception, characterized in that, Includes the following steps: Based on the first target state information sensed by the vehicle and the received state information of other vehicles, the trajectory of the first target and the trajectory of other vehicles are generated. When there is a first target trajectory similar to that of another vehicle, the state information of all other first targets is corrected based on the deviation between the trajectory of the other vehicle and the similar first target trajectory, and then fused with the state information of all second targets perceived by the other vehicle.

2. The post-fusion method for multi-vehicle cooperative perception as described in claim 1, characterized in that, After generating the first target trajectory and the trajectory of other vehicles, the method further includes: determining whether there is a first target trajectory similar to the trajectory of other vehicles based on the spatial distance between each first target trajectory and the trajectory of other vehicles.

3. The post-fusion method for multi-vehicle cooperative perception as described in claim 2, characterized in that, The method of determining whether there is a first target trajectory similar to the trajectory of another vehicle based on the spatial distance between each first target trajectory and the trajectory of another vehicle includes: For each first target trajectory and each other vehicle trajectory, the first maximum spatial distance value is obtained based on the minimum spatial distance from each trajectory point in the first target trajectory to the other vehicle trajectory, and the second maximum spatial distance value is obtained based on the minimum spatial distance from each trajectory point in the other vehicle trajectory to the first target trajectory. If both the first maximum spatial distance value and the second maximum spatial distance value are less than a set threshold, then the trajectory of the first target is determined to be similar to the trajectory of the corresponding other vehicle.

4. The post-fusion method for multi-vehicle cooperative perception as described in claim 1, characterized in that, The correction of the state information of all other first targets based on the deviation between the trajectory of other vehicles and the trajectory of similar first targets includes: Based on the trajectory points at sampling time t in the similar first target trajectory, find the trajectory points at two times adjacent to the sampling time t in the trajectory of the corresponding other vehicle; The state information of the trajectory point at time t in the trajectory of another vehicle is obtained by using the difference method or taking the average of the two trajectory points adjacent to the sampling time t. Based on the state deviation between the trajectory points at each sampling time t in the similar first target trajectory and the corresponding trajectory points at the corresponding time t in the trajectory of other vehicles, the state information of all other first targets is corrected.

5. The post-fusion method for multi-vehicle cooperative perception as described in claim 4, characterized in that, If multiple other vehicle trajectories have corresponding similar first target trajectories, then obtain the deviation value between each similar first target trajectory and the trajectory point at the same time in the corresponding other vehicle trajectory. The deviation value used for correction is the minimum, average, or weighted average of multiple deviation values.

6. The post-fusion method for multi-vehicle cooperative perception as described in claim 1, characterized in that, When there is no first target trajectory similar to that of other vehicles, the state information of all first targets, the state information of other vehicles, and the state information of second targets are directly fused.

7. The post-fusion method for multi-vehicle cooperative perception as described in any one of claims 1 to 6, characterized in that, A temporal recurrent attention network model is used for fusion, and the input data is normalized and time-aligned.

8. A post-fusion device for multi-vehicle collaborative perception, characterized in that, include: The trajectory generation module is used to generate the trajectory of the first target and the trajectory of the other vehicle based on the first target state information perceived by the vehicle and the received state information of the other vehicle. The fusion module is used to, when there is a first target trajectory similar to that of another vehicle, correct the state information of all other first targets based on the deviation value between the trajectory of the other vehicle and the similar first target trajectory, and then fuse it with the state information of all second targets perceived by the other vehicle.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the post-fusion method for multi-vehicle cooperative perception as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-vehicle cooperative perception post-fusion program, wherein when the multi-vehicle cooperative perception post-fusion program is executed by a processor, it implements the steps of the multi-vehicle cooperative perception post-fusion method as described in any one of claims 1 to 7.

Images