A fusion processing method and device based on an unmanned vehicle and a medium

Abstract

The application provides a fusion processing method and device based on an unmanned vehicle and a medium, and relates to the field of unmanned driving. The method comprises the following steps: recording the time when the first information of the main sensor is received as the first time, obtaining the track time stamp of the obstacle, the track information, the second time stamp corresponding to each auxiliary sensor and the second information according to the first time stamp in the first information; according to the obtained time, the information corresponding to the time is processed by a Kalman filtering model to obtain compensation information corresponding to the respective compensation time difference; according to the obtained compensation information, the first information, the second information and the track information are respectively compensated and modified; and the modified compensation result is subjected to data fusion according to the correlation between the obstacles and the correlation between the obstacles and the track. The fusion result obtained by the method of the application is closer to the environment in which the vehicle is located when making a planning decision, which helps to improve the performance and safety of the system.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a fusion processing method, device and medium based on autonomous vehicles. Background Technology

[0002] Autonomous vehicles are integrated systems encompassing environmental perception, planning and decision-making, and multi-level driver assistance. Environmental perception is a crucial step in this integrated system, providing vital reference information for further planning and decision-making. Environmental perception primarily involves acquiring fused information, which must accurately reflect the actual state of the environment at the time of planning and decision-making. Sensors fuse the collected information to obtain the required fused information; therefore, the timing of the environmental information acquired by the sensors becomes particularly important.

[0003] In existing technologies, multiple sensors installed on autonomous vehicles do not always trigger information collection simultaneously. To address this, a strategy combining a unified clock source and time soft synchronization is adopted to ensure that the information acquired by each sensor is as close in time as possible, and that the fused information after the fusion of the various sensors reflects the environmental state at the same moment as much as possible.

[0004] However, although the fusion information obtained by the above acquisition method adopts a strategy of combining a unified clock source and time soft synchronization to make the obtained fusion information reflect the environmental state at the same moment as much as possible, the information collected by the sensors still needs to be processed by data algorithms to obtain the fusion information. During the data algorithm processing, obstacles in the environment may be in motion and the autonomous vehicle may also be in motion. Therefore, there will be a certain time difference between the fusion information obtained by only processing with a unified clock source and time soft synchronization strategy and the real environment when making planning decisions. Consequently, the planning decisions based on this fusion information will also have a certain gap with the planning decisions required by the real environment. Summary of the Invention

[0005] This application provides a fusion processing method, device, and medium based on autonomous vehicles to solve the problem of a certain time difference between the fused information obtained through traditional technologies and the real environment when making planning decisions.

[0006] Firstly, this application provides a fusion processing method based on autonomous vehicles, comprising:

[0007] The time corresponding to the receipt of the first information acquired by the main sensor is recorded as the first time, and the first timestamp in the first information, the track timestamp of the obstacle stored in the unmanned vehicle, and the track information corresponding to the track timestamp are obtained.

[0008] Based on the first timestamp, perform soft time synchronization on each sub-sensor to obtain the second timestamp and the second information corresponding to the second timestamp for each sub-sensor;

[0009] Based on the first time, the first timestamp, the second timestamp, the track timestamp, and the preset compensation threshold, the first information, the second information, and the track information are predicted and processed by a preset Kalman filter model to obtain the compensation obstacle information, the compensation vehicle's own state information, and the compensation track information corresponding to the compensation time difference.

[0010] Based on the compensated obstacle information, the compensated vehicle's own status information, and the compensated trajectory information, information compensation and correction are performed on the first information, the second information, and the trajectory information respectively to obtain the obstacle information to be fused and the trajectory information to be fused.

[0011] Based on the relationship between obstacles and the relationship between obstacles and flight paths, data fusion of obstacle information and flight path information to be fused is achieved.

[0012] In one possible design, the first information, the second information, and the trajectory information are predicted and processed using a preset Kalman filter model based on the first time, the first timestamp, the second timestamp, the trajectory timestamp, and a preset compensation threshold to obtain the compensation obstacle information, the compensation vehicle's own state information, and the compensation trajectory information corresponding to the compensation time difference, including:

[0013] Based on the first time, the first timestamp, the second timestamp, the track timestamp, and the preset compensation threshold, the first compensation time difference corresponding to the first information, the second compensation time difference corresponding to the second information, and the track compensation time difference corresponding to the track information are obtained respectively.

[0014] Using a Kalman filter model, based on the first compensation time difference and the second compensation time difference, the first information corresponding to the first compensation time difference and the second information corresponding to the second compensation time difference are subjected to uniform motion prediction processing to obtain the compensation obstacle information and the compensation vehicle's own state information.

[0015] Using a Kalman filter model, the trajectory information is processed to predict uniform motion based on the trajectory compensation time difference, thereby obtaining compensated trajectory information.

[0016] In one possible design, the step of obtaining the first compensation time difference corresponding to the first information, the second compensation time difference corresponding to the second information, and the track compensation time difference corresponding to the track information based on the first time, the first timestamp, the second timestamp, the track timestamp, and a preset compensation threshold includes:

[0017] The difference between the first time and the first timestamp is summed with the preset compensation threshold to obtain the first compensation time difference;

[0018] The difference between the first time and the second timestamp is summed with the preset compensation threshold to obtain the second compensation time difference;

[0019] The difference between the first time and the track timestamp is summed with the preset compensation threshold to obtain the track compensation time difference.

[0020] In one possible design, before performing data fusion of obstacle information and trajectory information to be fused based on obstacle association and obstacle-track association, the method includes:

[0021] Based on the first timestamp, the second timestamp and the track timestamp corresponding to each sub-sensor obtained through time soft synchronization, the second information and the track information are updated to the first timestamp to obtain the obstacle information to be associated and the track information to be associated.

[0022] The obstacle information to be associated obtained by each secondary sensor and the primary sensor is correlated to obtain the obstacle association relationship;

[0023] The obstacle information and the flight track information to be associated are correlated to obtain the association relationship between the obstacle and the flight track.

[0024] In one possible design, the step of associating the obstacle information obtained by each secondary sensor and the primary sensor to obtain the obstacle association relationship includes:

[0025] Arbitrarily acquire the position, velocity, acceleration, and orientation of motion information from the first and second obstacle information to be associated from the corresponding secondary or primary sensor;

[0026] Based on the position, velocity, acceleration, and orientation of motion information in the first and second obstacle information to be associated, a similarity measurement algorithm is used to calculate and filter the first and second obstacle information to be associated that meet the preset similarity threshold.

[0027] A matching algorithm is used to obtain the obstacle association relationship between the first obstacle information to be associated and the second obstacle information to be associated.

[0028] In one possible design, the step of data association between the obstacle information to be associated and the trajectory information to be associated, to obtain the association relationship between the obstacle and the trajectory, includes:

[0029] Obtain the position, velocity, acceleration, and orientation of motion information from any of the obstacle information to be associated and any of the trajectory information to be associated;

[0030] Based on the position, velocity, acceleration, and orientation of motion information in the obstacle information and the trajectory information to be associated, a similarity measurement algorithm is used to calculate and filter the obstacle information and trajectory information to be associated that meet the preset similarity threshold.

[0031] A matching algorithm is used to obtain the association relationship between the obstacle information and the trajectory information to be associated.

[0032] In one possible design, based on the compensated obstacle information, the compensated vehicle's own state information, and the compensated trajectory information, information compensation is performed on the first information, the second information, and the trajectory information respectively to obtain the obstacle information and the trajectory information to be fused, including:

[0033] The information changes of obstacles in the compensated obstacle information are respectively compensated into the corresponding obstacle information in the first information and the second information, and the compensated obstacle information is corrected according to the information changes in the compensated vehicle's own state information to obtain the obstacle information to be fused.

[0034] The changes in the track information in the compensated track information are compensated into the track information, and the compensated track information is corrected according to the changes in the information in the compensated vehicle's own state information to obtain the track information to be fused.

[0035] In one possible design, the step of performing time soft synchronization on each sub-sensor based on the first timestamp to obtain a second timestamp corresponding to each sub-sensor and second information corresponding to the second timestamp includes:

[0036] Based on the first timestamp, obtain the second timestamp closest to the first timestamp and the second information corresponding to the second timestamp in the buffer queue corresponding to each sub-sensor identifier.

[0037] In one possible design, the data fusion of obstacle information and trajectory information to be fused is performed based on obstacle association relationships and obstacle-track association relationships, respectively, including:

[0038] Based on the preset weights of each sensor, the obstacle information to be associated is weighted and fused according to the obstacle association relationship to obtain the measured obstacle information;

[0039] Based on the preset weights of the measured obstacle information and the trajectory information to be associated, the measured obstacle information containing the obstacle information to be associated and the trajectory information to be associated are weighted and fused according to the association relationship between the obstacle and the trajectory to obtain the optimal obstacle information.

[0040] Secondly, this application provides a fusion device, comprising:

[0041] The information acquisition module records the time corresponding to the first information acquired by the main sensor as the first time, and acquires the first timestamp in the first information, the track timestamp of the obstacle stored in the unmanned vehicle, and the track information corresponding to the track timestamp.

[0042] The time soft synchronization module performs time soft synchronization on each sub-sensor based on the first timestamp to obtain the second timestamp and the second information corresponding to the second timestamp for each sub-sensor.

[0043] The data prediction and processing module, based on the first time, the first timestamp, the second timestamp, the track timestamp, and the preset compensation threshold, performs prediction processing on the first information, the second information, and the track information respectively through a preset Kalman filter model to obtain the compensation obstacle information, the compensation vehicle's own state information, and the compensation track information corresponding to the compensation time difference;

[0044] The information compensation and correction module performs information compensation and correction on the first information, the second information, and the trajectory information based on the compensation obstacle information, the compensation vehicle's own status information, and the compensation trajectory information, respectively, to obtain the obstacle information to be fused and the trajectory information to be fused.

[0045] The association and fusion module, based on the relationship between obstacles and the relationship between obstacles and flight paths, realizes the data fusion of obstacle information and flight path information to be fused.

[0046] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0047] The memory stores computer-executed instructions;

[0048] The processor executes the computer execution instructions stored in the memory to implement the fusion processing method based on unmanned vehicles described in the above technical solution.

[0049] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the fusion processing method based on unmanned vehicles described in the above technical solution.

[0050] This application provides a fusion processing method, apparatus, and medium based on autonomous vehicles. The method compensates for obstacle information, vehicle state information, and trajectory information generated by algorithm latency by applying these to the first, second, and trajectory information, respectively. This results in obstacle information and trajectory information that more closely reflect the surrounding environment at the time the autonomous vehicle makes its planning decisions. Data fusion is achieved based on these two information, reducing the impact of obstacle detection errors on the final obstacle information. This improves data accuracy and reliability, enhances obstacle perception and tracking capabilities, and further optimizes subsequent decision-making algorithms. Ultimately, this contributes to improved system performance and safety, leading to better results in practical applications such as autonomous driving and intelligent transportation. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of data processing based on fusion processing methods for autonomous vehicles in existing technologies.

[0053] Figure 2 A schematic flowchart of a fusion processing method based on an autonomous vehicle provided in this application embodiment;

[0054] Figure 3 This is a schematic diagram of a method provided in this application for predicting and processing first information, second information, and track information respectively using a preset Kalman filter model;

[0055] Figure 4 This is a schematic diagram of a method for obtaining obstacle information and trajectory information to be fused, provided in an embodiment of this application.

[0056] Figure 5 This is a schematic diagram of the method for obtaining obstacle association relationships and obstacle-track association relationships provided in the embodiments of this application;

[0057] Figure 6 This is a schematic diagram illustrating a method for obtaining obstacle association relationships provided in an embodiment of this application;

[0058] Figure 7 A schematic diagram illustrating a method for obtaining the correlation between obstacles and flight paths provided in an embodiment of this application;

[0059] Figure 8 This is a schematic diagram of a method for fusing obstacle information and trajectory information data to be fused, provided in an embodiment of this application.

[0060] Figure 9 This is a schematic diagram of the fusion device provided in the embodiments of this application;

[0061] Figure 10 This is a schematic flowchart of a method for calculating time difference using an unmanned commercial vehicle as an example, provided in an embodiment of this application.

[0062] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0064] First, the relevant concepts or terms involved in this application will be explained:

[0065] Sensor fusion: Sensor fusion is the process of combining and analyzing data and information acquired from multiple sensors to more accurately and reliably describe the external environment, thereby improving the correctness of system decisions.

[0066] Kalman filtering (KF) is an algorithm that uses the state equations of a linear system to make an optimal estimate of the system state using the system's input and output observation data.

[0067] For example, Figure 1 This is a schematic diagram illustrating data processing in existing technologies based on autonomous vehicle fusion processing methods. For example... Figure 1As shown, the sensors used to acquire the above-mentioned processed data mainly include: camera 101, lidar 102, and millimeter-wave radar 103. Specifically, camera 101, lidar 102, and millimeter-wave radar 103 serve as data acquisition sensors to collect raw environmental data of the environment in which the autonomous vehicle is located. The acquired raw data is then processed by camera perception algorithms, lidar perception algorithms, and millimeter-wave radar perception algorithms, respectively. The identification results are then summarized and fused. Based on the fused information, planning and control are performed to make driving planning decisions for the autonomous vehicle.

[0068] The time corresponding to the fused information used for planning and control at this point is the time it takes for camera 101, lidar 102, and millimeter-wave radar 103 to acquire information. However, the perception algorithm incurs time consumption when identifying information from the sensors and summarizing and fusing the identified results. Since obstacles in the external environment may be in motion, the external environment may have changed by the time the algorithm consumes. There will be a certain time difference between the fused information obtained and the actual environment at the time of planning and control, and consequently, the planning decision based on this fused information will also differ from the planning decision required by the actual environment.

[0069] To address the aforementioned technical problems, this application provides a fusion processing method based on autonomous vehicles. This method compensates for environmental changes corresponding to the time consumed by the algorithm by incorporating them into the original information acquired by the sensors. Furthermore, during the time the algorithm consumes, the autonomous vehicle itself is also in motion, and its own motion affects the perception of the external environment. Therefore, the compensated original information is corrected based on the information regarding the change in the vehicle's own motion state corresponding to the algorithm's consumption time. Data is then aggregated and fused based on the corrected original information, resulting in fused information that more closely reflects the external environment at the time of planning and decision-making.

[0070] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0071] Figure 2 This application provides a schematic flowchart of a fusion processing method based on autonomous vehicles. (See attached diagram.) Figure 2 As shown, the method includes steps S201-S205:

[0072] S201, record the time corresponding to the first information acquired by the main sensor as the first time, and acquire the first timestamp in the first information, the track timestamp of the obstacle stored in the unmanned vehicle, and the track information corresponding to the track timestamp.

[0073] In this embodiment, when the first information collected by the main sensor is received, the time point of receipt of the information is first recorded as the first time. Then, the information is parsed to extract the first timestamp. The first timestamp is the acquisition time corresponding to the main sensor collecting the first information. The existing obstacle track timestamps and corresponding track information are then obtained from the autonomous vehicle's storage system. Specifically, the obstacle track timestamps and corresponding track information are the latest (acquired and updated in the previous frame) obstacle track timestamps and corresponding track information stored in the autonomous vehicle. Thus, not only is the first time recorded and the first timestamp obtained, but also the obstacle track timestamps and corresponding track information are acquired. This information will be used for subsequent processing and analysis, providing important input data for the autonomous vehicle's decision-making and control.

[0074] S202, perform time soft synchronization on each sub-sensor according to the first timestamp to obtain the second timestamp and the second information corresponding to the second timestamp for each sub-sensor.

[0075] In this embodiment, each sub-sensor undergoes soft time synchronization based on the first timestamp of the first information from the main sensor. This means that the data from each selected sub-sensor will be adjusted according to the timestamp of the main sensor, ensuring that the data acquisition time of the sub-sensors is as synchronized as possible with the data acquisition time of the main sensor. Through the soft time synchronization operation, a second timestamp corresponding to each sub-sensor can be obtained. Furthermore, the second information corresponding to the second timestamp, i.e., the data acquired by the sub-sensors at the second timestamp, can also be obtained.

[0076] S203, based on the first time, first timestamp, second timestamp, track timestamp, and preset compensation threshold, the first information, second information, and track information are predicted and processed by a preset Kalman filter model to obtain the compensation obstacle information, compensation vehicle state information, and compensation track information corresponding to the compensation time difference.

[0077] In this embodiment, the Kalman filter model can represent the target's motion model as a set of linear equations, and the Kalman filter is used to estimate and predict the target's position. That is, through the predictive processing of the Kalman filter model, the future state can be predicted based on existing information, thereby performing data compensation. Based on the first time, first timestamp, second timestamp, track timestamp, and a preset compensation threshold, a preset Kalman filter model is used to predict the first information, the second information, and the track information respectively. This yields the compensated obstacle information, compensated vehicle state information, and compensated track information corresponding to the compensated time difference.

[0078] S204, based on the compensation obstacle information, the compensation vehicle's own status information and the compensation trajectory information, information compensation and correction are performed on the first information, the second information and the trajectory information respectively to obtain the obstacle information to be fused and the trajectory information to be fused.

[0079] In this embodiment, information compensation and correction are performed on the first information, the second information, and the trajectory information using compensated obstacle information, compensated vehicle state information, and compensated trajectory information. Through this compensation and correction process, the original data (first information, second information, and trajectory information) can be corrected and adjusted to more accurately reflect the actual situation. Specifically, based on the compensated obstacle information, obstacle data in the first and second information can be compensated. Simultaneously, based on the compensated vehicle state information, the already compensated obstacle data in the first and second information can be corrected. Furthermore, the trajectory information can be compensated and corrected. The corrected obstacle information and trajectory information to be fused will serve as inputs for subsequent data fusion.

[0080] S205, based on the relationship between obstacles and the relationship between obstacles and flight paths, achieve data fusion of obstacle information and flight path information to be fused.

[0081] In this embodiment, data fusion is performed based on obstacle associations and obstacle-track associations. By fusing obstacle information and track information to be fused, final obstacle and track information can be obtained. The data fusion process integrates and merges data collected by different sensors to improve data accuracy and reliability. Obstacle information from different sensors is associated and fused based on obstacle associations. Simultaneously, obstacle information from different sensors is associated and fused with existing track information based on obstacle-track associations. The resulting fused obstacle and track information can provide more accurate input data for the decision-making and control of autonomous vehicles.

[0082] In this embodiment, by compensating for obstacle information, vehicle state information, and trajectory information generated during algorithm execution, information compensation and correction are applied to the first information, second information, and trajectory information, respectively. This results in obstacle information and trajectory information to be fused that more closely reflect the surrounding environment at the moment the autonomous vehicle makes its planning decisions. Information data fusion is achieved based on these two pieces of information, reducing the impact of obstacle detection errors on the final obstacle information. This improves the accuracy and reliability of the data, enhances the perception and tracking capabilities of obstacles, and further optimizes subsequent decision-making algorithms. Based on this, it helps improve the system's performance and safety, bringing better results to fields such as autonomous driving and intelligent transportation in practical applications.

[0083] It should be further explained that the sensors (main sensors and secondary sensors) include sensors installed on the autonomous vehicle for perceiving the environment (such as cameras, LiDAR, millimeter-wave radar), as well as sensors for detecting the vehicle's own status information (such as integrated navigation). The purpose of setting the main sensor is to trigger time soft synchronization. Those skilled in the art can select the appropriate sensor as the main sensor according to their needs or experience. In this case, the other sensors are secondary sensors.

[0084] In one specific embodiment, the time soft synchronization in step S202 specifically includes step S2021:

[0085] S2021: Based on the first timestamp, obtain the second timestamp that is closest to the first timestamp in the buffer queue corresponding to each sub-sensor identifier, and the second information corresponding to the second timestamp.

[0086] In this embodiment, soft time synchronization utilizes timestamps for matching between different sensors. Although a unified clock source is set to synchronize the time of each sensor, meaning the timestamps corresponding to the sensor's acquired information are assigned by the unified clock source, the timestamps differ due to the different acquisition frequencies of the sensors. The sensor-acquired information is first buffered, with a queue of a pre-defined size. When the first information acquired by the main sensor is received, the second information corresponding to the second timestamp closest to the first timestamp in the buffer queue corresponding to each secondary sensor identifier is retrieved based on the first timestamp in the first information.

[0087] Specifically, this will be further explained using one main sensor and three secondary sensors, where the integrated navigation system used to detect vehicle status information consists of secondary sensors. The time to receive the first information from the main sensor is denoted as t. current The first timestamp corresponding to the first information acquired by the main sensor is t. activateThe track timestamp obtained based on the first timestamp is t. track Comparison with t activate The second timestamp is obtained by finding the closest timestamp from the buffer queue of each sub-sensor, and then t. vice_1_n (n is the queue size), t vice_2_n t gnss Among them, t vice_1_n For the timestamp corresponding to sub-sensor No. 1, t vice_2_n For the timestamp corresponding to sub-sensor No. 2, t gnss This is the timestamp corresponding to the combined navigation.

[0088] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a method for predicting first information, second information, and track information using a preset Kalman filter model, as provided in this application embodiment. This embodiment further details step S203, specifically including steps S301-S303:

[0089] S301, based on the first time, first timestamp, second timestamp, track timestamp, and preset compensation threshold, obtain the first compensation time difference corresponding to the first information, the second compensation time difference corresponding to the second information, and the track compensation time difference corresponding to the track information.

[0090] In this embodiment, the time difference that needs to be compensated can be determined by comparing the difference between different timestamps and a preset compensation threshold. The first compensation time difference represents the time delay of the first information relative to the first timestamp, the second compensation time difference represents the time delay of the second information relative to the second timestamp, and the track compensation time difference represents the time delay of the track information relative to the track timestamp.

[0091] In one specific embodiment, how the time difference is obtained in step S301 will be further explained here, specifically including:

[0092] The first compensation time difference is obtained by subtracting the first time from the first timestamp and then adding it to a preset compensation threshold. This will be further explained using the example of one main sensor and three sub-sensors given above. Specifically, the first compensation time difference is expressed by the formula: δ fusion_activate =t current -t activate +δ fusion , where δ fusion_activate For the first compensation time difference, δ fusion This is the preset compensation threshold.

[0093] The second compensation time difference is obtained by subtracting the first and second timestamps and then summing the difference with a preset compensation threshold. Specifically, the second compensation time difference for sub-sensor 1 is expressed by the formula: δ fusion_vice _1=t current -t vice_1_n +δ fusion , where δ fusion_vice _1 represents the second compensation time difference of the first auxiliary sensor, t vice 1n This is the second timestamp for sub-sensor 1. For the second compensation time of sub-sensor 2...

[0094] The difference is expressed by the formula: δ fusion_vice _2=t current -t vice_2_n +δ fusion , where δ fusion_vice _2 represents the second compensation time difference of the second auxiliary sensor, t vice 2n This is the second timestamp of the second auxiliary sensor. The second compensation time difference corresponding to the integrated navigation is expressed by the formula: δ fusion_gnss =t current -t gnss +δ fusion , where δ fusion_gnss For the second compensation time difference corresponding to integrated navigation, t gnss This is the second timestamp corresponding to the combined navigation.

[0095] The time difference between the first time and the track timestamp is then summed with a preset compensation threshold to obtain the track compensation time difference. Specifically, this is expressed by the formula: δ fusion track =t current -t track +δ fusion , where δ fusion track To compensate for the time difference in the flight path, t track This is the track timestamp.

[0096] S302 uses a Kalman filter model to perform uniform motion prediction processing on the first information corresponding to the first compensation time difference and the second information corresponding to the second compensation time difference, based on the first compensation time difference and the second compensation time difference, to obtain the compensation obstacle information and the compensation vehicle's own state information.

[0097] In this embodiment, a Kalman filter model is employed, using a first compensation time difference and a second compensation time difference to predict the first information corresponding to the first compensation time difference and the second information corresponding to the second compensation time difference. This allows for the estimation of the future states of the first information corresponding to the first compensation time difference and the second information corresponding to the second compensation time difference. Thus, the information about the compensated obstacle and the compensated vehicle's own state are obtained.

[0098] S303 employs a Kalman filter model to perform uniform motion prediction processing on the trajectory information based on the trajectory compensation time difference, thereby obtaining compensated trajectory information.

[0099] In this embodiment, a Kalman filter model is used to predict the trajectory information based on the trajectory compensation time difference. The state of the trajectory information corresponding to the trajectory compensation time difference at future times can be estimated, thus obtaining the compensated trajectory information.

[0100] Since the algorithm consumes relatively little time, meaning the time difference to be compensated is relatively short, a Kalman filter model is used for predicting uniform motion. This yields information on compensated obstacles, compensated vehicle state, and compensated trajectory, providing more accurate data input for subsequent information compensation correction and data fusion.

[0101] In one embodiment, Figure 4 This is a schematic diagram illustrating a method for obtaining obstacle information and trajectory information to be fused, as provided in an embodiment of this application. Figure 4 The diagram shows a further detailed explanation of step S204, including steps S401-S402:

[0102] S401, the information change of the obstacle in the compensation obstacle information is compensated to the corresponding obstacle information in the first information and the second information respectively, and the compensated obstacle information is corrected according to the information change in the compensation vehicle's own state information to obtain the obstacle information to be fused.

[0103] In this embodiment, based on the changes in the information in the compensated obstacle information, these changes are compensated into the corresponding obstacle information in the first and second information, such as the obstacle's position, velocity, and acceleration, thus achieving obstacle information compensation. Next, the compensated obstacle information is corrected using the changes in the compensated vehicle's own state information. Based on changes in the compensated vehicle's own state information, such as vehicle position, velocity, and acceleration, the obstacle position, velocity, and acceleration information in the compensated obstacle information are adjusted to account for the influence of the vehicle's own motion state on the perceived obstacles. Finally, the obstacle information to be fused is obtained, which includes the compensated and corrected obstacle position, velocity, and acceleration information, preparing for subsequent data fusion.

[0104] S402, the changes in track information in the compensation track information are added to the track information, and the compensated track information is corrected according to the changes in information in the compensation vehicle's own state information to obtain the track information to be fused.

[0105] In this embodiment, based on the changes in information in the compensated trajectory information, these changes are compensated into the trajectory information, such as the position, velocity, and acceleration of obstacles represented in the trajectory information, thereby compensating the trajectory information. The compensated trajectory information is then corrected using changes in information in the compensated vehicle's own state information. Based on changes in the vehicle's position, velocity, and acceleration information in the compensated vehicle's own state information, the position, velocity, and acceleration information in the compensated trajectory information are adjusted to account for the influence of the vehicle's own motion state on the trajectory information. The resulting trajectory information to be fused includes the compensated and corrected information on the position, velocity, and acceleration of obstacles represented by the trajectory, preparing for subsequent data fusion.

[0106] In one embodiment, Figure 5 This is a schematic diagram illustrating the method for obtaining obstacle association relationships and obstacle-track association relationships provided in an embodiment of this application. For example... Figure 5 As shown, before step S205, it is necessary to obtain the obstacle association relationship and the obstacle-track association relationship, including steps S501-S503:

[0107] S501, based on the first timestamp, the second timestamp and track timestamp corresponding to each sub-sensor obtained through time soft synchronization, update the second information and track information to the first timestamp to obtain the obstacle information to be associated and the track information to be associated.

[0108] In this embodiment, the second information and track information acquired by different sub-sensors are aligned with the first timestamp, ensuring they are at the same timestamp. Based on the aligned first timestamp, the second information acquired by each sub-sensor is updated, and the updated second information corresponds to the first timestamp. This also involves information prediction based on the time difference between the first and second timestamps, thereby compensating the predicted information difference into the original second information and updating it. The same Kalman filter model is used for prediction and update processing as described above. The obtained obstacle information and track information to be associated prepare data for subsequent data association. Associating information at the same timestamp further improves the accuracy of data association and enhances the matching degree between data.

[0109] S502, perform data association on the obstacle information to be associated obtained from each secondary sensor and the main sensor to obtain the obstacle association relationship.

[0110] In one specific embodiment, Figure 6 This is a schematic diagram illustrating a method for obtaining obstacle association relationships provided in an embodiment of this application. Figure 6 As shown, steps S601-S603 are included:

[0111] S601, arbitrarily acquire the position, velocity, acceleration and motion orientation information from the first and second obstacle information to be associated from the corresponding secondary sensor or main sensor;

[0112] S602, based on the position, velocity, acceleration and orientation of motion information in the first obstacle information and the second obstacle information to be associated, a similarity measurement algorithm is used to calculate and filter the first obstacle information and the second obstacle information to be associated that meet the preset similarity threshold.

[0113] S603, using a matching algorithm, obtain the obstacle association relationship between the first obstacle information to be associated and the second obstacle information to be associated.

[0114] In this embodiment, the sensor can collect data from the surrounding environment. Different sensors (including the sensor and sub-sensors) may use different technologies to acquire obstacle information, meaning different types of sensors can acquire different information representations of the same obstacle. Therefore, it is necessary to associate the same obstacle information from different sensors, and then fuse this association to form more complete and accurate obstacle information. Specifically, first and second obstacle information to be associated, corresponding to different sensors, are acquired, where the information includes, but is not limited to, position, velocity, acceleration, and orientation. Then, a similarity measurement algorithm (e.g., Euclidean distance algorithm) is used to calculate the similarity between the first and second obstacle information to be associated. A preset similarity threshold is used to filter the calculated similarity to obtain the first and second obstacle information to be associated that meet the conditions. Specifically, if the similarity between the obstacles to be associated is not lower than the preset similarity threshold, then the two obstacle information from different sensors are considered to have a matching possibility; otherwise, they do not have a matching possibility. Those skilled in the art can flexibly set the similarity threshold according to the requirements for data accuracy. Furthermore, a matching algorithm (such as the Hungarian matching algorithm or the local nearest neighbor algorithm) is used to obtain the obstacle association relationship between the first obstacle information and the second obstacle information.

[0115] S503, perform data association between the obstacle information to be associated and the track information to be associated, and obtain the association relationship between the obstacle and the track.

[0116] In one specific embodiment, Figure 7This is a schematic diagram illustrating a method for obtaining the correlation between obstacles and flight paths, provided in an embodiment of this application. Figure 7 As shown, steps S701-S703 are included:

[0117] S701, acquire the position, velocity, acceleration and orientation of motion information from any obstacle information to be associated and any track information to be associated;

[0118] S702, based on the position, velocity, acceleration and orientation of motion information in the obstacle information and trajectory information to be associated, a similarity measurement algorithm is used to calculate and filter the obstacle information and trajectory information to be associated that meet the preset similarity threshold;

[0119] S703 uses a matching algorithm to obtain the association relationship between obstacles and tracks, including information on obstacles to be associated and track information to be associated.

[0120] In this embodiment, similar to the data association between obstacles in different sensors mentioned above, obstacle information is associated with track information. Any obstacle information and any track information to be associated (e.g., position, velocity, acceleration, and orientation) are acquired. A similarity measurement algorithm is used to calculate and filter obstacle and track information that meet a preset similarity threshold. If the similarity between the obstacle and track is not lower than the preset similarity threshold, the obstacle and track information are considered to have a matching possibility; otherwise, they are not considered to have a matching possibility. Next, a matching algorithm is used to obtain the association relationship between the obstacle and track information, providing accurate input for subsequent data fusion and target tracking.

[0121] In one embodiment, Figure 8 This is a schematic diagram illustrating a method for fusing obstacle information and trajectory information data to be fused, as provided in an embodiment of this application. Figure 8 The diagram shows a further explanation of step S205, including steps S801-S802:

[0122] S801, based on the preset weights of each sensor, performs weighted fusion of information on the obstacles to be associated according to the obstacle association relationship, and obtains the measured obstacle information.

[0123] In this embodiment, each sensor (including main and secondary sensors) is assigned a weight based on its importance or reliability. The weight allocation can be determined based on factors such as sensor performance indicators, accuracy, and reliability. According to obstacle correlations and considering the weight of each sensor, obstacle information acquired by different sensors is weighted and fused to obtain measured obstacle information. Taking sensor weights into account during the fusion process allows for better utilization of data from sensors with higher reliability. This improves data reliability and makes the final measured obstacle information more credible.

[0124] S802, based on the preset weights of the measured obstacle information and the trajectory information to be associated, performs information weighted fusion on the measured obstacle information and the trajectory information to be associated, which contain the obstacle information to be associated, according to the relationship between the obstacle and the trajectory, to obtain the optimal obstacle information.

[0125] In this embodiment, weights are assigned to each information source based on the weights of the measured obstacle information and the trajectory information to be associated. The weight assignment can be determined based on factors such as the importance and accuracy of the obstacles and trajectories. Based on the correlation between obstacles and trajectories, and considering the weights of the corresponding information sources, the measured obstacle information and the trajectory information to be associated are weighted and fused to obtain the optimal obstacle information. This includes a comprehensive result of the measured obstacle information and the trajectory information to be associated, providing higher quality and more accurate obstacle data for subsequent processing and decision-making.

[0126] The optimal obstacle information, after data fusion, is published for use by the planning and control module. This module is responsible for formulating optimal driving and control strategies (such as path planning, obstacle avoidance decisions, speed control, and collision warnings) based on current environmental information and system requirements to ensure safe and efficient vehicle operation. The fused optimal obstacle information is a crucial input to the planning and control module, providing key information such as the position, speed, and acceleration of other vehicles, pedestrians, and other obstacles on the road. Through this information fusion, the planning and control module obtains more accurate and comprehensive environmental perception information, enabling it to formulate more reliable and efficient planning and control strategies, and ensuring safe and smooth vehicle operation.

[0127] like Figure 10 As shown, Figure 10 This is a schematic diagram of the method for calculating time difference using an unmanned commercial vehicle as an example, provided in an embodiment of this application. Here, specific examples are given to further explain in detail the update time difference involved in data association and the compensation time difference involved in information fusion.

[0128] Taking an autonomous commercial vehicle as an example, this paper will provide a detailed introduction to the perception information of sensors including: integrated navigation 100, camera 101, lidar 102, and millimeter-wave radar 103.

[0129] The LiDAR 102 is designated as the primary sensor, while the integrated navigation system 100, camera 101, and millimeter-wave radar 103 are designated as secondary sensors. The ROS time is set as the unified clock source to provide time synchronization for the integrated navigation system 100, camera 101, LiDAR 102, and millimeter-wave radar 103, assigning the time of the acquired information to each sensor. The perception module processes and publishes information such as vehicle status, images, point clouds, and CAN messages collected by the integrated navigation system 100, camera 101, LiDAR 102, and millimeter-wave radar 103. Processing of information collected by the perception module from the camera 101 includes, but is not limited to, using computer vision technology to analyze the images and extract key information; processing of data collected by the LiDAR 102 includes, but is not limited to, analyzing the reflection points of each laser beam and calculating the position and shape of objects; processing of information collected by the millimeter-wave radar 103 includes, but is not limited to, removing noise and stray signals, analyzing the processed data, and accurately calculating the position and motion information of obstacles. Since this application does not involve improvements or optimizations to the sensor data acquisition and processing section of the sensing module, existing sensing module data processing techniques can be used, and will not be elaborated here.

[0130] After information processing by the perception module, the perception information from the integrated navigation 100, camera 101, lidar 102, and millimeter-wave radar 103 is obtained. This information will be received by the fusion device and fused.

[0131] The lidar 102 is equipped with a buffer queue of size 1, and the integrated navigation 100, camera 101, and millimeter-wave radar 103 are all equipped with buffer queues of size 3; the sensing information of each sensor enters the buffer queue until the fusion device receives the sensing information of lidar 102.

[0132] Assume the timestamp of the information collected by the lidar is t. lidar Let t be the ROS time at which the sensing information from the lidar 102 is received. current Compare this to t at this time. lidar It sequentially searches the buffer queues of the integrated navigation system 100, camera 101, and millimeter-wave radar 103 for the closest timestamp t. gnss_z (z=12,3),t camera_m (m=1,2,3),t radar_n (n = 1, 2, 3), and obtain the latest timestamp t of the stored track in the autonomous commercial vehicle. track .

[0133] The required update time difference during data association involves: firstly, aligning the timestamps of the information acquired by the secondary sensor with those acquired by the primary sensor, and then updating the corresponding sensing information of the secondary sensor. Specifically, camera 101 requires an update time difference δ. lidar_camera =t lidar -t camera_m The millimeter-wave radar 103 requires a time difference δ for updates. lidar_radar =t lidar -t radar_n ; for the time difference δ that needs to be updated for the track lidar_track =t lidar -t track .

[0134] Compensation time difference required for information fusion: Assume the average algorithm time of the fusion device is δ. fusion The time difference δ required for the integrated navigation system 100 fusion_gnss =t current -t gnss_z +δ fusion Camera 101, required compensation time difference δ fusion_camera =t current -t camera_m +δ fusion The compensation time difference δ required for LiDAR 102 fusion_lida r = t current -t lidar +δ fusion The millimeter-wave radar 103 requires compensation for the time difference δ. fusion_radar =t current -t radar_n +δ fusion The flight path needs to be compensated for the time difference δ fusion_track =t current -t track +δ fusion .

[0135] A Kalman filter prediction model is constructed to predict the obstacle's motion as uniformly accelerated. The first motion prediction is performed by combining obstacle state information from track information, camera perception information, and millimeter-wave radar perception information with the update time difference to predict obstacle motion and track. The information change in the obtained update compensation information is added to the perception information of the corresponding sensor to obtain the information required for data association.

[0136] The second motion prediction: Based on the time difference compensation, the trajectory information, combined navigation perception information, camera perception information, lidar perception information, and millimeter-wave radar perception information are used to predict obstacle motion and trajectory. The information change in the obtained compensation information is added to the perception information of the corresponding sensor. The difference between the added information and the vehicle's own state information change is calculated to obtain the information data required for information fusion.

[0137] In this embodiment of the invention, electronic devices or main control devices can be divided into functional modules according to the above method examples. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional module. It should be noted that the module division in this embodiment of the invention is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0138] Figure 9 This is a schematic diagram of the fusion device provided in an embodiment of this application. Figure 9 As shown, the fusion device 90 includes:

[0139] The information acquisition module 901 is used to record the time corresponding to the first information acquired by the main sensor as the first time, and to acquire the first timestamp in the first information, the track timestamp of the obstacle stored in the unmanned vehicle, and the track information corresponding to the track timestamp.

[0140] The time soft synchronization module 902 is used to perform time soft synchronization on each sub-sensor according to the first timestamp to obtain the second timestamp and the second information corresponding to the second timestamp for each sub-sensor.

[0141] The data prediction and processing module 903 is used to predict and process the first information, the second information and the trajectory information respectively through a preset Kalman filter model based on the first time, the first timestamp, the second timestamp, the trajectory timestamp and the preset compensation threshold, so as to obtain the compensation obstacle information, the compensation vehicle's own state information and the compensation trajectory information corresponding to the compensation time difference.

[0142] The information compensation and correction module 904 is used to perform information compensation and correction on the first information, the second information and the trajectory information according to the compensation obstacle information, the compensation vehicle's own status information and the compensation trajectory information, respectively, to obtain the obstacle information to be fused and the trajectory information to be fused.

[0143] The association fusion module 905 is used to achieve data fusion of obstacle information and trajectory information to be fused based on obstacle association and obstacle-track association.

[0144] The fusion device provided in this embodiment can execute the fusion processing method of the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0145] In the aforementioned specific implementation of the gear control device based on heavy-duty vehicles, each module can be implemented as a processor. The processor can execute computer execution instructions stored in the memory, so that the processor executes the aforementioned gear control method based on heavy-duty vehicles.

[0146] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 11 includes at least one processor 111 and a memory 112. The electronic device 11 also includes a communication component 113. The processor 111, the memory 112, and the communication component 113 are connected via a bus 114.

[0147] In the specific implementation process, at least one processor 111 executes computer execution instructions stored in memory 112, causing at least one processor 111 to execute the fusion processing method executed on the electronic device side as described above.

[0148] The specific implementation process of processor 111 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0149] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0150] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage.

[0151] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0152] The above description of the functions implemented by electronic devices and main control devices has introduced the solutions provided by the embodiments of the present invention. It is understood that, in order to implement the above functions, the electronic device or main control device includes hardware structures and / or software modules corresponding to the execution of each function. By combining the units and algorithm steps of the various examples described in the embodiments of the present invention, the embodiments of the present invention can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present invention.

[0153] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the method described in the above embodiments.

[0154] The aforementioned computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0155] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in an electronic device or a host device.

[0156] This application also provides a computer program product, comprising: a computer program stored in a readable storage medium, wherein at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to perform the scheme provided in any of the above embodiments.

[0157] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fusion processing method based on autonomous vehicles, characterized in that, include: The time corresponding to the receipt of the first information acquired by the main sensor is recorded as the first time, and the first timestamp in the first information, the track timestamp of the obstacle stored in the unmanned vehicle, and the track information corresponding to the track timestamp are obtained. Based on the first timestamp, perform soft time synchronization on each sub-sensor to obtain the second timestamp and the second information corresponding to the second timestamp for each sub-sensor; Based on the first time, the first timestamp, the second timestamp, the track timestamp, and the preset compensation threshold, the first information, the second information, and the track information are predicted and processed by a preset Kalman filter model to obtain the compensation obstacle information, the compensation vehicle's own state information, and the compensation track information corresponding to the compensation time difference. Based on the compensated obstacle information, the compensated vehicle's own status information, and the compensated trajectory information, information compensation and correction are performed on the first information, the second information, and the trajectory information respectively to obtain the obstacle information to be fused and the trajectory information to be fused. Based on the relationship between obstacles and the relationship between obstacles and flight paths, data fusion of obstacle information and flight path information to be fused is achieved.

2. The method according to claim 1, characterized in that, The process involves using a preset Kalman filter model to predict the first information, the second information, and the trajectory information based on the first time, the first timestamp, the second timestamp, the trajectory timestamp, and a preset compensation threshold, to obtain compensation obstacle information, compensation vehicle state information, and compensation trajectory information corresponding to the compensation time difference. This includes: Based on the first time, the first timestamp, the second timestamp, the track timestamp, and the preset compensation threshold, the first compensation time difference corresponding to the first information, the second compensation time difference corresponding to the second information, and the track compensation time difference corresponding to the track information are obtained respectively. Using a Kalman filter model, based on the first compensation time difference and the second compensation time difference, the first information corresponding to the first compensation time difference and the second information corresponding to the second compensation time difference are subjected to uniform motion prediction processing to obtain the compensation obstacle information and the compensation vehicle's own state information. Using a Kalman filter model, the trajectory information is processed to predict uniform motion based on the trajectory compensation time difference, thereby obtaining compensated trajectory information.

3. The method according to claim 2, characterized in that, The step of obtaining the first compensation time difference corresponding to the first information, the second compensation time difference corresponding to the second information, and the track compensation time difference corresponding to the track information based on the first time, the first timestamp, the second timestamp, the track timestamp, and a preset compensation threshold includes: The difference between the first time and the first timestamp is summed with the preset compensation threshold to obtain the first compensation time difference; The difference between the first time and the second timestamp is summed with the preset compensation threshold to obtain the second compensation time difference; The difference between the first time and the track timestamp is summed with the preset compensation threshold to obtain the track compensation time difference.

4. The method according to claim 1, characterized in that, Before fusing the obstacle information and the flight track information based on the obstacle association relationship and the obstacle-track association relationship, the method includes: Based on the first timestamp, the second timestamp and the track timestamp corresponding to each sub-sensor obtained through time soft synchronization, the second information and the track information are updated to the first timestamp to obtain the obstacle information to be associated and the track information to be associated. The obstacle information to be associated obtained by each secondary sensor and the primary sensor is correlated to obtain the obstacle association relationship; The obstacle information and the flight track information to be associated are correlated to obtain the association relationship between the obstacle and the flight track.

5. The method according to claim 4, characterized in that, The step of associating the obstacle information obtained from each secondary sensor and the primary sensor to obtain obstacle association relationships includes: Arbitrarily acquire the position, velocity, acceleration, and orientation of motion information from the first and second obstacle information to be associated from the corresponding secondary or primary sensor; Based on the position, velocity, acceleration, and orientation of motion information in the first and second obstacle information to be associated, a similarity measurement algorithm is used to calculate and filter the first and second obstacle information to be associated that meet the preset similarity threshold. A matching algorithm is used to obtain the obstacle association relationship between the first obstacle information to be associated and the second obstacle information to be associated.

6. The method according to claim 4, characterized in that, The step of associating the obstacle information and the flight track information to obtain the association relationship between the obstacle and the flight track includes: Obtain the position, velocity, acceleration, and orientation of motion information from any of the obstacle information to be associated and any of the trajectory information to be associated; Based on the position, velocity, acceleration, and orientation of motion information in the obstacle information and the trajectory information to be associated, a similarity measurement algorithm is used to calculate and filter the obstacle information and trajectory information to be associated that meet the preset similarity threshold. A matching algorithm is used to obtain the association relationship between the obstacle information and the trajectory information to be associated.

7. The method according to any one of claims 1 to 6, characterized in that, The step involves compensating the first information, the second information, and the trajectory information based on the compensated obstacle information, the compensated vehicle's own state information, and the compensated trajectory information, respectively, to obtain the obstacle information and trajectory information to be fused, including: The information changes of obstacles in the compensated obstacle information are respectively compensated into the corresponding obstacle information in the first information and the second information, and the compensated obstacle information is corrected according to the information changes in the compensated vehicle's own state information to obtain the obstacle information to be fused. The changes in the track information in the compensated track information are compensated into the track information, and the compensated track information is corrected according to the changes in the information in the compensated vehicle's own state information to obtain the track information to be fused.

8. The method according to any one of claims 1 to 6, characterized in that, The step of performing soft time synchronization on each sub-sensor based on the first timestamp to obtain a second timestamp corresponding to each sub-sensor and second information corresponding to the second timestamp includes: Based on the first timestamp, obtain the second timestamp closest to the first timestamp and the second information corresponding to the second timestamp in the buffer queue corresponding to each sub-sensor identifier.

9. The method according to any one of claims 4 or 6, characterized in that, The process of data fusion based on obstacle associations and obstacle-track associations includes: Based on the preset weights of each sensor, the obstacle information to be associated is weighted and fused according to the obstacle association relationship to obtain the measured obstacle information; Based on the preset weights of the measured obstacle information and the trajectory information to be associated, the measured obstacle information containing the obstacle information to be associated and the trajectory information to be associated are weighted and fused according to the association relationship between the obstacle and the trajectory to obtain the optimal obstacle information.

10. A fusion device based on an unmanned vehicle, characterized in that, include: The information acquisition module records the time corresponding to the first information acquired by the main sensor as the first time, and acquires the first timestamp in the first information, the track timestamp of the obstacle stored in the unmanned vehicle, and the track information corresponding to the track timestamp. The time soft synchronization module performs time soft synchronization on each sub-sensor based on the first timestamp to obtain the second timestamp and the second information corresponding to the second timestamp for each sub-sensor. The data prediction and processing module, based on the first time, the first timestamp, the second timestamp, the track timestamp, and the preset compensation threshold, performs prediction processing on the first information, the second information, and the track information respectively through a preset Kalman filter model to obtain the compensation obstacle information, the compensation vehicle's own state information, and the compensation track information corresponding to the compensation time difference; The information compensation and correction module performs information compensation and correction on the first information, the second information, and the trajectory information based on the compensation obstacle information, the compensation vehicle's own status information, and the compensation trajectory information, respectively, to obtain the obstacle information to be fused and the trajectory information to be fused. The association and fusion module, based on the relationship between obstacles and the relationship between obstacles and flight paths, realizes the data fusion of obstacle information and flight path information to be fused.

11. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 9.

Images