Multi-sensor data fusion method and device, electronic equipment and storage medium

Abstract

The invention relates to a multi-sensor data fusion method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining current time and sensing data of a plurality of sensors in a cache period; according to the global tracking database, tracking information of all known target objects is determined, prediction information of all known target objects at the current time is determined, sensing data of all sensors in the cache period are aligned to the current time, and multi-source observation data after time alignment are obtained. And based on the prediction information and the multi-source observation data of all the known targets at the current time, target matching is performed by using a preset matching algorithm to obtain a matching result, and according to the matching result, the prediction information and the multi-source observation data of all the known targets at the current time are fused to obtain a fusion result. Therefore, the problem that data of multiple sensors are not synchronous is solved, the data of the sensors are matched and fused at the same moment, and the accuracy of target detection is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a multi-sensor data fusion method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of automatic driving technology, multi-sensor fusion perception systems play a crucial role in environmental understanding. By integrating the advantages of multiple sensors such as cameras and millimeter wave radars, the system can more accurately and robustly identify and track surrounding targets, providing reliable basis for decision-making and control.

[0003] In practical applications, the differences in working characteristics of different sensors bring many challenges to data fusion. How to achieve efficient, stable and flexible multi-source information fusion has become a key technical problem to improve the safety of automatic driving.

[0004] The related art has the following defects: 1. Due to the significant asynchrony of sensors, i.e. different sampling frequencies and processing delays of the same type of sensors, the time alignment strategy in the related art is difficult to accurately compensate for the time delay, resulting in inconsistent states at the fusion time. 2. Single sensor has inherent limitations: millimeter wave radar is prone to target fission (a real target is misdetected as multiple targets) in complex electromagnetic environments, while cameras are prone to target jumping or loss in strong light, backlight, etc., affecting tracking stability. 3. The related art is usually designed for fixed sensor configurations, lacks flexibility, and is difficult to adapt to changes in sensor types and quantities in different vehicle models or application scenarios, limiting the universality and scalability of the system. SUMMARY

[0005] The present application provides a multi-sensor data fusion method, device, electronic equipment and storage medium to solve the problem of multi-sensor data asynchrony and single sensor error. The present application matches and fuses the data of each sensor at the same time to improve the accuracy of target detection.

[0006] The first aspect of the present application provides a multi-sensor data fusion method, comprising the following steps: obtaining the current time and the perception data of multiple sensors within a cache period; determining the tracking information of all known target objects according to the global tracking database, and determining the prediction information of all known target objects at the current time through a preset motion model, aligning the perception data of all sensors within the cache period to the current time to obtain multi-source observation data after time alignment; Based on the prediction information of all known targets at the current time and the multi-source observation data, target matching is performed using a preset matching algorithm to obtain a matching result, and based on the matching result, the prediction information of all known targets at the current time and the multi-source observation data are fused to obtain a fusion result.

[0007] Optionally, in some embodiments, after the fusion of the prediction information of all known targets at the current time and the multi-source observation data to obtain the fusion result, the following steps are included: According to the fusion result, the state information of all known targets at the current time is determined, and the global tracking database is updated according to the state information of all known targets at the current time.

[0008] Optionally, in some embodiments, the fusion of the prediction information of all known targets at the current time and the multi-source observation data to obtain the fusion result includes: determining a first weighting coefficient of the prediction information of each known target at the current time, and a second weighting coefficient of the observation data of the sensor matched to the corresponding known target; based on the first weighting coefficient and the prediction information of each known target at the current time, and the second weighting coefficient and the observation data of the sensor matched to the corresponding known target, the fusion data of each known target is obtained, and the fusion result is obtained according to the fusion data of all known targets.

[0009] Optionally, in some embodiments, aligning the perception data of all sensors within the cache period to the current time includes: for the perception data with a collection time earlier than the current time, linear interpolation and / or polynomial extrapolation is used to align to the current time.

[0010] Optionally, in some embodiments, before determining the prediction information of all known targets at the current time through the preset motion model, the following steps are included: obtain a historical tracking data set, and construct a data set based on the historical tracking data set; based on a preset division ratio, divide the data set into a training set, a validation set and a test set; construct a target neural network, input the training set into the target neural network for training to obtain initial model parameters; based on the initial model parameters, input the validation set into the target neural network for performance evaluation, and adjust the initial model parameters according to the performance evaluation result until the joint loss function of the validation set converges, to obtain optimal model parameters; Based on the optimal model parameters, the test set is input into the target neural network for model testing, and when the test result meets the preset requirement, the preset motion model is obtained.

[0011] The second aspect embodiment of the application provides a multi-sensor data fusion device, comprising: An acquisition module is configured to acquire a current time and perception data of a plurality of sensors in a cache period. A time alignment module is configured to determine tracking information of all known target objects according to a global tracking database, and determine predicted information of all known target objects at the current time through a preset motion model, align the perception data of all sensors in the cache period to the current time, and obtain multi-source observation data after time alignment. A fusion module is configured to perform target object matching based on the predicted information of all known target objects at the current time and the multi-source observation data using a preset matching algorithm, obtain a matching result, and fuse the predicted information of all known target objects at the current time and the multi-source observation data to obtain a fusion result according to the matching result.

[0012] Optionally, in some embodiments, after fusing the predicted information of all known target objects at the current time and the multi-source observation data to obtain the fusion result, the fusion module is configured to: determine state information of all known target objects at the current time according to the fusion result, and update the global tracking database according to the state information of all known target objects at the current time.

[0013] Optionally, in some embodiments, the fusion of the predicted information of all known target objects at the current time and the multi-source observation data to obtain the fusion result, the fusion module is further configured to: determine a first weighting coefficient of the predicted information of each known target object at the current time, and a second weighting coefficient of the observation data of the sensor matched to the corresponding known target object; fuse the first weighting coefficient, the predicted information of each known target object at the current time, the second weighting coefficient, and the observation data of the sensor matched to the corresponding known target object to obtain fusion data of each known target object, and obtain the fusion result according to the fusion data of all known target objects.

[0014] Optionally, in some embodiments, the time alignment module is configured to: for perception data with a collection time earlier than the current time, use linear interpolation and / or polynomial extrapolation to align to the current time.

[0015] Optionally, in some embodiments, before determining the prediction information of all known target objects at the current time through the preset motion model, the time alignment module is further configured to: obtain a historical tracking dataset, and construct a dataset based on the historical tracking dataset; divide the dataset into a training set, a validation set and a test set based on a preset division ratio; construct a target neural network, input the training set into the target neural network to obtain initial model parameters; input the validation set into the target neural network based on the initial model parameters to perform performance evaluation, and adjust the initial model parameters according to the performance evaluation result until the joint loss function of the validation set converges to obtain optimal model parameters; input the test set into the target neural network based on the optimal model parameters to perform model testing, and obtain the preset motion model when the test result meets a preset requirement.

[0016] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-sensor data fusion method as described in the above embodiments.

[0017] The fourth aspect of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the multi-sensor data fusion method as described in the above embodiments.

[0018] Thus, by obtaining the current time and the perception data of multiple sensors within a cache period, determining the tracking information of all known target objects according to a global tracking database, and determining the prediction information of all known target objects at the current time through a preset motion model, the perception data of all sensors within the cache period is aligned to the current time to obtain time-aligned multi-source observation data, and based on the prediction information of all known target objects at the current time and the multi-source observation data, a target object matching is performed using a preset matching algorithm to obtain a matching result, and based on the matching result, the prediction information of all known target objects at the current time and the multi-source observation data are fused to obtain a fusion result. Thus, the problem of multi-sensor data asynchronization and single sensor error is solved, and the accuracy of target detection is improved by matching and fusing the data of each sensor to the same time.

[0019] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of exemplary embodiments of the present application, wherein: Figure 1 A flowchart of a multi-sensor data fusion method according to an embodiment of the present application is provided. Figure 2 A block diagram of a multi-sensor data fusion apparatus according to an embodiment of the present application is provided. Figure 3 A structural diagram of an electronic device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0021] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which like or similar elements are denoted by the same or similar reference numerals, and the embodiments described below are merely exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0022] A multi-sensor data fusion method, apparatus, electronic device, and storage medium according to embodiments of the present application are described below with reference to the accompanying drawings. In view of the above background art, the present application provides a multi-sensor data fusion method that addresses the problems of multi-sensor data asynchronization and single-sensor error-prone. In the method, the current time and the perception data of multiple sensors within a cache period are obtained, the tracking information of all known target objects is determined according to a global tracking database, the prediction information of all known target objects at the current time is determined through a preset motion model, the perception data of all sensors within the cache period is aligned to the current time to obtain time-aligned multi-source observation data, and target object matching is performed using a preset matching algorithm based on the prediction information of all known target objects at the current time and the multi-source observation data to obtain a matching result. The prediction information of all known target objects at the current time and the multi-source observation data are fused according to the matching result to obtain a fusion result. Thus, the problems of multi-sensor data asynchronization and single-sensor error-prone are solved, and the accuracy of target detection is improved by matching and fusing the data of each sensor to the same time.

[0023] Specifically, Figure 1 A flowchart of a multi-sensor data fusion method according to an embodiment of the present application is provided.

[0024] As shown in the flowchart, the multi-sensor data fusion method includes the following steps: Figure 1 In step S101, the current time and the perception data of multiple sensors within a cache period are obtained.

[0025] ​The current time is the time when the fusion processing is performed; the sensors include a vehicle-mounted camera, a millimeter wave radar and the like; and the cache period is a time window (for example, the last 100 seconds) set in advance and used for temporarily storing the perception data collected by each sensor at different time points.

[0026] Specifically, the embodiment of the application first determines the current time, and then reads all the perception data reported within a cache period before the time from the data queues of the sensors. Because the sampling frequencies and transmission delays of different sensors are different (for example, the camera has one frame every 33 ms, and the radar has one frame every 20 ms), the data are time-asynchronous. By introducing the cache period, the system can collect enough time segments of original observations, thereby providing a complete data basis for subsequent time alignment and cross-sensor matching.

[0027] The embodiment of the application is applicable to various sensor combination types (for example, 4R1V, BEV+4R1V, 5R1V, and the like) and adopts a time synchronization strategy with memory.

[0028] Therefore, the application obtains the perception data of multiple sensors within a specified time range, prevents the problem of difficult data synchronization caused by different sampling frequencies and delays of the sensors, enables the system to process asynchronous inputs under a unified time reference, and realizes high-robustness multi-sensor fusion.

[0029] In step S102, the tracking information of all known target objects is determined according to the global tracking database, and the prediction information of all known target objects at the current time is determined through a preset motion model, so as to align the perception data of all sensors within the cache period to the current time and obtain the time-aligned multi-source observation data.

[0030] Further, in some embodiments, aligning the perception data of all sensors within the cache period to the current time includes: for the perception data collected at a time earlier than the current time, using linear interpolation and / or polynomial extrapolation to align to the current time.

[0031] The global tracking database is used to store the historical state information of known target objects confirmed by the system, including position, speed, time stamp and identity; the tracking information is the motion trajectory of the known target object within the cache period, which is composed of the position, speed and corresponding time stamp of the target object at multiple historical time points; the prediction information is the position and speed of the known target object at the current time output by the motion model; and the time alignment means that the perception data collected at different time points are mapped to the same time reference through interpolation or extrapolation.

[0032] Specifically, the system obtains the motion trajectories of all known target objects in the cache period as their tracking information from the global tracking database, and based on the motion trajectories, predicts the current time prediction information of each known target object through a preset motion model. At the same time, the system performs time alignment processing on the perception data reported by each sensor in the cache period, and adjusts the state quantity of the observation data collected at a time earlier than the current time to the current time by using a linear interpolation or a polynomial extrapolation method, to form the time-aligned multi-source observation data.

[0033] Optionally, in some embodiments, before determining the prediction information of all known target objects at the current time through the preset motion model, the method comprises: obtaining a historical tracking data set, and constructing a data set based on the historical tracking data set; dividing the data set into a training set, a validation set and a test set based on a preset division ratio; constructing a target neural network, inputting the training set into the target neural network for training to obtain initial model parameters; inputting the validation set into the target neural network for performance evaluation based on the initial model parameters, and adjusting the initial model parameters according to the performance evaluation result until the joint loss function of the validation set converges to obtain optimal model parameters; inputting the test set into the target neural network for model testing based on the optimal model parameters, and obtaining the preset motion model when the test result meets the preset requirements.

[0034] The historical tracking data set is the multi-time point state data of known target objects accumulated by the system in actual operation or simulation environment, including position, velocity, acceleration and time stamp; the training set, the validation set and the test set are subsets divided from the data set according to a preset ratio (such as 70%, 15% and 15%), which are respectively used for model training, parameter tuning and performance verification; the target neural network is a deep learning model for learning target motion law; and the joint loss function is a comprehensive index for measuring the error between the predicted value and the true value of the model.

[0035] Specifically, in some embodiments, before prediction using the motion model, the system first obtains a historical tracking data set, and constructs a complete data set for model training based on the data set. Subsequently, the data set is divided into a training set, a validation set and a test set according to a preset division ratio. After constructing the target neural network, the training set is input into the network for initial training to obtain initial model parameters; then, the validation set is input into the network for performance evaluation, and the model parameters are adjusted according to the change of the joint loss function until the loss function converges to obtain optimal model parameters; finally, the test set is input into the network for model testing, and when the test result meets the preset accuracy or error requirement, the network is determined as the preset motion model for subsequent target state prediction. In step S103, based on the prediction information of all known targets at the current time and the multi-source observation data, a preset matching algorithm is used to perform target matching to obtain a matching result, and based on the matching result, the prediction information of all known targets at the current time and the multi-source observation data are fused to obtain a fusion result.

[0036] Further, in some embodiments, the fusion of the prediction information of all known targets at the current time and the multi-source observation data to obtain the fusion result comprises: determining a first weighting coefficient of the prediction information of each known target at the current time and a second weighting coefficient of the observation data of the sensor matched to the corresponding known target; and fusing the prediction information of each known target at the current time based on the first weighting coefficient and the second weighting coefficient and the observation data of the sensor matched to the corresponding known target to obtain fusion data of each known target, and obtaining the fusion result based on the fusion data of all known targets.

[0037] The multi-source observation data is data of multiple sensors that have completed time alignment; the preset matching algorithm comprises a Hungarian algorithm, a nearest neighbor matching, and the like; the first weighting coefficient is used to represent the weight of the prediction information in the fusion, and the second weighting coefficient is used to represent the weight of the observation data of each sensor in the fusion.

[0038] Specifically, the system performs target matching based on the prediction information of all known targets at the current time and the multi-source observation data that have completed time alignment, using a preset matching algorithm to generate a matching result and determine which observation data belongs to a known target. Subsequently, based on the matching result, data fusion is performed for each known target: a first weighting coefficient is assigned to the prediction information of the target, a second weighting coefficient is assigned to the observation data of each sensor matched to the target, and based on the weighting coefficients, the prediction information and the corresponding observation data are weighted and calculated to obtain fusion data of the target; finally, the fusion data of all known targets is summarized to form the final fusion result of the current fusion period.

[0039] Thus, the present application can combine the prediction information and the multi-source observation data by a weighting manner to generate fusion data of each known target based on the matching result.

[0040] Optionally, in some embodiments, after the fusion of the prediction information of all known targets at the current time and the multi-source observation data to obtain the fusion result, the method comprises: determining state information of all known targets at the current time based on the fusion result, and updating a global tracking database based on the state information of all known targets at the current time.

[0041] The following will be described in combination with the system architecture of the data fusion method of the multi-sensor adopted in the embodiment of the application, and the system architecture comprises a sensor adaptation layer, a time synchronization module, a target object matching module, a core processing layer, an abstraction layer, and a user interface layer.

[0042] Among them, the sensor adaptation layer is responsible for receiving data from different sensors and converting it into a unified format; the time synchronization module implements a time synchronization strategy with memory, including global tracking information management and dynamic time correction functions; the target object matching module performs target object matching based on the current frame and historical tracking information; the core processing layer contains the main data processing logic and intelligent fusion algorithm; the abstraction layer provides a unified API interface to support cross-platform compatibility; and the user interface layer provides a friendly user interface for easy management and monitoring of sensor status.

[0043] The time synchronization process comprises: 1. initialization stage: establishing a global tracking database to record the initial state and timestamp of each sensor; 2. data acquisition stage: each sensor collects environmental data according to its sampling frequency and sends the data to the time synchronization module; 3. time synchronization processing: for each frame of data, first predict the position of each target object at the current time according to the timestamp of the sensor and the global tracking information, and use interpolation or other prediction models to update the historical tracking information to the current time point; 4. target object matching: combine the target object information of the current frame and the global tracking information predicted to the current time, and use matching algorithms (such as Hungarian algorithm, Kalman filter, etc.) for target object matching; 5. result output: pass the matching results to the subsequent processing module for further decision support or environment modeling.

[0044] The multi-sensor fusion strategy is: 1. millimeter wave radar and camera fusion: use the stability and anti-interference ability of millimeter wave radar, combined with the high resolution and detail capture ability of the camera, in strong light environment, preferentially rely on millimeter wave radar data; in low light conditions, preferentially rely on camera data. 2. BEV (bird's eye view) fusion: the embodiment of the application can project the data of multiple sensors to the same plane through the BEV view to enhance the spatial perception ability; the embodiment of the application can also use deep learning algorithm to optimize target detection and classification in the BEV view.

[0045] In summary, the embodiments of the present application have the following beneficial effects: (1) high compatibility and reusability: supporting multiple sensor combination types (such as 4R1V, 5R1V, BEV+4R1V), and being able to flexibly increase or decrease sensors; (2) time synchronization strategy with memory: through retaining global tracking information, time synchronization is performed on target object information in each frame of data; (3) combining the advantages of millimeter wave radar and camera: using the stability of millimeter wave radar and the high resolution of camera, solving the problems of radar fission and camera jumping, loss and the like under strong light.

[0046] According to the multi-sensor data fusion method provided in the embodiments of the present application, by acquiring the current time and the perception data of the plurality of sensors in the cache period, the tracking information of all known target objects is determined according to the global tracking database, and the prediction information of all known target objects at the current time is determined through a preset motion model. The perception data of all sensors in the cache period is aligned to the current time to obtain time-aligned multi-source observation data. Based on the prediction information of all known target objects at the current time and the multi-source observation data, a preset matching algorithm is used for target object matching to obtain a matching result. According to the matching result, the prediction information of all known target objects at the current time and the multi-source observation data are fused to obtain a fusion result. Thus, the problem of asynchronous multi-sensor data and the error-prone single sensor is solved. The data of each sensor is matched and fused to the same moment, and the accuracy of target detection is improved.

[0047] Secondly, the multi-sensor data fusion device according to the embodiments of the present application is described with reference to the accompanying drawings.

[0048] Figure 2 is a block schematic diagram of the multi-sensor data fusion device of the embodiments of the present application.

[0049] As shown in Figure 2 , the multi-sensor data fusion device 10 comprises an acquisition module 100, a time alignment module 200 and a fusion module 300.

[0050] The acquisition module 100 is configured to acquire the current time and the perception data of the plurality of sensors in the cache period.

[0051] The time alignment module 200 is configured to determine the tracking information of all known target objects according to the global tracking database, and determine the prediction information of all known target objects at the current time through a preset motion model. The perception data of all sensors in the cache period is aligned to the current time to obtain time-aligned multi-source observation data.

[0052] The fusion module 300 is configured to perform target matching based on the predicted information of all known targets at the current time and the multi-source observation data by using a preset matching algorithm to obtain a matching result, and fuse the predicted information of all known targets at the current time and the multi-source observation data based on the matching result to obtain a fusion result.

[0053] Optionally, in some embodiments, after the fusion of the predicted information of all known targets at the current time and the multi-source observation data to obtain the fusion result, the fusion module 300 is configured to determine state information of all known targets at the current time based on the fusion result, and update the global tracking database based on the state information of all known targets at the current time.

[0054] Optionally, in some embodiments, the fusion of the predicted information of all known targets at the current time and the multi-source observation data to obtain a fusion result, the fusion module 300 is further configured to determine a first weighting coefficient of the predicted information of each known target at the current time and a second weighting coefficient of the observation data of the sensor matched to the corresponding known target, fuse the predicted information of each known target at the current time based on the first weighting coefficient and the observation data of the sensor matched to the corresponding known target based on the second weighting coefficient to obtain fusion data of each known target, and obtain the fusion result based on the fusion data of all known targets.

[0055] Optionally, in some embodiments, the time alignment module 200 is configured to, for perception data collected at a time earlier than the current time, align to the current time by using linear interpolation and / or polynomial extrapolation.

[0056] Optionally, in some embodiments, before determining the predicted information of all known targets at the current time by using the preset motion model, the time alignment module 200 is further configured to obtain a historical tracking data set, construct a data set based on the historical tracking data set, divide the data set into a training set, a validation set and a test set based on a preset division ratio, construct a target neural network, input the training set into the target neural network to obtain initial model parameters by training, input the validation set into the target neural network based on the initial model parameters to perform performance evaluation, and adjust the initial model parameters according to the performance evaluation result until the joint loss function of the validation set converges to obtain optimal model parameters, and input the test set into the target neural network based on the optimal model parameters to perform model testing, and obtain the preset motion model when the test result meets a preset requirement.

[0057] It should be noted that the foregoing explanation of the multi-sensor data fusion method embodiment is also applicable to the multi-sensor data fusion device of the embodiment, and will not be repeated here.

[0058] The multi-sensor data fusion device provided by the embodiment of the application obtains the current time and the perception data of the plurality of sensors in the cache period, determines the tracking information of all known target objects according to the global tracking database, determines the prediction information of all known target objects at the current time through a preset motion model, aligns the perception data of all sensors in the cache period to the current time to obtain time-aligned multi-source observation data, performs target object matching based on the prediction information of all known target objects at the current time and the multi-source observation data using a preset matching algorithm to obtain a matching result, and fuses the prediction information of all known target objects at the current time and the multi-source observation data according to the matching result to obtain a fusion result. In this way, the problem of asynchronous multi-sensor data and the problem of single-sensor error are solved, and the accuracy of target detection is improved by matching and fusing the data of each sensor to the same time.

[0059] Figure 3 The electronic device provided by the embodiment of the application has the structure shown in the structural schematic diagram of the electronic device. The electronic device can include: The memory 301, the processor 302, and the computer program stored in the memory 301 and executable on the processor 302.

[0060] The processor 302 implements the multi-sensor data fusion method provided in the above embodiments when executing the program.

[0061] Further, the electronic device further includes: The communication interface 303 is used for communication between the memory 301 and the processor 302.

[0062] The memory 301 is used to store the computer program executable on the processor 302.

[0063] The memory 301 can include a high-speed RAM (Random Access Memory, random access memory) memory and can also include a non-volatile memory, such as at least one disk memory.

[0064] If the memory 301, the processor 302 and the communication interface 303 are implemented independently, the communication interface 303, the memory 301 and the processor 302 can be connected with each other through a bus and complete communication between each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, Figure 3 Only one thick line is used to represent the bus in the figure, but it does not mean that there is only one bus or only one type of bus.

[0065] Optionally, in a specific implementation, if the memory 301, the processor 302 and the communication interface 303 are integrated on a chip, the memory 301, the processor 302 and the communication interface 303 can complete communication between each other through an internal interface.

[0066] The processor 302 can be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0067] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the multi-sensor data fusion method as above.

[0068] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example" or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms is not necessarily for the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0069] In addition, the terms "first", "second", etc. are used herein only to describe different instances, and do not imply or suggest relative importance or a number of the indicated technical features. Thus, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically limited.

[0070] Any process or method descriptions or descriptions of the flow diagrams described herein or otherwise described in the present application can be understood as representing executable instructions, code or modules stored in a computer-readable storage medium, and the scope of the preferred embodiments of the present application includes additional implementations or implementations in which the functions described in the processes or methods described herein or otherwise described in the present application are implemented in an order different from that shown or discussed, including substantially simultaneously, or in reverse order, or in an order that is not shown or discussed, and the scope of the preferred embodiments of the present application includes additional implementations or implementations in which the functions described in the processes or methods described herein or otherwise described in the present application are implemented by hardware, software, firmware or any combination thereof.

[0071] It should be understood that various parts of the present application can be implemented in hardware, software, firmware or a combination thereof. In the above-described embodiments, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. As in another embodiment, if implemented in hardware, any of the following technologies known in the art or their combinations can be used: discrete logic circuit with logic gate circuit for implementing logic functions on data signals, application specific integrated circuit with suitable combination logic gate circuit, programmable gate array, field programmable gate array, etc.

[0072] Those skilled in the art of the present technology can understand that all or part of the steps carried out by the above-mentioned embodiment methods can be completed by programs instructing related hardware, and the programs can be stored in a computer-readable storage medium, and the programs include one or a combination of the steps of the method embodiments when executed.

[0073] Although the embodiments of the present application have been shown and described above, it should be understood that the above-described embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-described embodiments within the scope of the present application.

Claims

1. A multi-sensor data fusion method, characterized in that, The method comprises the following steps: obtaining current time and perception data of multiple sensors within a cache period; determining tracking information of all known target objects according to a global tracking database, and determining prediction information of all known target objects at the current time through a preset motion model, aligning the perception data of all sensors within the cache period to the current time to obtain time-aligned multi-source observation data; based on the prediction information of all known target objects at the current time and the multi-source observation data, using a preset matching algorithm for target object matching to obtain a matching result, and fusing the prediction information of all known target objects at the current time and the multi-source observation data according to the matching result to obtain a fusion result.

2. The method of claim 1, wherein, After fusing the prediction information of all known target objects at the current time and the multi-source observation data to obtain the fusion result, comprising: determining the state information of all known target objects at the current time according to the fusion result, and updating the global tracking database according to the state information of all known target objects at the current time.

3. The method of claim 1, wherein, The fusion of the prediction information of all known target objects at the current time and the multi-source observation data to obtain the fusion result comprises: determining a first weighting coefficient of the prediction information of each known target object at the current time, and a second weighting coefficient of the observation data of the sensor matched to the corresponding known target object; based on the first weighting coefficient, the prediction information of each known target object at the current time, and the second weighting coefficient, the observation data of the sensor matched to the corresponding known target object, fusing to obtain the fusion data of each known target object, and obtaining the fusion result according to the fusion data of all known target objects.

4. The method of claim 1, wherein, Aligning the perception data of all sensors within the cache period to the current time comprises: for the perception data with a collection time earlier than the current time, using linear interpolation and / or polynomial extrapolation to align to the current time.

5. The method according to claim 1, characterized in that, Before determining the prediction information of all known target objects at the current time through the preset motion model, comprising: obtaining a historical tracking data set, and constructing a data set based on the historical tracking data set; based on a preset division ratio, dividing the data set into a training set, a validation set and a test set; constructing a target neural network, inputting the training set into the target neural network for training to obtain initial model parameters; based on the initial model parameters, inputting the validation set into the target neural network for performance evaluation, and adjusting the initial model parameters according to the performance evaluation result until the joint loss function of the validation set converges to obtain optimal model parameters; based on the optimal model parameters, inputting the test set into the target neural network for model testing, and obtaining the preset motion model when the test result meets the preset requirements.

6. A multi-sensor data fusion apparatus, characterized by comprising: comprising: an acquisition module for acquiring current time and perception data of multiple sensors within a cache period; The time alignment module is configured to determine tracking information of all known target objects according to a global tracking database, determine predicted information of all known target objects at a current time through a preset motion model, and align perception data of all sensors in a cache period to the current time to obtain time-aligned multi-source observation data. The fusion module is configured to perform target matching based on the predicted information of all known target objects at the current time and the multi-source observation data through a preset matching algorithm to obtain a matching result, and fuse the predicted information of all known target objects at the current time and the multi-source observation data according to the matching result to obtain a fusion result.

7. The apparatus of claim 6, wherein, After fusing the predicted information of all known target objects at the current time and the multi-source observation data to obtain the fusion result, the fusion module is configured to: determine state information of all known target objects at the current time according to the fusion result, and update the global tracking database according to the state information of all known target objects at the current time.

8. The apparatus of claim 6, wherein, The fusion module is further configured to: determine a first weighting coefficient of the predicted information of each known target object at the current time and a second weighting coefficient of observation data of a sensor matched to the corresponding known target object; fuse the first weighting coefficient, the predicted information of each known target object at the current time, the second weighting coefficient, and the observation data of the sensor matched to the corresponding known target object to obtain fusion data of each known target object, and obtain the fusion result according to the fusion data of all known target objects.

9. An electronic device, comprising: The computer program is executed by the processor to implement the multi-sensor data fusion method according to any one of claims 1-5. The program is executed by the processor to implement the multi-sensor data fusion method according to any one of claims 1-5.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images