A method, device and system for fusion of multi-sensor data

Abstract

The embodiment of the present invention discloses a method, device and system for fusing multi-sensor data. The method includes: after determining that the current designated sensor data collected by the designated sensor is obtained, the processor determines whether a preset storage space stores sensor data with a corresponding collection time before a first time; if it is determined that it is stored, obtains target sensor data with a corresponding collection time before the first time and after a second time from the preset storage space; uses the current filter to filter the target sensor data to obtain a filtering fusion result; uses the current posture predictor, the filtering fusion result, the current designated sensor data and the designated sensor data between the current collection time and the first time to determine the current posture information of the target vehicle corresponding to the current designated sensor data, so as to obtain the vehicle positioning result of the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a multi-sensor data fusion method, device and system. Background Art

[0002] In unmanned driving technology, vehicle positioning technology is crucial. In related technologies, when performing vehicle positioning, the sensor data collected by multiple sensors such as image acquisition units, IMUs (Inertial measurement units), wheel speed sensors, and inertial navigation units installed in the target vehicle are generally fused to obtain the vehicle positioning result of the target vehicle. It can be seen that in vehicle positioning technology, the fusion method of multi-sensor data is crucial.

[0003] Then, how to provide a multi-sensor data fusion method becomes an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a multi-sensor data fusion method, device and system to achieve the acquisition of vehicle positioning results. The specific technical solution is as follows:

[0005] In a first aspect, an embodiment of the present invention provides a multi-sensor data fusion method, which is applied to a processor of a multi-sensor data fusion system, wherein the system further includes at least two types of sensors and a preset storage space; each sensor is configured to collect corresponding sensor data, and is disposed in the same target vehicle, including:

[0006] After determining to obtain the current designated sensor data collected by the designated sensor of the at least two types of sensors, determining whether the preset storage space stores sensor data whose corresponding collection time is before the first time, wherein the difference between the current collection time corresponding to the current designated sensor data and the first time is a preset time difference;

[0007] If it is determined that the preset storage space stores sensor data with corresponding collection times before the first time, obtain target sensor data with corresponding collection times before the first time and after a second time from the sensor data collected by the at least two types of sensors stored in the preset storage space, wherein the difference between the collection times corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference;

[0008] Filtering the target sensor data using the current filter to obtain a filtering fusion result;

[0009] The current pose information of the target vehicle corresponding to the current designated sensor data is determined by using the current pose predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time.

[0010] Optionally, the designated sensor is an IMU inertial measurement unit; the current designated sensor data is current IMU data;

[0011] After determining to obtain the current designated sensor data collected by the designated sensor, and before the step of determining whether the preset storage space stores sensor data corresponding to the generation time before the first time, the method further includes:

[0012] The process of obtaining the current IMU data includes:

[0013] Obtaining initial IMU data collected by the IMU;

[0014] Convert the initial IMU data into data in a first specified format to obtain intermediate IMU data corresponding to the initial IMU data;

[0015] Determine the current IMU data corresponding to the hourly moment by using the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data;

[0016] Storing the current IMU data and its corresponding collection time in the preset storage space;

[0017] After the step of determining the current pose information of the target vehicle corresponding to the current designated sensor data by using the current pose predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time, the method further includes:

[0018] Based on the current posture information, determining a map area corresponding to the current posture information from a target map as the map area corresponding to the current designated sensor data, wherein the target map includes map data;

[0019] The map area and the corresponding collection time corresponding to the currently specified sensor data converted into the second specified format are stored in the preset storage space.

[0020] Optionally, the at least two types of sensors include at least two of an IMU inertial measurement unit, a wheel speed sensor, an inertial navigation unit, and an image acquisition unit.

[0021] Optionally, if the at least two types of sensors include: wheel speed sensors, the sensor data collected by the at least two types of sensors include: spare wheel speed data collected by the wheel speed sensors; the method further includes:

[0022] The process of obtaining the spare wheel speed data collected by the wheel speed sensor includes:

[0023] Obtaining initial wheel speed data collected by the wheel speed sensor;

[0024] Converting the initial wheel speed data into data in a third specified format to obtain the backup wheel speed data;

[0025] The spare wheel speed data and the corresponding collection time are stored in the preset storage space.

[0026] Optionally, if the at least two types of sensors include: an inertial navigation unit, the sensor data collected by the at least two types of sensors include: backup inertial navigation data collected by the inertial navigation unit; the method further includes:

[0027] The process of obtaining the backup inertial navigation data collected by the inertial navigation unit includes:

[0028] Obtaining initial inertial navigation data collected by the inertial navigation unit;

[0029] Converting the initial inertial navigation data into data in a fourth specified format to obtain the backup inertial navigation data;

[0030] The backup inertial navigation data and the corresponding collection time are stored in the preset storage space.

[0031] Optionally, if the at least two types of sensors include: an image acquisition unit, the sensor data acquired by the at least two types of sensors include: spare image data acquired by the image acquisition unit; the method further includes:

[0032] A process of obtaining the spare image data acquired by the image acquisition unit, wherein the process comprises:

[0033] Obtaining an image acquired by the image acquisition unit;

[0034] Detecting the image using a pre-trained object detection model to obtain perception data corresponding to the image;

[0035] Converting the perception data corresponding to the image into data in a fifth specified format to obtain intermediate perception data;

[0036] Storing the intermediate sensing data and its corresponding collection time in the preset storage space;

[0037] and determining map data matching the intermediate perception data from a target map based on the intermediate perception data and the position information of the target vehicle corresponding to the image, wherein the target map includes map data;

[0038] storing the map data matched with the intermediate perception data converted into a sixth specified format in the preset storage space;

[0039] and extracting feature points from the image to determine feature point information in the image;

[0040] Encoding the feature point information in the image to obtain the image including the feature point information and the encoding result;

[0041] The image converted into the seventh specified format and including the feature point information and the encoding result, and its corresponding acquisition time, are stored in the preset storage space.

[0042] In a second aspect, an embodiment of the present invention provides a multi-sensor data fusion device, which is applied to a processor of a multi-sensor data fusion system, wherein the system further includes at least two types of sensors and a preset storage space; each sensor is configured to collect corresponding sensor data, and all are arranged in the same target vehicle, and the device includes:

[0043] a judgment module configured to, after determining to obtain the current designated sensor data collected by the designated sensor of the at least two types of sensors, judge whether the preset storage space stores sensor data whose corresponding collection time is before a first time, wherein the difference between the current collection time corresponding to the current designated sensor data and the first time is a preset time difference;

[0044] A first obtaining module is configured to obtain, if it is determined that the preset storage space stores sensor data with corresponding collection times before the first time, target sensor data with corresponding collection times before the first time and after a second time from the sensor data collected by the at least two types of sensors stored in the preset storage space, wherein the difference between the collection times corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference;

[0045] A filtering module is configured to filter the target sensor data using a current filter to obtain a filtering fusion result;

[0046] The first determination module is configured to use the current posture predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time to determine the current posture information of the target vehicle corresponding to the current designated sensor data.

[0047] Optionally, the designated sensor is an IMU inertial measurement unit; the current designated sensor data is current IMU data;

[0048] The device also includes:

[0049] A second obtaining module is configured to obtain the current IMU data after determining to obtain the current designated sensor data collected by the designated sensor and before determining whether the preset storage space stores sensor data corresponding to the generation time before the first time, wherein the second obtaining module is specifically configured to obtain the initial IMU data collected by the IMU;

[0050] Convert the initial IMU data into data in a first specified format to obtain intermediate IMU data corresponding to the initial IMU data;

[0051] Determine the current IMU data corresponding to the hourly moment by using the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data;

[0052] Storing the current IMU data and its corresponding collection time in the preset storage space;

[0053] A second determination module is configured to determine, after determining the current posture information of the target vehicle corresponding to the current designated sensor data by using the current posture predictor, the filter fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time, a map area corresponding to the current posture information from a target map based on the current posture information as the map area corresponding to the current designated sensor data, wherein the target map includes map data;

[0054] The storage module is configured to store the map area and the corresponding collection time corresponding to the currently specified sensor data converted into the second specified format in the preset storage space.

[0055] Optionally, the at least two types of sensors include at least two of an IMU inertial measurement unit, a wheel speed sensor, an inertial navigation unit, and an image acquisition unit.

[0056] Optionally, if the at least two types of sensors include: wheel speed sensors, the sensor data collected by the at least two types of sensors include: spare wheel speed data collected by the wheel speed sensors; the device further includes:

[0057] a third obtaining module, configured to obtain the spare wheel speed data collected by the wheel speed sensor, wherein the third obtaining module is specifically configured to obtain the initial wheel speed data collected by the wheel speed sensor;

[0058] Converting the initial wheel speed data into data in a third specified format to obtain the backup wheel speed data;

[0059] The spare wheel speed data and the corresponding collection time are stored in the preset storage space.

[0060] Optionally, if the at least two types of sensors include: an inertial navigation unit, the sensor data collected by the at least two types of sensors include: backup inertial navigation data collected by the inertial navigation unit; the device further includes:

[0061] a fourth obtaining module, configured to obtain the backup inertial navigation data collected by the inertial navigation unit, wherein the fourth obtaining module is specifically configured to obtain the initial inertial navigation data collected by the inertial navigation unit;

[0062] Converting the initial inertial navigation data into data in a fourth specified format to obtain the backup inertial navigation data;

[0063] The backup inertial navigation data and the corresponding collection time are stored in the preset storage space.

[0064] Optionally, if the at least two types of sensors include: an image acquisition unit, the sensor data acquired by the at least two types of sensors include: spare image data acquired by the image acquisition unit; the device further includes:

[0065] a fifth acquisition module, configured to obtain the spare image data acquired by the image acquisition unit, wherein the fifth acquisition module is specifically configured to obtain the image acquired by the image acquisition unit;

[0066] Detecting the image using a pre-trained object detection model to obtain perception data corresponding to the image;

[0067] Converting the perception data corresponding to the image into data in a fifth specified format to obtain intermediate perception data;

[0068] Storing the intermediate sensing data and its corresponding collection time in the preset storage space;

[0069] and determining map data matching the intermediate perception data from a target map based on the intermediate perception data and the position information of the target vehicle corresponding to the image, wherein the target map includes map data;

[0070] storing the map data matched with the intermediate perception data converted into a sixth specified format in the preset storage space;

[0071] and extracting feature points from the image to determine feature point information in the image;

[0072] Encoding the feature point information in the image to obtain the image including the feature point information and the encoding result;

[0073] The image converted into the seventh specified format and including the feature point information and the encoding result, and its corresponding acquisition time, are stored in the preset storage space.

[0074] In a third aspect, an embodiment of the present invention provides a multi-sensor data fusion system, the system comprising a processor, at least two types of sensors, and a preset storage space; each sensor is configured to collect corresponding sensor data, and is disposed in the same target vehicle; the preset storage space is configured to store the sensor data collected by the at least two types of sensors;

[0075] The processor is configured to determine whether the preset storage space stores sensor data whose corresponding collection time is before a first time after determining to obtain the current designated sensor data collected by the designated sensor of the at least two types of sensors, wherein the difference between the current collection time corresponding to the current designated sensor data and the first time is a preset time difference;

[0076] If it is determined that the preset storage space stores sensor data with corresponding collection times before the first time, obtain target sensor data with corresponding collection times before the first time and after a second time from the sensor data collected by the at least two types of sensors stored in the preset storage space, wherein the difference between the collection times corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference;

[0077] Filtering the target sensor data using the current filter to obtain a filtering fusion result;

[0078] The current pose information of the target vehicle corresponding to the current designated sensor data is determined by using the current pose predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time.

[0079] Optionally, the designated sensor is an IMU inertial measurement unit; the current designated sensor data is current IMU data;

[0080] The processor is further configured to obtain the current IMU data after determining to obtain the current designated sensor data collected by the designated sensor and before determining whether the preset storage space stores sensor data corresponding to the generation time before the first time, wherein the processing is specifically configured to obtain the initial IMU data collected by the IMU;

[0081] Convert the initial IMU data into data in a first specified format to obtain intermediate IMU data corresponding to the initial IMU data;

[0082] Determine the current IMU data corresponding to the hourly moment by using the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data;

[0083] Storing the current IMU data and its corresponding collection time in the preset storage space;

[0084] The processor is further configured to determine, after determining the current pose information of the target vehicle corresponding to the current designated sensor data by using the current pose predictor, the filter fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time, a map area corresponding to the current pose information from a target map based on the current pose information as the map area corresponding to the current designated sensor data, wherein the target map includes map data;

[0085] The map area and the corresponding collection time corresponding to the currently specified sensor data converted into the second specified format are stored in the preset storage space.

[0086] Optionally, the at least two types of sensors include at least two of an IMU inertial measurement unit, a wheel speed sensor, an inertial navigation unit, and an image acquisition unit.

[0087] Optionally, if the at least two types of sensors include: wheel speed sensors, the sensor data collected by the at least two types of sensors include: spare wheel speed data collected by the wheel speed sensors; the processor is further configured to obtain the spare wheel speed data collected by the wheel speed sensors, wherein the processor is specifically configured to obtain the initial wheel speed data collected by the wheel speed sensors;

[0088] Converting the initial wheel speed data into data in a third specified format to obtain the backup wheel speed data;

[0089] The spare wheel speed data and the corresponding collection time are stored in the preset storage space.

[0090] Optionally, if the at least two types of sensors include: an inertial navigation unit, the sensor data collected by the at least two types of sensors include: backup inertial navigation data collected by the inertial navigation unit; the processor is further configured to obtain the backup inertial navigation data collected by the inertial navigation unit, wherein the processor is specifically configured to obtain the initial inertial navigation data collected by the inertial navigation unit;

[0091] Converting the initial inertial navigation data into data in a fourth specified format to obtain the backup inertial navigation data;

[0092] The backup inertial navigation data and the corresponding collection time are stored in the preset storage space.

[0093] Optionally, if the at least two types of sensors include: an image acquisition unit, the sensor data acquired by the at least two types of sensors include: spare image data acquired by the image acquisition unit; the processor is further configured to obtain the spare image data acquired by the image acquisition unit, wherein the processor is specifically configured to obtain the image acquired by the image acquisition unit;

[0094] Detecting the image using a pre-trained object detection model to obtain perception data corresponding to the image;

[0095] Converting the perception data corresponding to the image into data in a fifth specified format to obtain intermediate perception data;

[0096] Storing the intermediate sensing data and its corresponding collection time in the preset storage space;

[0097] and determining map data matching the intermediate perception data from a target map based on the intermediate perception data and the position information of the target vehicle corresponding to the image, wherein the target map includes map data;

[0098] storing the map data matched with the intermediate perception data converted into a sixth specified format in the preset storage space;

[0099] and extracting feature points from the image to determine feature point information in the image;

[0100] Encoding the feature point information in the image to obtain the image including the feature point information and the encoding result;

[0101] The image converted into the seventh specified format and including the feature point information and the encoding result, and its corresponding acquisition time, are stored in the preset storage space.

[0102] As can be seen from the above content, an embodiment of the present invention provides a multi-sensor data fusion method, device and system, which is applied to a processor of a multi-sensor data fusion system, and the system also includes at least two types of sensors and a preset storage space; each sensor is configured to collect corresponding sensor data and is set in the same target vehicle; after determining to obtain the current designated sensor data collected by the designated sensor of at least two types of sensors, the processor determines whether the preset storage space stores sensor data with a corresponding collection time before the first time, wherein the difference between the current collection time and the first time corresponding to the current designated sensor data is the preset time difference; if it is determined that the preset storage space stores sensor data with a corresponding collection time before the first time, the target sensor data with a corresponding collection time before the first time and after the second time is obtained from the sensor data collected by at least two types of sensors stored in the preset storage space, wherein the difference between the collection time and the second time corresponding to the previous designated sensor data of the current designated sensor data is the preset time difference; the target sensor data is filtered by using the current filter to obtain a filtering fusion result; the current posture information of the target vehicle corresponding to the current designated sensor data is determined by using the current posture predictor, the filtering fusion result, the current designated sensor data and the designated sensor data between the current collection time and the first time.

[0103] By applying the embodiment of the present invention, the fusion process can be triggered after the current designated sensor data collected by the designated touch sensor is obtained, so that the fusion system of the multi-sensor data has strong expansibility, wherein, when the designated sensor data collected by the designated sensor can be obtained normally, the increase or decrease of other sensor data in at least two types of sensors will not affect the execution of the data fusion process of the fusion system of the multi-sensor data. In addition, in this embodiment, when the processor determines that the preset storage space stores the sensor data with the corresponding collection time before the first moment, it obtains the target sensor data with the corresponding collection time before the first moment and after the second moment stored in the preset storage space, and uses the current filter to filter the target sensor data to obtain the filtering fusion result, thereby avoiding the disorder of the filter when filtering the data due to the characteristics of the filter strictly filtering according to the time of the input data and the inconsistency of the transmission delay of the multi-sensor data, avoiding the burden of the filter data filtering process, ensuring the orderliness of the filter when filtering the data, and further achieving the vehicle positioning result with higher accuracy. Of course, any product or method implementing the present invention does not necessarily need to achieve all the advantages described above at the same time.

[0104] The innovative features of the embodiments of the present invention include:

[0105] 1. The fusion process can be triggered when the current designated sensor data collected by the designated touch sensor is obtained, so that the fusion system of the multi-sensor data has strong expansibility. In the case that the designated sensor data collected by the designated sensor can be obtained normally, the increase or decrease of other sensor data in at least two types of sensors will not affect the execution of the data fusion process of the fusion system of the multi-sensor data. In addition, in this embodiment, when the processor determines that the preset storage space stores the sensor data with the corresponding collection time before the first moment, it obtains the target sensor data with the corresponding collection time before the first moment and after the second moment stored in the preset storage space, and uses the current filter to filter the target sensor data to obtain the filtering fusion result, thereby avoiding the disorder of the filter when filtering the data due to the characteristics of the filter strictly filtering according to the time of the input data and the inconsistency of the transmission delay of the multi-sensor data, avoiding the burden of the filter data filtering process, ensuring the orderliness of the filter when filtering the data, and further achieving a vehicle positioning result with higher accuracy.

[0106] 2. Set the designated sensor to IMU. Due to the low data delay of IMU, the real-time performance of the current position information of the target vehicle is improved to a certain extent. After obtaining the initial IMU data, the intermediate IMU data in the first specified format corresponding to the previous IMU data collected by IMU and the intermediate IMU data in the first specified format corresponding to the initial IMU data are used to determine the current IMU data corresponding to the hour, so that when the positioning result accuracy is evaluated with other high-precision combined navigation devices in the future, additional interpolation work is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative work.

[0108] Figure 1 A schematic diagram of a flow chart of a multi-sensor data fusion method provided in an embodiment of the present invention;

[0109] Figure 2 A schematic diagram of the structure of a multi-sensor data fusion device provided in an embodiment of the present invention;

[0110] Figure 3 A schematic diagram of the structure of a multi-sensor data fusion system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0111] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0112] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0113] The present invention provides a method, device and system for fusing multi-sensor data to achieve the acquisition of vehicle positioning results. The embodiments of the present invention are described in detail below.

[0114] Figure 1 A flowchart of a multi-sensor data fusion method provided by an embodiment of the present invention. The method is applied to a processor of a multi-sensor data fusion system, and the system may also include at least two types of sensors and a preset storage space; each sensor is configured to collect corresponding sensor data, and is set in the same target vehicle; the preset storage space is configured to store sensor data collected by at least two types of sensors, and the method may include the following steps:

[0115] S101: after determining to obtain currently designated sensor data collected by a designated sensor from at least two types of sensors, determining whether a preset storage space stores sensor data corresponding to a collection time before a first time.

[0116] The difference between the current collection time corresponding to the currently specified sensor data and the first time is the preset time difference.

[0117] In one implementation, the processor of the multi-sensor data fusion system can be arranged in a vehicle-mounted platform in the target vehicle. The processor can communicate data with at least two types of sensors arranged in the target vehicle, and can obtain data collected by at least two types of sensors.

[0118] In one implementation, the at least two types of sensors may include but are not limited to at least two of an IMU (Inertial measurement unit), a wheel speed sensor, an inertial navigation unit, and an image acquisition unit. The inertial navigation unit may be a GNSS (Global Navigation Satellite System) positioning unit or a GPS (Global Positioning System) positioning unit. The image acquisition unit may be a camera, etc.

[0119] In an embodiment of the present invention, the processor can automatically or manually control to pre-designate a sensor from at least two types of sensors as a designated sensor, and the processor can immediately trigger the fusion process of the multi-sensor data after each determination to obtain sensor data collected by the designated sensor. The sensor data collected by the designated sensor can be referred to as designated sensor data. The current designated sensor data can be any designated sensor data that needs to be processed currently.

[0120] In this step, after determining that the current designated sensor data collected by the designated sensor is obtained, the processor can first traverse the preset storage space, and obtain the collection time of the current designated sensor data based on the designated sensor collection, that is, the current collection time, and determine the first time; then, determine whether the preset storage space stores sensor data with a corresponding collection time before the first time, and if it is determined that the preset storage space stores sensor data with a corresponding collection time before the first time, execute the subsequent S102.

[0121] The process of determining the first moment based on the acquisition of the current designated sensor data, i.e., the current acquisition moment, by the designated sensor may be: calculating the difference between the current acquisition moment and the preset time difference, and taking the obtained difference as the first moment. The preset time difference may be a time difference determined by default by the processor based on at least two types of sensors included in the fusion system of the multi-sensor data, or may be a time difference manually set by the staff based on at least two types of sensors included in the fusion system of the multi-sensor data.

[0122] The setting of the preset time difference is determined by the combination of at least two types of sensors included in the multi-sensor data fusion system, and the specific value of the preset time difference is determined by a preset principle, which is the time interval from the generation of a frame of data to the storage in the preset storage space among the at least two types of sensors. After waiting for the preset time difference, the data of the sensor with the slowest data transmission has also arrived at the preset storage space. Correspondingly, the preset time difference can be greater than or equal to the transmission delay of the target sensor among the at least two types of sensors included in the multi-sensor data fusion system, wherein the target sensor is: the sensor that takes the longest time to transmit the collected data to the preset storage space among the at least two types of sensors.

[0123] The preset storage space may be provided by a buffer.

[0124] In another case, when the processor determines that the preset storage space does not store sensor data corresponding to the collection time before the first time, the fusion process for the current designated sensor data can be terminated. Accordingly, the processor can continue to determine whether to obtain new current designated sensor data collected by the designated sensor.

[0125] The designated sensor may be any sensor in the multi-sensor data fusion system. In one implementation, considering that the transmission delay of IMU data is relatively short, the designated sensor may be IMU to improve the real-time determination of the position information of the target vehicle to a certain extent.

[0126] S102: If it is determined that the preset storage space stores sensor data with a corresponding collection time before the first time, obtain target sensor data with a corresponding collection time before the first time and after the second time from sensor data collected by at least two types of sensors stored in the preset storage space.

[0127] The difference between the collection time corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference.

[0128] In this step, if it is determined that the preset storage space stores sensor data with a corresponding collection time before the first time, the processor can obtain sensor data with a corresponding collection time before the first time and after the second time from the sensor data collected by at least two types of sensors stored in the preset storage space as the target sensor data. The difference between the collection time corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference.

[0129] S103: Filter the target sensor data using the current filter to obtain a filtering fusion result.

[0130] After obtaining the target sensor data, the processor inputs the target sensor data into the current filter, and uses the current filter to filter the target sensor data to obtain a filtering fusion result. In one case, the filter may be a Kalman filter, and a preset positioning fusion algorithm may be pre-set in the Kalman filter. The preset positioning fusion algorithm set in the Kalman filter can be used to fuse the target sensor data to obtain a filtering fusion result. Among them, the filtering fusion result may include the vehicle positioning result of the target vehicle at the first moment, that is, the posture information. The preset positioning fusion algorithm may be any positioning fusion algorithm in the relevant vehicle positioning, and the embodiment of the present invention does not limit the specific type of the preset positioning fusion algorithm.

[0131] S104: Determine the current posture information of the target vehicle corresponding to the current designated sensor data by using the current posture predictor, the filter fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time.

[0132] In this embodiment, after the processor obtains the above-mentioned filtering fusion result including the vehicle positioning result of the target vehicle at the first moment, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition moment and the first moment can be input into the current posture predictor, and the current posture predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition moment and the first moment are used to determine the posture information of the target vehicle corresponding to the current designated sensor data as the current posture information. Among them, the current posture information can include the acquisition moment of the current designated sensor data, that is, the position information and posture information of the target vehicle at the current acquisition moment.

[0133] Among them, the posture predictor may be pre-set with a preset posture prediction algorithm, and the preset posture prediction algorithm may be any posture prediction algorithm in the relevant vehicle positioning, and the embodiment of the present invention does not limit the specific type of the preset posture prediction algorithm. The process of determining the current posture information of the target vehicle corresponding to the current specified sensor data using the preset posture prediction algorithm in the posture predictor can refer to the relevant technology, which will not be repeated here.

[0134] In one implementation, after determining the current posture information of the target vehicle, the processor may output the current posture information to a corresponding posture-using application.

[0135] By applying the embodiment of the present invention, the fusion process can be triggered when the current designated sensor data collected by the designated touch sensor is obtained, so that the fusion system of the multi-sensor data has strong expansibility, wherein, when the designated sensor data collected by the designated sensor can be obtained normally, the increase or decrease of other sensor data in at least two types of sensors will not affect the execution of the data fusion process of the fusion system of the multi-sensor data. In addition, in this embodiment, when the processor determines that the preset storage space stores the sensor data with the corresponding collection time before the first moment, it obtains the target sensor data with the corresponding collection time before the first moment and after the second moment stored in the preset storage space, and uses the current filter to filter the target sensor data to obtain the filtering fusion result, thereby avoiding the disorder of the filter when filtering the data due to the characteristics of the filter strictly filtering according to the time of the input data and the inconsistency of the transmission delay of the multi-sensor data, avoiding the burden of the filter data filtering process, ensuring the orderliness of the filter when filtering the data, and further achieving a vehicle positioning result with higher accuracy.

[0136] In another embodiment of the present invention, the designated sensor is an IMU inertial measurement unit; the current designated sensor data is current IMU data;

[0137] Before S101, the method may further include:

[0138] The process of obtaining current IMU data may include:

[0139] Obtain the initial IMU data collected by the IMU;

[0140] Convert the initial IMU data into data in a first specified format to obtain intermediate IMU data corresponding to the initial IMU data;

[0141] Determine the current IMU data corresponding to the hour by using the intermediate IMU data corresponding to the previous IMU data and the intermediate IMU data corresponding to the initial IMU data collected by the IMU;

[0142] The current IMU data and its corresponding acquisition time are stored in a preset storage space;

[0143] After S104, the method may further include:

[0144] Based on the current posture information, determining a map area corresponding to the current posture information from a target map as a map area corresponding to the current specified sensor data, wherein the target map includes map data;

[0145] The map area and the corresponding collection time corresponding to the currently specified sensor data converted into the second specified format are stored in a preset storage space.

[0146] The IMU may include: a gyroscope for collecting the angular velocity of the target vehicle and an acceleration sensor for collecting the acceleration of the target vehicle and other devices.

[0147] In this embodiment, the designated sensor is an IMU, and accordingly, the current designated sensor data collected by the designated sensor is the current IMU data; in view of the differences in the formats of the IMU data collected by different types of IMUs, in order to facilitate the subsequent multi-sensor data fusion process. In the embodiment of the present invention, after obtaining the IMU data collected by the IMU, the processor first converts the collected IMU data into IMU data of a unified format in the multi-sensor data fusion system, and then executes the subsequent process. Accordingly, the processor obtains the initial IMU data collected by the IMU in real time, and processes the initial IMU data to obtain IMU data in a format that is convenient for the subsequent process, that is, the current IMU data. After the processor obtains the initial IMU data, the initial IMU data can be converted into data in the first specified format to obtain intermediate IMU data corresponding to the initial IMU data. Then, in order to facilitate subsequent fusion, the intermediate IMU data corresponding to the initial IMU data is aligned at the hourly point, that is, the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data are used to determine the current IMU data corresponding to the hourly point, and then the current IMU data and the collection time corresponding to the current IMU data are stored in a preset storage space.

[0148] Among them, the process of using the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data to determine the current IMU data corresponding to the hourly moment can be: using the difference algorithm, the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data to determine the current IMU data corresponding to the hourly moment. For example: taking a 100 Hz IMU as an example, the time interval between each two frames of IMU data collected by the IMU is 10 milliseconds, and the real time of the IMU collected by the IMU may be 1.1234 seconds and 1.1334 seconds. In order to traverse the subsequent process, it is necessary to perform an hourly alignment operation on the IMU data collected by the IMU, that is, calculate the IMU data collected by the IMU corresponding to 1.120 seconds and 1.130 seconds. Accordingly, the intermediate IMU data corresponding to the initial IMU data collected at 1.1234 seconds and the intermediate IMU data corresponding to the initial IMU data collected at 1.1334 seconds can be used to calculate the IMU data collected by the IMU corresponding to 1.130 seconds using a difference algorithm.

[0149] The first specified format can be any format in the related art that is convenient for the subsequent fusion process, and the embodiment of the present invention does not limit the specific type of the first specified format. For example: the first specified format may include the estimated speed and estimated posture of the target vehicle at the current acquisition moment calculated through the initial IMU data and the speed and posture information of the target vehicle at the previous moment of the current acquisition moment; or it may include the speed change and posture information change of the target vehicle between the current acquisition moment and the previous moment of the current acquisition moment. Among them, the current IMU data in the first specified format can be represented as an ImuFrame data frame.

[0150] Subsequently, in an embodiment of the present invention, the multi-sensor data fusion system also includes a target map, which is a map corresponding to the driving scene of the target vehicle, including map data; after the processor determines the current posture information of the target vehicle corresponding to the current specified sensor data, the map area corresponding to the current posture information can be determined from the target map based on the current posture, as the map area corresponding to the current specified sensor data; and the map area corresponding to the current specified sensor data is converted to a second specified format, and then the map area corresponding to the current specified sensor data in the second specified format and the collection time corresponding to the map area are stored in a preset storage space. Among them, the collection time corresponding to the map area can be the collection time of the current specified sensor data. In one case, the target map can be a high-precision map. The map area in the second specified format can be represented as an HdmapGeometryFrame data frame. The second specified format can be any format of a map area in the related technology that is convenient for the subsequent fusion process, and the embodiment of the present invention is not limited.

[0151] Among them, the area within a preset range centered on the current posture information in the target map can be used as the map area corresponding to the current posture information.

[0152] In this embodiment, the designated sensor is set to IMU. Due to the low data delay of IMU, the real-time performance of the current position information of the target vehicle is improved to a certain extent. After obtaining the initial IMU data, the intermediate IMU data in the first specified format corresponding to the previous IMU data collected by IMU and the intermediate IMU data in the first specified format corresponding to the initial IMU data are used to determine the current IMU data corresponding to the hour, so that when the positioning result accuracy is evaluated with other high-precision combined navigation devices in the future, additional interpolation work is avoided.

[0153] In another embodiment of the present invention, if the at least two types of sensors include: wheel speed sensors, and the sensor data collected by the at least two types of sensors include: spare wheel speed data collected by the wheel speed sensors; the method may further include:

[0154] The process of obtaining the spare wheel speed data collected by the wheel speed sensor may include:

[0155] Obtaining initial wheel speed data collected by a wheel speed sensor;

[0156] Converting the initial wheel speed data into data in a third specified format to obtain spare wheel speed data;

[0157] The spare wheel speed data and the corresponding collection time are stored in a preset storage space.

[0158] In this embodiment, the at least two types of sensors may include wheel speed sensors, which can collect the module values ​​of the wheel speeds of the four wheels of the target vehicle. In view of the different formats of data collected by wheel speed sensors of different models, for example: the collected data is the angular velocity of the wheel, or the collected data is the linear velocity of the wheel. In order to facilitate the subsequent fusion process, the initial wheel speed data collected by the wheel speed sensor is converted into data in a third specified format to obtain spare wheel speed data, and the spare wheel speed data and its corresponding collection time are stored in a preset storage space. The spare wheel speed data in the third specified format can be represented as an OdoFrame data frame. The third specified format can be any format of wheel speed data in the relevant technology that is convenient for the subsequent fusion process, and the embodiments of the present invention are not limited thereto.

[0159] In another embodiment of the present invention, if the at least two types of sensors include: an inertial navigation unit, the sensor data collected by the at least two types of sensors include: backup inertial navigation data collected by the inertial navigation unit; the method may further include:

[0160] A process of obtaining backup inertial navigation data collected by an inertial navigation unit, wherein the process may include:

[0161] Obtaining initial inertial navigation data collected by an inertial navigation unit;

[0162] Converting the initial inertial navigation data into data in a fourth specified format to obtain backup inertial navigation data;

[0163] The backup inertial navigation data and its corresponding collection time are stored in a preset storage space.

[0164] In view of the different formats of inertial navigation data collected by different inertial navigation units, after obtaining the initial inertial navigation data collected by the inertial navigation unit, the processor first converts the initial inertial navigation data into data in a fourth specified format to obtain backup inertial navigation data, and then stores the backup inertial navigation data and its corresponding collection time in a preset storage space. For example, in one case, the inertial navigation unit is a GNSS, and the initial inertial navigation data collected by the GNSS may include position information and speed information, and its format is usually NMEA sentences or binary sentences with a high compression rate. In order to facilitate the subsequent fusion process, the position information and speed information in the fourth specified format can be extracted from the initial inertial navigation data and stored in the preset storage space. The backup inertial navigation data in the fourth specified format can be represented as a GnssFrame data frame.

[0165] In another embodiment of the present invention, if the at least two types of sensors include: an image acquisition unit, the sensor data acquired by the at least two types of sensors include: spare image data acquired by the image acquisition unit; the method may further include:

[0166] A process of obtaining spare image data acquired by an image acquisition unit, wherein the process may include:

[0167] Obtaining an image acquired by an image acquisition unit;

[0168] Use the pre-trained object detection model to detect the image and obtain the perception data corresponding to the image;

[0169] Convert the perception data corresponding to the image into data in a fifth specified format to obtain intermediate perception data;

[0170] Storing the intermediate sensing data and its corresponding collection time in a preset storage space;

[0171] and determining map data matching the intermediate perception data from a target map based on the intermediate perception data and the position information of the target vehicle corresponding to the image, wherein the target map includes the map data;

[0172] storing the map data matched with the intermediate perception data converted into the sixth specified format in a preset storage space;

[0173] And extract feature points of the image to determine the feature point information in the image;

[0174] Encode the feature point information in the image to obtain an image containing the feature point information and the encoding result;

[0175] The image converted into the seventh specified format and including the feature point information and the encoding result, and its corresponding acquisition time, are stored in a preset storage space.

[0176] In this embodiment, at least two types of sensors may include an image acquisition unit, and accordingly, the sensor data collected by at least two types of sensors include: spare image data collected by the image acquisition unit. After the processor obtains the image collected by the image acquisition unit, on the one hand, the image can be converted into a preset image format so that it can be input into a pre-trained target detection model, and the image is detected using the pre-trained target detection model to obtain perception data corresponding to the image; the pre-trained target detection model is a neural network model trained based on sample images marked with targets and their annotation information, wherein the target may include traffic signs such as lane lines, parking spaces, street lamp poles, and traffic signs, and the annotation information may include the location information of the target in the corresponding sample image. The specific training process can refer to the training process of the model in the relevant technology, which will not be repeated here.

[0177] The perception data corresponding to the image may include the location and type of the target contained in the image, such as traffic signs such as lane lines, parking spaces, lamp posts and / or traffic signs and their locations. The perception data corresponding to the image is converted into data in the fifth specified format to obtain intermediate perception data, and the intermediate perception data in the fifth specified format and its corresponding acquisition time are stored in a preset storage space. The acquisition time corresponding to the intermediate perception data in the fifth specified format is the acquisition time of the corresponding image. The intermediate perception data in the fifth specified format can be represented as a PerceptionFrame data frame.

[0178] Based on the intermediate perception data and the position information of the target vehicle corresponding to the image, the map data matching each intermediate perception data is determined from the target map; the map data matching each intermediate perception data in the image is converted to the sixth specified format, and the map data and its corresponding acquisition time are stored in the preset storage space. The acquisition time corresponding to the map data matching each intermediate perception data in the image of the sixth specified format is the acquisition time of the image. The map data matching each intermediate perception data in the image of the sixth specified format can be represented as a SemanticMatchFrame data frame, wherein the SemanticMatchFrame data frame includes the intermediate perception data and its matching map data that have a corresponding relationship.

[0179] Among them, when IMU is the designated sensor, the frequency of obtaining IMU data is higher than the frequency of obtaining images. After obtaining a frame of image, the posture information of the target vehicle corresponding to the image has been stored in the preset storage space. The processor can directly read the posture information of the target vehicle whose corresponding acquisition time is closest to the acquisition time of the image from the preset storage space as the posture information of the target vehicle corresponding to the image.

[0180] On the other hand, a preset feature point extraction algorithm is used to extract feature points from an image to determine feature point information in the image, where the feature points may include corner points in the image, and correspondingly, the feature point information may include position information of corner points in the image; the feature point information in the image is encoded to obtain feature points in the image containing feature point information and encoding results, wherein the encoding ensures that the feature point information of the same code on different images corresponds to the same object in the actual scene. The image containing feature point information and encoding results is converted to the seventh specified format, and the image and its corresponding acquisition time are stored in a preset storage space. The acquisition time corresponding to the image containing feature point information and encoding results in the seventh specified format is the acquisition time of the image. The image containing feature point information and encoding results in the seventh specified format can be represented as a FeatureFrame data frame.

[0181] In one implementation, when the processor stores the sensor data collected by at least two types of sensors into a preset storage space, the processor may store the data in a sequence from front to back or from back to front according to the collection moments corresponding to the sensor data.

[0182] Corresponding to the above method embodiment, the embodiment of the present invention provides a multi-sensor data fusion device, a processor applied to a multi-sensor data fusion system, the system also includes at least two types of sensors and a preset storage space; each sensor is configured to collect corresponding sensor data, and is set in the same target vehicle, such as Figure 2 As shown, this may include:

[0183] The judgment module 210 is configured to, after determining to obtain the current designated sensor data collected by the designated sensor of the at least two types of sensors, judge whether the preset storage space stores sensor data whose corresponding collection time is before the first time, wherein the difference between the current collection time corresponding to the current designated sensor data and the first time is a preset time difference;

[0184] The first obtaining module 220 is configured to obtain, if it is determined that the preset storage space stores sensor data with corresponding collection times before the first time, target sensor data with corresponding collection times before the first time and after a second time from the sensor data collected by the at least two types of sensors stored in the preset storage space, wherein the difference between the collection times corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference;

[0185] The filtering module 230 is configured to filter the target sensor data using the current filter to obtain a filtering fusion result;

[0186] The first determination module 240 is configured to use the current posture predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time to determine the current posture information of the target vehicle corresponding to the current designated sensor data.

[0187] By applying the embodiment of the present invention, the fusion process can be triggered when the current designated sensor data collected by the designated touch sensor is obtained, so that the fusion system of the multi-sensor data has strong expansibility, wherein, when the designated sensor data collected by the designated sensor can be obtained normally, the increase or decrease of other sensor data in at least two types of sensors will not affect the execution of the data fusion process of the fusion system of the multi-sensor data. In addition, in this embodiment, when the processor determines that the preset storage space stores the sensor data with the corresponding collection time before the first moment, it obtains the target sensor data with the corresponding collection time before the first moment and after the second moment stored in the preset storage space, and uses the current filter to filter the target sensor data to obtain the filtering fusion result, thereby avoiding the disorder of the filter when filtering the data due to the characteristics of the filter strictly filtering according to the time of the input data and the inconsistency of the transmission delay of the multi-sensor data, avoiding the burden of the filter data filtering process, ensuring the orderliness of the filter when filtering the data, and further achieving a vehicle positioning result with higher accuracy.

[0188] In another embodiment of the present invention, the designated sensor is an IMU inertial measurement unit; the current designated sensor data is current IMU data;

[0189] The device also includes:

[0190] A second obtaining module is configured to obtain the current IMU data after determining to obtain the current designated sensor data collected by the designated sensor and before determining whether the preset storage space stores sensor data corresponding to the generation time before the first time, wherein the second obtaining module is specifically configured to obtain the initial IMU data collected by the IMU;

[0191] Convert the initial IMU data into data in a first specified format to obtain intermediate IMU data corresponding to the initial IMU data;

[0192] Determine the current IMU data corresponding to the hourly moment by using the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data;

[0193] Storing the current IMU data and its corresponding collection time in the preset storage space;

[0194] A second determination module is configured to determine, after determining the current posture information of the target vehicle corresponding to the current designated sensor data by using the current posture predictor, the filter fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time, a map area corresponding to the current posture information from a target map based on the current posture information as the map area corresponding to the current designated sensor data, wherein the target map includes map data;

[0195] The storage module is configured to store the map area and the corresponding collection time corresponding to the currently specified sensor data converted into the second specified format in the preset storage space.

[0196] In another embodiment of the present invention, the at least two types of sensors include at least two types of IMU inertial measurement units, wheel speed sensors, inertial navigation units, and image acquisition units.

[0197] In another embodiment of the present invention, if the at least two types of sensors include: wheel speed sensors, the sensor data collected by the at least two types of sensors include: spare wheel speed data collected by the wheel speed sensors; the device further includes:

[0198] a third obtaining module, configured to obtain the spare wheel speed data collected by the wheel speed sensor, wherein the third obtaining module is specifically configured to obtain the initial wheel speed data collected by the wheel speed sensor;

[0199] Converting the initial wheel speed data into data in a third specified format to obtain the backup wheel speed data;

[0200] The spare wheel speed data and the corresponding collection time are stored in the preset storage space.

[0201] In another embodiment of the present invention, if the at least two types of sensors include: an inertial navigation unit, the sensor data collected by the at least two types of sensors include: backup inertial navigation data collected by the inertial navigation unit; the device further includes:

[0202] a fourth obtaining module, configured to obtain the backup inertial navigation data collected by the inertial navigation unit, wherein the fourth obtaining module is specifically configured to obtain the initial inertial navigation data collected by the inertial navigation unit;

[0203] Converting the initial inertial navigation data into data in a fourth specified format to obtain the backup inertial navigation data;

[0204] The backup inertial navigation data and the corresponding collection time are stored in the preset storage space.

[0205] In another embodiment of the present invention, if the at least two types of sensors include: an image acquisition unit, the sensor data acquired by the at least two types of sensors include: spare image data acquired by the image acquisition unit; the device further includes:

[0206] a fifth acquisition module, configured to obtain the spare image data acquired by the image acquisition unit, wherein the fifth acquisition module is specifically configured to obtain the image acquired by the image acquisition unit;

[0207] Detecting the image using a pre-trained object detection model to obtain perception data corresponding to the image;

[0208] Converting the perception data corresponding to the image into data in a fifth specified format to obtain intermediate perception data;

[0209] Storing the intermediate sensing data and its corresponding collection time in the preset storage space;

[0210] and determining map data matching the intermediate perception data from a target map based on the intermediate perception data and the position information of the target vehicle corresponding to the image, wherein the target map includes map data;

[0211] storing the map data matched with the intermediate perception data converted into a sixth specified format in the preset storage space;

[0212] and extracting feature points from the image to determine feature point information in the image;

[0213] Encoding the feature point information in the image to obtain the image including the feature point information and the encoding result;

[0214] The image converted into the seventh specified format and including the feature point information and the encoding result, and its corresponding acquisition time, are stored in the preset storage space.

[0215] Corresponding to the above method embodiment, the embodiment of the present invention provides a multi-sensor data fusion system, such as Figure 3 As shown, the system includes a processor 310, at least two types of sensors 320 and a preset storage space 330; each sensor 320 is configured to collect corresponding sensor data, and is set in the same target vehicle; the preset storage space 330 is configured to store the sensor data collected by the at least two types of sensors;

[0216] The processor 310 is configured to determine whether the preset storage space stores sensor data whose corresponding collection time is before a first time after determining to obtain the current designated sensor data collected by the designated sensor of the at least two types of sensors, wherein the difference between the current collection time corresponding to the current designated sensor data and the first time is a preset time difference;

[0217] If it is determined that the preset storage space stores sensor data with corresponding collection times before the first time, obtain target sensor data with corresponding collection times before the first time and after a second time from the sensor data collected by the at least two types of sensors stored in the preset storage space, wherein the difference between the collection times corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference;

[0218] Filtering the target sensor data using the current filter to obtain a filtering fusion result;

[0219] The current pose information of the target vehicle corresponding to the current designated sensor data is determined by using the current pose predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time.

[0220] By applying the embodiment of the present invention, the fusion process can be triggered when the current designated sensor data collected by the designated touch sensor is obtained, so that the fusion system of the multi-sensor data has strong expansibility, wherein, when the designated sensor data collected by the designated sensor can be obtained normally, the increase or decrease of other sensor data in at least two types of sensors will not affect the execution of the data fusion process of the fusion system of the multi-sensor data. In addition, in this embodiment, when the processor determines that the preset storage space stores the sensor data with the corresponding collection time before the first moment, it obtains the target sensor data with the corresponding collection time before the first moment and after the second moment stored in the preset storage space, and uses the current filter to filter the target sensor data to obtain the filtering fusion result, thereby avoiding the disorder of the filter when filtering the data due to the characteristics of the filter strictly filtering according to the time of the input data and the inconsistency of the transmission delay of the multi-sensor data, avoiding the burden of the filter data filtering process, ensuring the orderliness of the filter when filtering the data, and further achieving a vehicle positioning result with higher accuracy.

[0221] In another embodiment of the present invention, the designated sensor is an IMU inertial measurement unit; the current designated sensor data is current IMU data;

[0222] The processor 310 is further configured to obtain the current IMU data after determining to obtain the current designated sensor data collected by the designated sensor and before determining whether the preset storage space stores sensor data corresponding to the generation time before the first time, wherein the processing is specifically configured to obtain the initial IMU data collected by the IMU;

[0223] Convert the initial IMU data into data in a first specified format to obtain intermediate IMU data corresponding to the initial IMU data;

[0224] Determine the current IMU data corresponding to the hourly moment by using the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data;

[0225] Storing the current IMU data and its corresponding collection time in the preset storage space;

[0226] The processor 310 is further configured to determine, after determining the current pose information of the target vehicle corresponding to the current designated sensor data by using the current pose predictor, the filter fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time, a map area corresponding to the current pose information from a target map based on the current pose information as the map area corresponding to the current designated sensor data, wherein the target map includes map data;

[0227] The map area and the corresponding collection time corresponding to the currently specified sensor data converted into the second specified format are stored in the preset storage space.

[0228] In another embodiment of the present invention, the at least two types of sensors 320 include at least two types of IMU inertial measurement unit, wheel speed sensor, inertial navigation unit and image acquisition unit.

[0229] In another embodiment of the present invention, if the at least two types of sensors 320 include: wheel speed sensors, the sensor data collected by the at least two types of sensors include: spare wheel speed data collected by the wheel speed sensors; the processor is further configured to obtain the spare wheel speed data collected by the wheel speed sensors, wherein the processor 310 is specifically configured to obtain the initial wheel speed data collected by the wheel speed sensors;

[0230] Converting the initial wheel speed data into data in a third specified format to obtain the backup wheel speed data;

[0231] The spare wheel speed data and the corresponding collection time are stored in the preset storage space.

[0232] In another embodiment of the present invention, if the at least two types of sensors 320 include: an inertial navigation unit, the sensor data collected by the at least two types of sensors include: backup inertial navigation data collected by the inertial navigation unit; the processor 310 is further configured to obtain the backup inertial navigation data collected by the inertial navigation unit, wherein the processor 310 is specifically configured to obtain the initial inertial navigation data collected by the inertial navigation unit;

[0233] Converting the initial inertial navigation data into data in a fourth specified format to obtain the backup inertial navigation data;

[0234] The backup inertial navigation data and the corresponding collection time are stored in the preset storage space.

[0235] In another embodiment of the present invention, if the at least two types of sensors 320 include: an image acquisition unit, the sensor data acquired by the at least two types of sensors include: spare image data acquired by the image acquisition unit; the processor 310 is further configured to obtain the spare image data acquired by the image acquisition unit, wherein the processor 310 is specifically configured to obtain the image acquired by the image acquisition unit;

[0236] Detecting the image using a pre-trained object detection model to obtain perception data corresponding to the image;

[0237] Converting the perception data corresponding to the image into data in a fifth specified format to obtain intermediate perception data;

[0238] Storing the intermediate sensing data and its corresponding collection time in the preset storage space;

[0239] and determining map data matching the intermediate perception data from a target map based on the intermediate perception data and the position information of the target vehicle corresponding to the image, wherein the target map includes map data;

[0240] storing the map data matched with the intermediate perception data converted into a sixth specified format in the preset storage space;

[0241] and extracting feature points from the image to determine feature point information in the image;

[0242] Encoding the feature point information in the image to obtain the image including the feature point information and the encoding result;

[0243] The image converted into the seventh specified format and including the feature point information and the encoding result, and its corresponding acquisition time, are stored in the preset storage space.

[0244] The above device and system embodiments correspond to the method embodiments and have the same technical effects as the method embodiments. For specific descriptions, please refer to the method embodiments. The device embodiments are obtained based on the method embodiments. For specific descriptions, please refer to the method embodiments part, which will not be repeated here.

[0245] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0246] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0247] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensor data fusion method, characterized in that: A processor for a multi-sensor data fusion system, the system further comprising at least two types of sensors and a preset storage space; each sensor is configured to collect corresponding sensor data, and both are arranged in the same target vehicle, the at least two types of sensors comprising at least two of an IMU inertial measurement unit, a wheel speed sensor, an inertial navigation unit, and an image acquisition unit, the method comprising: After determining to obtain the current designated sensor data collected by the designated sensor among the at least two types of sensors, determine whether the preset storage space stores sensor data with a corresponding collection time before a first time, wherein the difference between the current collection time corresponding to the current designated sensor data and the first time is a preset time difference, and the preset time difference is greater than or equal to the transmission delay of the target sensor among the at least two types of sensors included in the multi-sensor data fusion system, wherein the target sensor is: the sensor among the at least two types of sensors that takes the longest time to transmit the collected data to the preset storage space; If it is determined that the preset storage space stores sensor data with corresponding collection times before the first time, obtain target sensor data with corresponding collection times before the first time and after a second time from the sensor data collected by the at least two types of sensors stored in the preset storage space, wherein the difference between the collection times corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference; Filtering the target sensor data using the current filter to obtain a filtering fusion result; The current pose information of the target vehicle corresponding to the current designated sensor data is determined by using the current pose predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time.

2. The method according to claim 1, characterized in that The designated sensor is an IMU inertial measurement unit; the current designated sensor data is current IMU data; After determining to obtain the current designated sensor data collected by the designated sensor, and before the step of determining whether the preset storage space stores sensor data corresponding to the generation time before the first time, the method further includes: The process of obtaining the current IMU data includes: Obtaining initial IMU data collected by the IMU; Convert the initial IMU data into data in a first specified format to obtain intermediate IMU data corresponding to the initial IMU data; Determine the current IMU data corresponding to the hourly moment by using the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data; Storing the current IMU data and its corresponding collection time in the preset storage space; After the step of determining the current pose information of the target vehicle corresponding to the current designated sensor data by using the current pose predictor, the filter fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time, the method further includes: Based on the current posture information, determining a map area corresponding to the current posture information from a target map as the map area corresponding to the current designated sensor data, wherein the target map includes map data; The map area and the corresponding collection time corresponding to the currently specified sensor data converted into the second specified format are stored in the preset storage space.

3. The method according to any one of claims 1 to 2, characterized in that: If the at least two types of sensors include: wheel speed sensors, the sensor data collected by the at least two types of sensors include: spare wheel speed data collected by the wheel speed sensors; the method further includes: The process of obtaining the spare wheel speed data collected by the wheel speed sensor includes: Obtaining initial wheel speed data collected by the wheel speed sensor; Converting the initial wheel speed data into data in a third specified format to obtain the backup wheel speed data; The spare wheel speed data and the corresponding collection time are stored in the preset storage space.

4. The method according to any one of claims 1 to 2, characterized in that: If the at least two types of sensors include: an inertial navigation unit, the sensor data collected by the at least two types of sensors include: backup inertial navigation data collected by the inertial navigation unit; the method further includes: The process of obtaining the backup inertial navigation data collected by the inertial navigation unit includes: Obtaining initial inertial navigation data collected by the inertial navigation unit; Converting the initial inertial navigation data into data in a fourth specified format to obtain the backup inertial navigation data; The backup inertial navigation data and the corresponding collection time are stored in the preset storage space.

5. The method according to any one of claims 1 to 2, characterized in that: If the at least two types of sensors include: an image acquisition unit, the sensor data acquired by the at least two types of sensors include: spare image data acquired by the image acquisition unit; the method further includes: A process of obtaining the spare image data acquired by the image acquisition unit, wherein the process comprises: Obtaining an image acquired by the image acquisition unit; Detecting the image using a pre-trained object detection model to obtain perception data corresponding to the image; Converting the perception data corresponding to the image into data in a fifth specified format to obtain intermediate perception data; Storing the intermediate sensing data and its corresponding collection time in the preset storage space; and determining map data matching the intermediate perception data from a target map based on the intermediate perception data and the position information of the target vehicle corresponding to the image, wherein the target map includes map data; storing the map data matched with the intermediate perception data converted into a sixth specified format in the preset storage space; and extracting feature points from the image to determine feature point information in the image; Encoding the feature point information in the image to obtain the image including the feature point information and the encoding result; The image converted into the seventh specified format and including the feature point information and the encoding result, and its corresponding acquisition time, are stored in the preset storage space.

6. A multi-sensor data fusion device, characterized in that: A processor applied to a multi-sensor data fusion system, the system further comprising at least two types of sensors and a preset storage space; Each sensor is configured to collect corresponding sensor data and is disposed in the same target vehicle. The device comprises: a judgment module configured to, after determining to obtain the current designated sensor data collected by the designated sensor of the at least two types of sensors, judge whether the preset storage space stores sensor data whose corresponding collection time is before a first time, wherein the difference between the current collection time corresponding to the current designated sensor data and the first time is a preset time difference; A first obtaining module is configured to obtain target sensor data whose corresponding collection time is before the first time and after a second time from the sensor data collected by the at least two types of sensors stored in the preset storage space if it is determined that the preset storage space stores sensor data whose corresponding collection time is before the first time, wherein the difference between the collection time corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference, and the preset time difference is greater than or equal to the transmission delay of the target sensor among the at least two types of sensors included in the multi-sensor data fusion system, wherein the target sensor is: the sensor whose collected data among the at least two types of sensors takes the longest time to be transmitted to the preset storage space; A filtering module is configured to filter the target sensor data using a current filter to obtain a filtering fusion result; The first determination module is configured to use the current posture predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time to determine the current posture information of the target vehicle corresponding to the current designated sensor data.

7. The device according to claim 6, characterized in that The designated sensor is an IMU inertial measurement unit; the current designated sensor data is current IMU data; The device also includes: A second obtaining module is configured to obtain the current IMU data after determining to obtain the current designated sensor data collected by the designated sensor and before determining whether the preset storage space stores sensor data corresponding to the generation time before the first time, wherein the second obtaining module is specifically configured to obtain the initial IMU data collected by the IMU; Convert the initial IMU data into data in a first specified format to obtain intermediate IMU data corresponding to the initial IMU data; Determine the current IMU data corresponding to the hourly moment by using the intermediate IMU data corresponding to the previous IMU data collected by the IMU and the intermediate IMU data corresponding to the initial IMU data; Storing the current IMU data and its corresponding collection time in the preset storage space; A second determination module is configured to determine, after determining the current posture information of the target vehicle corresponding to the current designated sensor data by using the current posture predictor, the filter fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time, a map area corresponding to the current posture information from a target map based on the current posture information as the map area corresponding to the current designated sensor data, wherein the target map includes map data; The storage module is configured to store the map area and the corresponding collection time corresponding to the currently specified sensor data converted into the second specified format in the preset storage space.

8. The device according to claim 6 or 7, characterized in that The at least two types of sensors include at least two types of IMU inertial measurement unit, wheel speed sensor, inertial navigation unit and image acquisition unit.

9. A multi-sensor data fusion system, the system comprising a processor, at least two types of sensors and a preset storage space; each sensor is configured to collect corresponding sensor data, and all sensors are arranged in the same target vehicle; the preset storage space is configured to store the sensor data collected by the at least two types of sensors; The processor is configured to determine whether the preset storage space stores sensor data with a corresponding collection time before the first time after determining to obtain the current designated sensor data collected by the designated sensor of the at least two types of sensors, wherein: The difference between the current collection time corresponding to the currently specified sensor data and the first time is a preset time difference; If it is determined that the preset storage space stores sensor data with corresponding collection times before the first time, obtain target sensor data with corresponding collection times before the first time and after the second time from the sensor data collected by the at least two types of sensors stored in the preset storage space, wherein the difference between the collection time corresponding to the previous designated sensor data of the current designated sensor data and the second time is the preset time difference, and the preset time difference is greater than or equal to the transmission delay of the target sensor among the at least two types of sensors included in the multi-sensor data fusion system, wherein the target sensor is: the sensor of the at least two types of sensors that takes the longest time to transmit the collected data to the preset storage space; Filtering the target sensor data using the current filter to obtain a filtering fusion result; The current pose information of the target vehicle corresponding to the current designated sensor data is determined by using the current pose predictor, the filtering fusion result, the current designated sensor data, and the designated sensor data between the current acquisition time and the first time.

Images