Multi-sensor data synchronization method and device and electronic equipment

Abstract

The embodiment of the invention relates to the technical field of vehicles, and provides a multi-sensor data synchronization method and device and electronic equipment, and the method comprises the steps: obtaining real data frames and data collection moments of a plurality of to-be-synchronized sensors in a vehicle in a to-be-synchronized time period, taking one to-be-synchronized sensor as a main sensor, and taking the other to-be-synchronized sensors as auxiliary sensors, determining a to-be-synchronized moment in the data acquisition moments of the main sensor corresponding to each auxiliary sensor, and determining at least one related moment of each to-be-synchronized moment from all the data acquisition moments of the auxiliary sensors, determining a virtual data value of the auxiliary sensor at a corresponding to-be-synchronized moment on the basis of the quantized data value of the real data frame of the auxiliary sensor at the related moment so as to obtain a virtual data frame of the auxiliary sensor at each to-be-synchronized moment; according to the method, the real data frames of the auxiliary sensors and the real data frames of the main sensor can be synchronized in the time dimension, the method can be achieved through an algorithm, and the requirement for the performance of the sensors is relatively low.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a multi-sensor data synchronization method, device and electronic equipment. Background Art

[0002] There are a large number of sensors in a vehicle. In some scenarios, it is necessary to combine the data collected by multiple sensors for fusion analysis. However, the data collection frequencies set by different sensors may be different, or they may have the same data collection frequency but different starting data collection times, or there may be differences in both data collection frequency and starting data collection time. These situations will cause the data frames collected by multiple sensors to be out of sync in the time dimension, thereby affecting the accuracy of data fusion analysis.

[0003] In the related technology, a hardware-triggered data synchronization method can be used, such as PTP (Precision Time Protocol). Although this can achieve high-precision time synchronization through Ethernet, all devices involved in the synchronization need to support the PTP protocol. There are costs and technical difficulties in actual use, and the performance differences of different sensors will also cause the final data to not be synchronized with high precision. It is also possible to use methods such as offline statistics of sensor timestamp errors and online compensation to reduce the error between the sensor timestamp and the actual data acquisition time, but this method has a long calibration link and there are uncontrollable factors such as data delay in the receiving system; in addition, the characteristics and acquisition frequency differences of different sensors, such as the rotation scanning of the laser radar and the instantaneous exposure of the camera, result in that even in the case of hardware synchronization, only approximately synchronous acquisition can be guaranteed, and there is a certain time deviation.

[0004] It can be seen that the solutions of related technologies all have certain limitations, cannot accurately achieve high-precision data synchronization for multi-sensor data, and have high requirements on the performance of sensors. Summary of the invention

[0005] The embodiments of the present application provide a multi-sensor data synchronization method, device and electronic device to solve the technical problem in the related art that high-precision data synchronization of multi-sensor data cannot be accurately achieved and that high performance requirements are placed on the sensors.

[0006] The embodiment of the present application provides a multi-sensor data synchronization method, the multi-sensor data synchronization method comprising: obtaining multiple real data frames collected by sensors to be synchronized in a vehicle within a time period to be synchronized and a data collection time of each real data frame, wherein the number of sensors to be synchronized is at least two; determining one sensor to be synchronized as a main sensor, and determining other sensors to be synchronized except the main sensor as auxiliary sensors; executing a data synchronization strategy for the real data frame of each auxiliary sensor respectively, so that the real data frame of each auxiliary sensor is synchronized with the real data frame of the main sensor in a time dimension, wherein the data synchronization strategy comprises: determining a time to be synchronized, the time to be synchronized being a time among the data collection times of the main sensor that is different from all the data collection times of an auxiliary sensor, determining at least one relevant time of each time to be synchronized from all the data collection times of the auxiliary sensor, determining a virtual data value of the auxiliary sensor at the time to be synchronized based on the quantized data values ​​of the real data frame of the auxiliary sensor at all the relevant times corresponding to the time to be synchronized, and determining a virtual data frame of the auxiliary sensor at the time to be synchronized based on the virtual data value, thereby obtaining a virtual data frame of the auxiliary sensor at each time to be synchronized.

[0007] In one embodiment of the present application, a sensor to be synchronized is determined as the main sensor, including: determining the data sampling frequency of each sensor to be synchronized according to the data collection time; if the number of sensors to be synchronized with the highest data sampling frequency is one, determining the sensor to be synchronized with the highest data sampling frequency as the main sensor; if the number of sensors to be synchronized with the highest data sampling frequency is greater than one, determining one of the multiple sensors to be synchronized with the highest data sampling frequency as the main sensor.

[0008] In one embodiment of the present application, at least one relevant moment of each moment to be synchronized is determined from all the data collection moments of the auxiliary sensor, and a virtual data value of the auxiliary sensor at the moment to be synchronized is determined based on the quantized data values ​​of the real data frame of the auxiliary sensor at all the relevant moments corresponding to the moment to be synchronized, including: determining a data collection moment closest to the moment to be synchronized among all the data collection moments of the auxiliary sensor that are less than the moment to be synchronized as a relevant moment; determining a data collection moment closest to the moment to be synchronized among all the data collection moments of the auxiliary sensor that are greater than the moment to be synchronized as another relevant moment; determining a moment weight of each relevant moment according to the moment to be synchronized and each of the relevant moments; and determining the virtual data value of the auxiliary sensor at the moment to be synchronized based on the moment weight of each relevant moment and the quantized data value of the real data frame of the auxiliary sensor at the corresponding relevant moment.

[0009] In one embodiment of the present application, the moment weight of each relevant moment is determined based on the one to-be-synchronized moment and each of the relevant moments, including: determining the first time difference between each of the relevant moments and the one to-be-synchronized moment, and determining the second time difference between the two relevant moments; determining the moment weight of each relevant moment based on the ratio between the first time difference between each relevant moment and the one to-be-synchronized moment and the second time difference.

[0010] In one embodiment of the present application, the virtual data value of the auxiliary sensor at the time to be synchronized is determined based on the time weight of each relevant moment and the quantized data value of the real data frame of the auxiliary sensor at the corresponding relevant moment, including: determining a first virtual data sub-value at the relevant moment according to the product between the time weight of the relevant moment and the quantized data value of the real data frame of the auxiliary sensor at the relevant moment; determining a second virtual data sub-value at another relevant moment according to the product between the time weight of another relevant moment and the quantized data value of the real data frame of the auxiliary sensor at the other relevant moment; and obtaining the virtual data value of the auxiliary sensor at the time to be synchronized based on the sum of the first virtual data sub-value and the second virtual data sub-value.

[0011] In one embodiment of the present application, at least one relevant moment of each moment to be synchronized is determined from all the data collection moments of the auxiliary sensor, and a virtual data value of the auxiliary sensor at the moment to be synchronized is determined based on the quantized data values ​​of the real data frame of the auxiliary sensor at all the relevant moments corresponding to the moment to be synchronized, including: respectively determining the time difference between all the data collection moments of the auxiliary sensor and each moment to be synchronized; if the time difference is less than a preset difference threshold, determining a data collection moment with the smallest time difference as the relevant moment of the corresponding moment to be synchronized; and determining the quantized data value of the real data frame of the auxiliary sensor at the relevant moment corresponding to the moment to be synchronized as the virtual data value of the auxiliary sensor at the moment to be synchronized.

[0012] In one embodiment of the present application, before executing the data synchronization strategy for each real data frame of the auxiliary sensor, the method further includes: if there are missing data frames in all the real data frames of the main sensor, completing the missing data frames.

[0013] In one embodiment of the present application, after executing the data synchronization strategy on the real data frame of an auxiliary sensor, the method also includes: determining the virtual data evaluation parameters of the auxiliary sensor according to the quantized data values ​​of the virtual data frames of the auxiliary sensor at all the synchronization moments after executing the data synchronization strategy and the quantized data values ​​of at least part of the real data frames of the auxiliary sensor; determining the data synchronization strategy execution status of the auxiliary sensor based on the virtual data evaluation parameters; if the data synchronization strategy execution status is failure, adjusting the weights corresponding to each relevant moment, and re-executing the data synchronization strategy on the real data frame of the auxiliary sensor based on the adjusted weights corresponding to each relevant moment, until the data synchronization strategy execution success status is changed to success.

[0014] The embodiment of the present application also provides a multi-sensor data synchronization device, which includes: an acquisition module, used to acquire multiple real data frames collected by the sensors to be synchronized in the vehicle within the synchronization time period and the data collection time of each real data frame, and the number of the sensors to be synchronized is at least two; a main and auxiliary sensor determination module, used to determine one sensor to be synchronized as the main sensor, and determine the other sensors to be synchronized except the main sensor as auxiliary sensors; a synchronization module, used to execute a data synchronization strategy for the real data frame of each auxiliary sensor, so that the real data frame of each auxiliary sensor is synchronized with the real data frame of the main sensor in the time dimension. Synchronization, wherein the data synchronization strategy includes: determining a time to be synchronized, the time to be synchronized being a time among the data collection times of the main sensor that is different from all the data collection times of an auxiliary sensor, determining at least one related time of each time to be synchronized from all the data collection times of the auxiliary sensor, determining a virtual data value of the auxiliary sensor at the time to be synchronized based on a quantized data value of a real data frame of the auxiliary sensor at all the related times corresponding to the time to be synchronized, and determining a virtual data frame of the auxiliary sensor at the time to be synchronized based on the virtual data value, thereby obtaining a virtual data frame of the auxiliary sensor at each time to be synchronized.

[0015] An embodiment of the present application also provides a vehicle, comprising the multi-sensor data synchronization device provided by any of the above embodiments and a plurality of sensors to be synchronized.

[0016] An embodiment of the present application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any one of the above embodiments when executing the computer program.

[0017] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method described in any one of the above embodiments is implemented.

[0018] In the scheme implemented by the multi-sensor data synchronization method, device and electronic device provided above, the multi-sensor data synchronization method determines a sensor to be synchronized as the main sensor and the rest as auxiliary sensors by obtaining multiple real data frames collected by the sensors to be synchronized in the vehicle within the synchronization time period and the data collection time of each real data frame, and takes the time in the data collection time of the main sensor that is different from all the data collection times of an auxiliary sensor as the time to be synchronized, and determines at least one relevant time of each time to be synchronized from all the data collection times of an auxiliary sensor, and determines the virtual data value of the next auxiliary sensor at a time to be synchronized based on the quantized data value of the real data frame of the next auxiliary sensor at all the relevant times corresponding to the time to be synchronized, and determines the virtual data frame of the next auxiliary sensor at a time to be synchronized based on the virtual data value, and then obtains the virtual data frame of the next auxiliary sensor at each time to be synchronized, so that the real data frame of each auxiliary sensor is synchronized with the real data frame of the main sensor in the time dimension. The method can be implemented by an algorithm, can ensure the accuracy of time synchronization, has high execution flexibility and low cost, and has relatively low performance requirements for sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor.

[0020] Figure 1 is a schematic diagram of an application scenario of an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a flow chart of a multi-sensor data synchronization method provided in an embodiment of the present application;

[0022] Figure 3 A specific flowchart of the multi-sensor data synchronization method provided in an embodiment of the present application;

[0023] Figure 4 A specific flow chart of a process for determining a virtual data frame at a synchronization time provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of a data frame for completing data synchronization of two sensors provided in an embodiment of the present application;

[0025] Figure 6 for Figure 5 A schematic diagram of a camera data frame provided in an embodiment;

[0026] Figure 7 for Figure 5 A schematic diagram of a radar data frame provided in an embodiment;

[0027] Figure 8 A schematic diagram of the structure of a multi-sensor data synchronization device provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] See also Figure 1 As shown, Figure 1 1 is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario may include a vehicle and at least two sensors of the vehicle. The frequencies of the data collected by the two sensors are inconsistent, which causes certain difficulties for subsequent related data analysis. In this regard, the method provided in the following embodiment can be used to achieve data synchronization of the data collected by the sensor in the time dimension.

[0031] See also Figure 2 As shown, Figure 2 A schematic flow chart of a multi-sensor data synchronization method provided in an embodiment of the present application, the method comprising the following steps:

[0032] Step S210, obtaining a plurality of real data frames collected by the to-be-synchronized sensors in the vehicle within the to-be-synchronized time period and the data collection time of each real data frame.

[0033] The number of sensors to be synchronized is at least two. The sensors to be synchronized may be sensors of the same category, such as two radars, or sensors of different categories, such as a radar and a camera. Alternatively, there may be some sensors of the same category and some sensors of different categories, such as two radars and two cameras.

[0034] In the field of vehicle driving planning, it is necessary to rely on the perception data collected by one or more sensors of the vehicle. Since different sensors may have inconsistent data collection frequencies, the data finally obtained cannot be aligned in the time dimension, which is not conducive to subsequent data analysis and processing. Assume that there are two sensors, camera and radar, in the multi-sensor system. The acquisition frequency of the camera is 20Hz, and the acquisition frequency of the radar is 10Hz. If you want to obtain the data of the radar at a certain sampling moment, but the radar does not have the sampling data at this moment. Therefore, the perception data of these sensors can be unified in the time dimension through the data synchronization method provided in this embodiment, which is convenient for subsequent analysis and processing. In a multi-sensor system, achieving high-precision synchronization of data is crucial to improving the overall performance of the perception system. By providing an innovative multi-sensor time synchronization method based on interpolation, the time consistency of the sensor data fusion moment is ensured through refined algorithm processing, thereby significantly improving the accuracy of the fusion data.

[0035] In one embodiment, before executing step S210, the entire sensor system is first initialized, and the sensors participating in the synchronization (sensors to be synchronized) are initialized, such as cameras, radars, lidars and other sensors, and a globally unique identifier is assigned to each sensor participating in the synchronization, so that the source of different perception data can be confirmed after the subsequent processing is completed. After the initialization is completed, the data of each sensor to be synchronized (that is, the real data frame) is collected, and the timestamp of each data is recorded to obtain multiple real data frames collected by the sensor to be synchronized and the data collection time of each real data frame. The timestamp can be used as the data collection time of the corresponding real data frame. When storing data, the timestamp and the corresponding real data frame are stored in association.

[0036] As an example, the quantized data value of the real data frame collected by the sensor to be synchronized is a value that changes accordingly with the change of time. Within a preset time range, the change of the quantized data value can be proportional to the change of time. For example, if the position of a target object is collected by a camera, the quantized data value can be the distance of the target object from the vehicle. Due to the relative motion between the vehicle and the target object (the vehicle can move alone, the target object can move alone, the vehicle and the target object can move relative to or towards each other, etc.), the distance changes with the change of time.

[0037] As an example, among the multiple sensors to be synchronized, the real data frames collected by at least a part of the sensors to be synchronized represent the data of the same physical quantity collected by different sensors. For example, there are two sensors, camera and radar, among the multiple sensors to be synchronized, both of which monitor the distance between a target object X near the vehicle and the vehicle to obtain corresponding real data frames. It can be understood that the data acquisition objects corresponding to the real data frames collected by the multiple sensors to be synchronized can be of the same category (such as distance) or the same object (such as the target object of the aforementioned embodiment), or the data acquisition objects corresponding to the real data frames collected by at least part of the sensors to be synchronized can be of different categories or different objects. Those skilled in the art can determine the range of the corresponding sensors to be synchronized according to the actual needs of the controller.

[0038] Step S220: one sensor to be synchronized is determined as a main sensor, and other sensors to be synchronized except the main sensor are determined as auxiliary sensors.

[0039] As an example, one of the multiple sensors to be synchronized may be randomly determined as the main sensor, and the other sensors to be synchronized except the main sensor may be determined as auxiliary sensors.

[0040] As another example, one of the multiple sensors to be synchronized may be selected as the main sensor according to the designation of the controller or other data user, which can realize the need of supporting the user to independently select the main sensor.

[0041] As another example, the data sampling frequency of each sensor to be synchronized can be determined according to the data collection time, and the data sampling frequency of each sensor can be compared based on the pre-specified preset time frequency, and then the sensor to be synchronized corresponding to the closest data sampling frequency can be used as the main sensor. In this way, the customized requirements of the user of the synchronized data for the data sampling frequency can be met.

[0042] In one embodiment, determining a sensor to be synchronized as a master sensor includes: determining a data sampling frequency of each sensor to be synchronized according to a data collection time; and determining one of all sensors to be synchronized as the master sensor according to the data sampling frequency.

[0043] Following the above embodiment, one of the sensors to be synchronized is determined as the main sensor according to the data sampling frequency, including: if the number of sensors to be synchronized with the highest data sampling frequency is one, the sensor to be synchronized with the highest data sampling frequency is determined as the main sensor; and / or if the number of sensors to be synchronized with the highest data sampling frequency is greater than one, one of the multiple sensors to be synchronized with the highest data sampling frequency is determined as the main sensor. At this time, there may be multiple parallel sensors to be synchronized with the highest data sampling frequency, and one of them can be randomly selected as the main sensor, or these sensors to be synchronized can be sorted according to a preset selection priority, and then the sensor with the highest priority is selected as the main sensor.

[0044] As another example, the data collection moments of all sensors to be synchronized may be counted, and then the main sensor may be selected based on the degree of overlap of the data collection moments, so that the data collection moments of other auxiliary sensors are as close as possible to the data collection moments of the main sensor, thereby reducing the amount of subsequent data processing.

[0045] Some examples of methods for selecting the main sensor are provided above, and those skilled in the art may also set other main sensor selection rules as required.

[0046] By selecting the main sensor, the time point for subsequent data synchronization can be determined, which facilitates the subsequent execution of data synchronization.

[0047] Step S230: executing a data synchronization strategy for each real data frame of the auxiliary sensor, so that the real data frame of each auxiliary sensor is synchronized with the real data frame of the main sensor in the time dimension.

[0048] Among them, as an example, the data synchronization strategy includes: determining a time to be synchronized, the time to be synchronized is a time among the data collection times of the main sensor that is different from all the data collection times of an auxiliary sensor, determining at least one related time of each time to be synchronized from all the data collection times of an auxiliary sensor, determining a virtual data value of the next auxiliary sensor at a time to be synchronized based on the quantized data value of the real data frame of the next auxiliary sensor at all the related times corresponding to the time to be synchronized, and determining a virtual data frame of the next auxiliary sensor at a time to be synchronized based on the virtual data value, thereby obtaining a virtual data frame of the next auxiliary sensor at each time to be synchronized.

[0049] The data collection time of the real data frame collected by the main sensor may partially overlap (be the same) with the data collection time of the real data frame collected by the auxiliary sensor. These overlapping data collection times do not need to construct virtual data frames and can be used as synchronized times. Then, some data collection times of the real data frames collected by the main sensor may be partially different from the data collection time of the real data frames collected by the auxiliary sensor. In this case, these different data collection times of the main sensor are used as the time to be synchronized, and the virtual data frames of the auxiliary sensor at these time to be synchronized are subsequently calculated. For example, the data collection time of the main sensor is 01:11, 01:12, 01:13, 01:14, 01:15, 01:16, and the data collection time of a certain auxiliary sensor is 01:11, 01:13, 01:15, 01:17, 01:19. Then the time to be synchronized can be determined to be 01:12, 01:14, 01:16. The time to be synchronized determined by different auxiliary sensors may be at least partially the same, or may be completely different.

[0050] The number of relevant moments may be one or more. Of course, generally speaking, the closer the relevant moment is to the moment to be synchronized and the greater the number is, the better the reliability of the virtual data value finally determined. If the number of relevant moments is only one, the virtual data value can be determined by the theoretical numerical unit time increment pre-set by those skilled in the art and the time difference between the relevant moment and the moment to be synchronized. The virtual data value can also be determined by the strategy provided in the following embodiments, or by other methods known to those skilled in the art.

[0051] In one embodiment, at least one relevant moment of each time to be synchronized is determined from all the data acquisition moments of an auxiliary sensor, and the virtual data value of the next auxiliary sensor at the time to be synchronized is determined based on the quantized data value of the real data frame of the next auxiliary sensor at all the relevant moments corresponding to the time to be synchronized, including: respectively determining the time difference between all the data acquisition moments of an auxiliary sensor and each time to be synchronized; if the time difference is less than the preset difference threshold, determining a data acquisition moment with the smallest time difference as the relevant moment of the corresponding time to be synchronized; determining the quantized data value of the real data frame of the next auxiliary sensor at the relevant moment corresponding to the time to be synchronized as the virtual data value of the next auxiliary sensor at the time to be synchronized. For the case where the auxiliary sensor does not have high requirements for data smoothness, the nearest known point value is directly taken. For example, the auxiliary sensor is a millimeter wave radar. In the millimeter wave radar data, since its point cloud data is relatively sparse, this method can be used as a simple data filling method. The preset difference threshold can be set by those skilled in the art as needed.

[0052] In another embodiment, the data synchronization strategy can also be implemented by spline interpolation, which provides a smooth interpolation curve and is suitable for data with more complex changes. For example, in laser radar data processing, spline interpolation can be used to generate smooth point cloud data for subsequent data processing and analysis.

[0053] In view of the situation that there are data collection moments later than the time to be synchronized in the auxiliary sensor and there are data collection moments earlier than the time to be synchronized, in another embodiment, at least one relevant moment of each time to be synchronized is determined from all the data collection moments of an auxiliary sensor, and the virtual data value of the next auxiliary sensor at the time to be synchronized is determined based on the quantized data value of the real data frame of the next auxiliary sensor at all the relevant moments corresponding to the time to be synchronized, including: determining a data collection moment closest to the time to be synchronized among all the data collection moments of an auxiliary sensor less than the time to be synchronized as a relevant moment; determining a data collection moment closest to the time to be synchronized among all the data collection moments of an auxiliary sensor greater than the time to be synchronized as another relevant moment; determining the time weight of each relevant moment according to the time to be synchronized and each relevant moment; determining the virtual data value of the next auxiliary sensor at the time to be synchronized based on the time weight of each relevant moment and the quantized data value of the real data frame of the next auxiliary sensor at the corresponding relevant moment. The above-mentioned calculation method of the closest moment can be determined by time difference, and the smaller the time difference, the closer it is.

[0054] Continuing from the above embodiment, the moment weight of each relevant moment is determined according to a moment to be synchronized and each relevant moment, including: determining a first time difference between each relevant moment and a moment to be synchronized, and determining a second time difference between two relevant moments; determining the moment weight of each relevant moment based on the ratio between the first time difference and the second time difference between each relevant moment and a moment to be synchronized.

[0055] Continuing from the above embodiment, determining the virtual data value of the next auxiliary sensor at the time to be synchronized based on the moment weight of each relevant moment and the quantized data value of the real data frame of the next auxiliary sensor at the corresponding relevant moment includes: determining a first virtual data sub-value at a relevant moment according to the product between the moment weight of a relevant moment and the quantized data value of the real data frame of the next auxiliary sensor at the relevant moment; determining a second virtual data sub-value at another relevant moment according to the product between the moment weight of another relevant moment and the quantized data value of the real data frame of the next auxiliary sensor at the relevant moment; and obtaining the virtual data value of the next auxiliary sensor at the time to be synchronized based on the sum of the first virtual data sub-value and the second virtual data sub-value.

[0056] According to the time difference between the data collection time of the auxiliary sensor data frame and the data collection time of the main sensor, the weight of the virtual frame data can be dynamically allocated. Inconsistent time differences lead to inconsistent calculated weights, and finally the process of dynamic weight allocation is realized. The linear interpolation algorithm is used to estimate the data (virtual data frame) of the auxiliary sensor at the main sensor collection time (to be synchronized). The algorithm can calculate the interpolation result based on the previous and next frame data and their weights. It can be seen that this method can be applied to the case where the sampling frequencies of multiple sensors are inconsistent or there is a non-integer multiple relationship. This method can achieve timely and accurate sensor data synchronization.

[0057] In one embodiment, before executing the data synchronization strategy for each real data frame of the auxiliary sensor, the method further includes: if there are missing data frames in all the real data frames of the main sensor, completing the missing data frames.

[0058] Since the sensor may have frame loss and other problems during data collection and data transmission, resulting in the loss of data frames at a certain data collection time, after determining the main sensor, data collection based on the sensor is often performed at a certain frequency, and the interval time of its data collection time is often a fixed value. If the difference between two adjacent data collection times is significantly greater than the difference between other adjacent data collection times, then it means that there is a missing data frame between the two adjacent data collection times. Based on this principle, the data of the main sensor can be analyzed, and the existence of missing data frames can be determined through the overall data collection frequency, and data can be supplemented in time. For example, if the data collection time of a part of the main sensor is 02:01, 02:02, 02:03, 02:04, 02:06, and 02:07, then it can be clearly found that there is a missing data frame between 02:04 and 02:06.

[0059] The method of completing missing data can be implemented in a manner known to those skilled in the art, or can be implemented with reference to the above-mentioned data synchronization strategy. For example, taking the above-mentioned embodiment as an example, if there is a missing data frame between 02:04 and 02:06, then the virtual data value corresponding to 02:05 can be determined by the average value of the quantized data values ​​corresponding to 02:04 and 02:06 (the two adjacent data collection moments corresponding to the missing data frame). Then, based on the above-determined virtual data value and the theoretical data collection moment 02:05, a missing data frame is constructed to complete the missing data. If two consecutive missing data frames are missing between two adjacent data collection moments, the virtual data value can also be determined in proportion (at this time, it can be regarded that the quantized data value changes linearly with time).

[0060] In one embodiment, after executing the data synchronization strategy on the real data frame of an auxiliary sensor, the method further includes: determining a virtual data evaluation parameter of an auxiliary sensor according to the quantized data values ​​of the virtual data frames of all the to-be-synchronized moments of an auxiliary sensor after executing the data synchronization strategy and the quantized data values ​​of at least part of the real data frames of an auxiliary sensor; determining the data synchronization strategy execution status of an auxiliary sensor based on the virtual data evaluation parameter; if the data synchronization strategy execution status is failure, adjusting the weights corresponding to each relevant moment (such as the moment weights of the above embodiment) (if there are multiple relevant moments corresponding to the same to-be-synchronized moment, the adjustment can be achieved by increasing the moment weights of some relevant moments and reducing the time weights of other relevant moments, or by other adjustment rules known to those skilled in the art), and re-executing the data synchronization strategy on the real data frame of an auxiliary sensor based on the adjusted weights corresponding to each relevant moment, until the data synchronization strategy execution success status is changed to success.

[0061] The virtual data evaluation parameter is used to characterize the dispersion of all data of the auxiliary sensor (the quantized data values ​​corresponding to the real data frame and the quantized data values ​​corresponding to the virtual data frame). As an example, the virtual data evaluation parameter can be characterized by one or more data such as mean, variance, standard deviation, average gradient, etc.

[0062] When calculating the virtual data evaluation parameters, one or more parameters such as mean, variance, standard deviation, average gradient, etc. may be determined based on the quantized data values ​​of the real data frames of all the auxiliary sensors and the quantized data values ​​of the obtained virtual data frames as the virtual data evaluation parameters. One or more parameters such as mean, variance, standard deviation, average gradient, etc. may also be determined based on the quantized data values ​​of the real data frames collected by the auxiliary sensors at the same data collection time as the data collection time of the main sensor and the quantized data values ​​of the obtained virtual data frames as the virtual data evaluation parameters.

[0063] Continuing from the above embodiment, the data synchronization policy execution status of an auxiliary sensor is determined based on the virtual data evaluation parameter, including: if the virtual data evaluation parameter is greater than the preset evaluation parameter threshold, the data synchronization policy execution status is determined as failure; if the virtual data evaluation parameter is less than or equal to the preset evaluation parameter threshold, the data synchronization policy execution status is determined as success.

[0064] The above method can be used to evaluate the accuracy of the fused data (synchronized data) to ensure that the time synchronization meets the system requirements. If the accuracy of the fused data does not meet expectations, adjust the parameters of the interpolation algorithm (the weights mentioned above), such as the interpolation method or the weight distribution strategy, and continue to iterate to meet the corresponding requirements. By comparing the difference between the fused data and the benchmark data, the error is calculated, including position error, direction error, and size error. Then, in the target detection and target tracking index evaluation, the detection rate, tracking stability and other parameters of the fused data are estimated to achieve the evaluation of detection and tracking performance. For example, the quality of the fused image can be evaluated by objective evaluation indicators such as the mean, standard deviation, average gradient and other parameters of the image.

[0065] See also Figure 3 , Figure 3 A specific flow chart of the multi-sensor data synchronization method provided in the embodiment of the present application is as follows: Figure 3As shown, a specific process of an example is: first, the system is initialized and an identifier is assigned, for example, the entire sensor system is initialized, and the sensors participating in the synchronization are initialized, such as cameras, radars, laser radars and other sensors, and a globally unique identifier is assigned to each sensor participating in the synchronization (sensor to be synchronized); after the initialization is completed, the data is collected and the timestamp is recorded, for example, the data of each sensor (real data frame) is collected, and the timestamp of each data (data collection time) is recorded. Then the main sensor is selected to obtain the timestamp of the main sensor data set, for example, the sensor with a higher sampling frequency among all sensors is selected as the main sensor, and the sampling frequency of the sensor is used as a reference. Then the auxiliary sensor (also known as the auxiliary sensor) is selected, and the front and back frame data of the auxiliary sensor are indexed. For example, after the auxiliary sensor is selected, for each data collection of the main sensor, the two data frames closest to the auxiliary sensor before and after the moment are found and the data collection time (related time) of the two closest data frames is based on the time. Then the time difference and weight are calculated, for example, the time difference between the main sensor data collection time is calculated. According to the above-mentioned time difference, weights are assigned to the front and back frames of the auxiliary sensor to facilitate the subsequent interpolation calculation. Then, a linear difference calculation is performed to form a virtual frame, for example, linear interpolation or other advanced interpolation algorithms are applied, and a virtual data frame corresponding to the data acquisition time of the main sensor is generated according to the calculated weight, and the interpolated virtual data frame is fused with the data frame of the main sensor to form a data set with a unified time reference. The main function of the weight is to weighted average the previous and next frame data of the auxiliary sensor to generate a virtual data frame, which is aligned with the data frame of the main sensor in time. As an example, the process of obtaining the virtual frame target information is: by obtaining the timestamps of the main sensor and the auxiliary sensor, and then calculating the timestamp difference between the timestamp of the main sensor data frame and the timestamp before and after the auxiliary sensor data frame, and then using the motion information contained in the auxiliary sensor data frame, such as the quantized data values ​​of speed and acceleration, combined with the time difference and motion information, the position of each target in the auxiliary sensor at the time of the main sensor data acquisition is calculated, and then a new virtual frame is established between the original two frames according to the calculated result. The virtual frame of the auxiliary sensor sampling period on the main sensor is obtained according to the time difference and motion information, and then the data frames of the two sensors are preprocessed, and the data is fused by weighted average algebraic method or transformation.

[0066] Then, data fusion is performed to evaluate the accuracy of the fused data, such as performing an accuracy assessment on the fused data to ensure that time synchronization meets the system requirements. If the accuracy of the fused data does not meet expectations, adjust the parameters of the interpolation algorithm (iterative optimization of the difference parameters), such as the interpolation method or the weight distribution strategy, and continue to iterate to meet the corresponding requirements. An exemplary implementation of the accuracy assessment is: first, the error is calculated by comparing the difference between the fused data and the benchmark data, including position error, direction error, and size error. Then, in the target detection and target tracking index evaluation, the detection rate, tracking stability and other parameters of the fused data are estimated to achieve the evaluation of the detection and tracking performance. Finally, the quality of the fused image is evaluated by objective evaluation indicators such as the mean, standard deviation, average gradient and other parameters of the image. If the accuracy assessment passes, the synchronization calibration is completed.

[0067] See also Figure 4 , Figure 4 A specific flow chart of a process for determining a virtual data frame to be synchronized at a time provided in an embodiment of the present application is as follows: Figure 4 As shown in the figure, it is assumed that there are two sensors in the multi-sensor system, namely, camera and radar. The acquisition frequency of the camera is 20Hz, and the acquisition frequency of the radar is 10Hz. If you want to obtain the data of the radar obtained by the camera at a certain sampling time, but the radar does not have the sampling data at this time. For such a scenario, you can obtain the camera sampling time sync_time and the radar front and back timestamps front_time and back_time, calculate the difference between the radar front and back timestamps, calculate the weight (moment weight) according to the time difference, and obtain the virtual frame by linear difference according to the weight. For example, you can collect data from the camera and radar and record the timestamps, where sync_time is the time when the camera data is collected, and it is also the time when you want to estimate the auxiliary sensor data. front_time is the timestamp of the most recent data frame of the auxiliary sensor before sync_time. back_time is the timestamp of the most recent data frame of the auxiliary sensor after sync_time. Calculate the difference between the data frame timestamps of sync_time and the front and back data of the auxiliary sensor:

[0068] Δt_front = sync_time - front_time formula (1);

[0069] Δt_back = back_time - sync_time formula (2);

[0070] Among them, Δt_front is the first time difference, Δt_back is the second time difference, sync_time is the time when the camera collects data (that is, the time to be synchronized), front_time is the timestamp of the most recent data frame of the auxiliary sensor before sync_time (that is, a relevant moment), and back_time is the timestamp of the most recent data frame of the auxiliary sensor after sync_time (that is, another relevant moment).

[0071] The moment weights are calculated based on the time difference. These moment weights will be used for the weighted average of the previous and next data frames in the interpolation calculation. The moment weight calculation method is as follows:

[0072] front_scale = Δt_back / (Δt_front + Δt_back) formula (4);

[0073] back_scale = Δt_front / (Δt_front + Δt_back) formula (5);

[0074] Among them, front_scale is the moment weight of one relevant moment, back_scale is the moment weight of another relevant moment, Δt_front is the first time difference, and Δt_back is the second time difference.

[0075] According to the obtained time difference and weight, linear interpolation is performed to obtain:

[0076] synced_data = front_data × front_scale + back_data × back_scale formula (6);

[0077] Among them, front_data is the quantized data value of the data frame of the auxiliary sensor at the time of front_time, back_data is the quantized data value of the data frame of the auxiliary sensor at the time of back_time. Synced_data is the quantized data value of the virtual data frame estimated at the time of sync_time, which is aligned with the timestamp of the main sensor data.

[0078] Through this process, the sensor data at a specific moment can be estimated without a precise synchronization mechanism, thereby achieving time synchronization and fusion of multi-sensor data.

[0079] See also Figure 5 , Figure 5 A schematic diagram of a data frame for completing data synchronization of two sensors provided in an embodiment of the present application is shown in FIG. Figure 5As shown, taking the sensors to be synchronized as a camera and a radar as an example, the camera collects camera frames (that is, camera data frames), and the radar collects radar frames. The acquisition frequency of radar frames (that is, radar data frames) is relatively low. The multi-sensor data synchronization method provided in the above embodiment can be used to generate multiple virtual frames as shown in the figure with the camera as the main sensor and the radar as the auxiliary sensor, and complete them into the original radar frames to obtain synchronized radar frames.

[0080] See also Figure 6 and Figure 7 , Figure 6 for Figure 5 A schematic diagram of a camera data frame provided in the embodiment, Figure 7 for Figure 5 A schematic diagram of a radar data frame provided in the embodiment, such as Figure 6 and Figure 7 As shown, t3 is one of the synchronization moments ( Figure 6 The left and right sides of t3 in the figure correspond to t1 and t2 respectively, so they can be regarded as synchronized moments). To determine the virtual radar data frame at moment t3, t1 and t2 can be used as two related moments, and then two time differences t13 and t23 can be determined. Then, the moment weights can be determined in the manner provided in the aforementioned embodiment to obtain the virtual radar data frame at moment t3.

[0081] Through precise linear interpolation, multiple sensors are aligned on the same timestamp dimension, which improves the accuracy of data fusion and lays a solid foundation for improving the effectiveness of decision-making. The method provided in this embodiment can be implemented by algorithm-based software synchronization, which does not rely on external synchronization signals or devices. Therefore, even when the external signal is unstable or unavailable, synchronization between multiple sensors can still be achieved according to the internal algorithm. This adaptive time synchronization mechanism improves the stability and reliability of the system in various environments. The linear interpolation algorithm execution process is simple and easy to implement, and can be quickly deployed in practice, ensuring the real-time and synchronization of sensor data, which is conducive to the system to respond accurately and timely. Time synchronization is achieved through software algorithms, avoiding additional hardware investment and complex synchronization equipment, which not only saves deployment costs, but also simplifies system design, providing an effective solution for fields that require rapidity, such as autonomous driving and robot navigation.

[0082] The above-mentioned multi-sensor data synchronization method obtains the real data frames and data collection times of multiple sensors to be synchronized in the vehicle within the time period to be synchronized, takes one sensor to be synchronized as the main sensor and the rest as auxiliary sensors, determines the time to be synchronized in the data collection time of the main sensor corresponding to each auxiliary sensor, determines at least one related time of each time to be synchronized from all the data collection times of the auxiliary sensors, determines the virtual data value of the auxiliary sensor at the corresponding time to be synchronized based on the quantized data value of the real data frame of the auxiliary sensor at the related time, and then obtains the virtual data frame of the auxiliary sensor at each time to be synchronized, so that the real data frame of each auxiliary sensor is synchronized with the real data frame of the main sensor in the time dimension. This method can be implemented by an algorithm and has relatively low requirements on sensor performance.

[0083] In one embodiment, a multi-sensor data synchronization device is provided, and the multi-sensor data synchronization device is used to perform the steps in the multi-sensor data synchronization method provided in any of the above embodiments. Figure 8 , Figure 8 A schematic diagram of the structure of a multi-sensor data synchronization device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the multi-sensor data synchronization device 800 includes an acquisition module 801, a main and auxiliary sensor determination module 802 and a synchronization module 803, wherein: the acquisition module 801 is used to acquire multiple real data frames collected by the sensors to be synchronized in the vehicle within the synchronization time period and the data acquisition time of each real data frame, and the number of sensors to be synchronized is at least two; the main and auxiliary sensor determination module 802 is used to determine a sensor to be synchronized as a main sensor, and determine the other sensors to be synchronized except the main sensor as auxiliary sensors; the synchronization module 803 is used to execute the data synchronization strategy for each real data frame of the auxiliary sensor, so that the real data frames of each auxiliary sensor The data frame is synchronized with the real data frame of the main sensor in the time dimension, wherein the data synchronization strategy includes: determining a time to be synchronized, the time to be synchronized is a time in the data collection time of the main sensor that is different from all the data collection times of an auxiliary sensor, determining at least one related time of each time to be synchronized from all the data collection times of an auxiliary sensor, determining a virtual data value of the next auxiliary sensor at a time to be synchronized based on the quantized data value of the real data frame of the next auxiliary sensor at all the related times corresponding to the time to be synchronized, and determining a virtual data frame of the next auxiliary sensor at a time to be synchronized based on the virtual data value, thereby obtaining a virtual data frame of the next auxiliary sensor at each time to be synchronized.

[0084] In one embodiment, the primary and secondary sensor determination module is configured to: determine the data sampling frequency of each sensor to be synchronized according to the data collection time; if the number of sensors to be synchronized with the highest data sampling frequency is one, determine the sensor to be synchronized with the highest data sampling frequency as the primary sensor; if the number of sensors to be synchronized with the highest data sampling frequency is greater than one, determine one of the multiple sensors to be synchronized with the highest data sampling frequency as the primary sensor.

[0085] In one embodiment, the synchronization module is configured to: determine a data collection moment that is closest to a time to be synchronized among all data collection moments of an auxiliary sensor that is less than a time to be synchronized as a relevant moment; determine a data collection moment that is closest to a time to be synchronized among all data collection moments of an auxiliary sensor that is greater than a time to be synchronized as another relevant moment; determine a time weight of each relevant moment according to a time to be synchronized and each relevant moment; determine a virtual data value of the next auxiliary sensor at the time to be synchronized based on the time weight of each relevant moment and the quantized data value of the real data frame of the next auxiliary sensor at the corresponding relevant moment.

[0086] In one embodiment, the synchronization module is further configured to: determine a first time difference between each relevant moment and a moment to be synchronized, and determine a second time difference between two relevant moments; determine the moment weight of each relevant moment based on the ratio between the first time difference and the second time difference between each relevant moment and a moment to be synchronized.

[0087] In one embodiment, the synchronization module is further configured to: determine a first virtual data sub-value at a relevant moment according to the product between a moment weight at a relevant moment and a quantized data value of a real data frame of a next auxiliary sensor at the relevant moment; determine a second virtual data sub-value at another relevant moment according to the product between a moment weight at another relevant moment and a quantized data value of a real data frame of a next auxiliary sensor at another relevant moment; and obtain a virtual data value of a next auxiliary sensor at the moment to be synchronized based on the sum of the first virtual data sub-value and the second virtual data sub-value.

[0088] In another embodiment, the synchronization module is configured to: respectively determine the time differences between all data collection moments of an auxiliary sensor and each time to be synchronized; if the time difference is less than a preset difference threshold, determine the data collection moment with the smallest time difference as the relevant time of the corresponding time to be synchronized; determine the quantized data value of the real data frame of the next auxiliary sensor at the relevant time corresponding to the time to be synchronized as the virtual data value of the next auxiliary sensor at the time to be synchronized.

[0089] In one embodiment, the device also includes an evaluation module, which is used to determine a virtual data evaluation parameter of an auxiliary sensor according to the quantized data values ​​of the virtual data frame of the auxiliary sensor at all the synchronization moments after the data synchronization strategy is executed and the quantized data values ​​of at least part of the real data frame of the auxiliary sensor; determine the data synchronization strategy execution status of the auxiliary sensor based on the virtual data evaluation parameter; if the data synchronization strategy execution status is failure, adjust the weights corresponding to each relevant moment, and re-execute the data synchronization strategy on the real data frame of the auxiliary sensor based on the adjusted weights corresponding to each relevant moment, until the data synchronization strategy execution success status is changed to success.

[0090] For the specific definition of the multi-sensor data synchronization device, please refer to the definition of the multi-sensor data synchronization method above, which will not be repeated here. Each module in the above-mentioned multi-sensor data synchronization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory of the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0091] In this embodiment, the multi-sensor data synchronization device is essentially provided with multiple modules for executing the steps in the multi-sensor data synchronization method in any of the above embodiments. The specific functions and technical effects can be referred to the above embodiments and will not be repeated here.

[0092] In one embodiment, a vehicle is provided, the vehicle comprising the multi-sensor data synchronization device provided by any of the above embodiments and a plurality of sensors to be synchronized. The specific functions and technical effects of the vehicle can refer to the above embodiments, and will not be described in detail here.

[0093] See also Fig. 9 An embodiment of the present invention further provides an electronic device 900, including a processor 901, a memory 902 and a communication bus 903; the communication bus 903 is used to connect the processor 901 and the memory 902; the processor 901 is used to execute the computer program stored in the memory 902 to implement any method provided in the above embodiments.

[0094] As an example, the electronic device may be a device such as a vehicle or a vehicle computer.

[0095] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. The computer program is used to enable a computer to execute a method provided in any one of the above embodiments.

[0096] The embodiment of the present application also provides a non-volatile readable storage medium, which stores one or more modules (programs). When the one or more modules are applied to a device, the device can execute instructions (instructions) of the steps included in the embodiment 1 of the embodiment of the present application.

[0097] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0098] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0099] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0100] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0101] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0102] It should be understood that the terms "first", "second", etc. mentioned in this application are used to distinguish similar objects, and do not necessarily represent that they refer to a specific order or sequence. The technical features in which these terms are located can be interchanged under appropriate circumstances, so that the embodiments of the application described here can be implemented in an order other than the illustrated or written description.

[0103] It should be understood that although the flowcharts provided in the embodiments of the present application indicate the steps by arrows, the order indicated by the arrows does not necessarily limit the order in which the steps are implemented. Those skilled in the art can perform the steps in other orders according to the requirements for different implementation scenarios.

[0104] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A multi-sensor data synchronization method, characterized in that: The multi-sensor data synchronization method comprises: Acquire multiple real data frames collected by sensors to be synchronized in the vehicle within a time period to be synchronized and the data collection time of each real data frame, wherein the number of sensors to be synchronized is at least two; Determine a sensor to be synchronized as a main sensor, and determine other sensors to be synchronized except the main sensor as auxiliary sensors; A data synchronization strategy is executed for each real data frame of each auxiliary sensor respectively, so that the real data frame of each auxiliary sensor is synchronized with the real data frame of the main sensor in the time dimension, wherein the data synchronization strategy includes: determining a time to be synchronized, the time to be synchronized is a time among the data collection times of the main sensor that is different from all the data collection times of an auxiliary sensor, determining at least one relevant time of each time to be synchronized from all the data collection times of the auxiliary sensor, determining a virtual data value of the auxiliary sensor at the time to be synchronized based on the quantized data value of the real data frame of the auxiliary sensor at all the relevant times corresponding to the time to be synchronized, and determining a virtual data frame of the auxiliary sensor at the time to be synchronized based on the virtual data value, thereby obtaining a virtual data frame of the auxiliary sensor at each time to be synchronized.

2. The multi-sensor data synchronization method according to claim 1, characterized in that: Determine a sensor to be synchronized as the master sensor, including: Determine the data sampling frequency of each sensor to be synchronized according to the data collection time; If the number of the to-be-synchronized sensor with the highest data sampling frequency is one, the to-be-synchronized sensor with the highest data sampling frequency is determined as the main sensor; If the number of sensors to be synchronized with the highest data sampling frequency is greater than one, one of the multiple sensors to be synchronized with the highest data sampling frequency is determined as the master sensor.

3. The multi-sensor data synchronization method according to claim 1, characterized in that: Determining at least one relevant time of each to-be-synchronized time from all data collection times of the one auxiliary sensor, and determining a virtual data value of the one auxiliary sensor at the to-be-synchronized time based on quantized data values ​​of real data frames of the one auxiliary sensor at all relevant times corresponding to the one to-be-synchronized time, comprising: Determine a data collection time closest to the time to be synchronized among all data collection times of the auxiliary sensor that are less than the time to be synchronized as a relevant time; Determine a data collection time closest to the time to be synchronized among all data collection times of the auxiliary sensor that are greater than the time to be synchronized as another relevant time; Determine a time weight of each relevant time according to the to-be-synchronized time and each of the relevant times; The virtual data value of the auxiliary sensor at the to-be-synchronized time is determined based on the time weight of each relevant time and the quantized data value of the real data frame of the auxiliary sensor at the corresponding relevant time.

4. The multi-sensor data synchronization method according to claim 3, characterized in that: Determining the time weight of each relevant time according to the to-be-synchronized time and each of the relevant times includes: Determine a first time difference between each of the relevant moments and the one to-be-synchronized moment, and determine a second time difference between two of the relevant moments; The time weight of each relevant moment is determined based on the ratio between the first time difference and the second time difference between each relevant moment and the one to-be-synchronized moment.

5. The multi-sensor data synchronization method according to claim 3, characterized in that: Determining a virtual data value of the auxiliary sensor at the to-be-synchronized time based on a time weight of each relevant time and a quantized data value of a real data frame of the auxiliary sensor at the corresponding relevant time includes: Determine a first virtual data sub-value at a relevant moment according to a product of a moment weight at a relevant moment and a quantized data value of a real data frame of the auxiliary sensor at the relevant moment; Determine a second virtual data sub-value at another relevant moment according to a product of a moment weight at another relevant moment and a quantized data value of a real data frame of the auxiliary sensor at the another relevant moment; The virtual data value of the auxiliary sensor at the to-be-synchronized time is obtained based on the sum of the first virtual data sub-value and the second virtual data sub-value.

6. The multi-sensor data synchronization method according to claim 1, characterized in that: Determining at least one relevant time of each to-be-synchronized time from all data collection times of the one auxiliary sensor, and determining a virtual data value of the one auxiliary sensor at the to-be-synchronized time based on quantized data values ​​of real data frames of the one auxiliary sensor at all relevant times corresponding to the one to-be-synchronized time, comprising: respectively determining the time difference between all data collection moments of the auxiliary sensor and each time to be synchronized; If the time difference is less than the preset difference threshold, a data collection moment with the smallest time difference is determined as the relevant moment of the corresponding moment to be synchronized; The quantized data value of the real data frame of the auxiliary sensor at a relevant time corresponding to a to-be-synchronized time is determined as the virtual data value of the auxiliary sensor at the to-be-synchronized time.

7. The data synchronization method according to any one of claims 1 to 6, characterized in that: Before respectively executing the data synchronization strategy on the real data frame of each auxiliary sensor, the method further includes: If there are missing data frames in all the real data frames of the main sensor, the missing data frames are supplemented with missing data.

8. The data synchronization method according to any one of claims 1 to 6, characterized in that: After executing the data synchronization strategy on the real data frame of an auxiliary sensor, the method further includes: Determine a virtual data evaluation parameter of the one auxiliary sensor according to quantized data values ​​of the virtual data frames of the one auxiliary sensor at all the to-be-synchronized moments after executing the data synchronization strategy and quantized data values ​​of at least part of the real data frames of the one auxiliary sensor; determining a data synchronization strategy execution state of the one auxiliary sensor based on the virtual data evaluation parameter; If the data synchronization strategy execution status is failure, the weights corresponding to each relevant moment are adjusted, and the data synchronization strategy is re-executed on the real data frame of the auxiliary sensor based on the adjusted weights corresponding to each relevant moment until the data synchronization strategy execution success status is changed to success.

9. A multi-sensor data synchronization device, characterized in that: The multi-sensor data synchronization device comprises: An acquisition module, used to acquire a plurality of real data frames acquired by a sensor to be synchronized in the vehicle within a time period to be synchronized and a data acquisition time of each real data frame, wherein the number of the sensors to be synchronized is at least two; A main sensor determination module, used to determine a sensor to be synchronized as a main sensor, and determine other sensors to be synchronized except the main sensor as auxiliary sensors; A synchronization module is used to execute a data synchronization strategy for each real data frame of each auxiliary sensor respectively, so that the real data frame of each auxiliary sensor is synchronized with the real data frame of the main sensor in the time dimension, wherein the data synchronization strategy includes: determining a time to be synchronized, the time to be synchronized is a time among the data collection times of the main sensor that is different from all the data collection times of an auxiliary sensor, determining at least one relevant time of each time to be synchronized from all the data collection times of the auxiliary sensor, determining a virtual data value of the auxiliary sensor at the time to be synchronized based on the quantized data value of the real data frame of the auxiliary sensor at all the relevant times corresponding to the time to be synchronized, and determining a virtual data frame of the auxiliary sensor at the time to be synchronized based on the virtual data value, thereby obtaining a virtual data frame of the auxiliary sensor at each time to be synchronized.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 8 is implemented.

Images