Multi-sensor data synchronization method and apparatus, and device and medium

Abstract

Provided in the present application are a multi-sensor data synchronization method and apparatus, and an electronic device and a computer-readable storage medium. The method comprises: storing, in a respective current data queue of each sensor, sensor data in a respective receiving queue of each sensor according to the respective receiving order of the sensor data; when none of the current data queues of sensors is empty, copying sensor data that arrives the earliest in each current data queue to a candidate queue; using as a pivot a sensor name that corresponds to sensor data that arrives the latest in each current data queue, and using as a pivot time the latest arrival timestamp of sensor data in the current data queue for the pivot; storing, in a respective historical data queue of each sensor, the sensor data that arrives the earliest in each current data queue; and if the sensor data for the pivot has become the current data that arrives the earliest, publishing sensor data in the candidate queue as synchronized sensor data. The present application can ensure the high efficiency and accuracy of time synchronization of multi-sensor data.

Description

Multi-sensor data synchronization methods, devices, equipment and media Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a method, apparatus, device and medium for data synchronization of multiple sensors. Background Technology

[0002] Autonomous vehicles typically require multiple sensors, such as cameras, LiDAR, or 4D millimeter-wave radar, to collect data about the surrounding environment. These sensors are then fused together to obtain a comprehensive understanding of the environment. Because this process involves fusing data from multiple sensors, the accuracy of time synchronization between these sensors becomes crucial to ensuring the accuracy of the perception results.

[0003] However, each sensor on the vehicle triggers data acquisition with a link delay, and there is also a time delay in the process of each sensor transmitting data to the autonomous driving system. This causes the autonomous driving system to receive data sent by different sensors at inconsistent times, or a temporary malfunction of a certain sensor may cause a time misalignment, which may lead to the autonomous driving system failing to work properly.

[0004] Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for synchronizing data from multiple sensors in time without interfering with multiple sensors, and ensuring the efficiency and accuracy of the time synchronization of multi-sensor data.

[0006] Firstly, this application provides a multi-sensor data synchronization method, including:

[0007] The sensor data in the receiving queue of each sensor is stored into the current data queue of each sensor in the order of receiving.

[0008] If none of the current data queues are empty, then copy the earliest arriving sensor data from each current data queue to the candidate queue.

[0009] The sensor name corresponding to the latest arriving sensor data in each of the current data queues is taken as the hub, and the latest arrival timestamp of the sensor data in the current data queue of the hub is taken as the hub time.

[0010] The earliest arriving sensor data in each current data queue is stored in the respective historical data queue of each sensor;

[0011] If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time, then the sensor data in the candidate queue will be published as the synchronized sensor data.

[0012] Secondly, this application provides a multi-sensor data synchronization device, comprising:

[0013] The data acquisition module is used to store the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to their respective receiving order;

[0014] The candidate acquisition module is used to copy the earliest arriving sensor data from each current data queue to the candidate queue when each current data queue is not empty.

[0015] The hub determination module is used to take the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and take the latest arrival timestamp of the sensor data in the current data queue of the hub as the hub time.

[0016] The historical data storage module is used to store the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor.

[0017] The matching processing module is used to publish the sensor data in the candidate queue as synchronized sensor data if the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time.

[0018] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-sensor data synchronization method as described in any of the embodiments of this application.

[0019] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the multi-sensor data synchronization method as described in any of the embodiments of this application.

[0020] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-sensor data synchronization method as described in any of the embodiments of this application.

[0021] The multi-sensor data synchronization method, apparatus, device, and medium provided in this application involve each sensor receiving sensor data through its own thread, and a separate thread for data synchronization. Specifically, sensor data is first retrieved from the receiving queue of each sensor and stored in each sensor's current data queue. Then, the earliest arriving sensor data in each current data queue is copied to a pre-established candidate queue. The sensor name corresponding to the latest arriving sensor data in each current data queue is designated as the hub, and the latest arrival timestamp of the sensor data in the hub's current data queue is designated as the hub time. The earliest arriving sensor data in each current data queue is then stored in each sensor's historical data queue. Next, a time-matching check is performed on the sensor data in the candidate queues. If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time, a match is successful, and the sensor data in the candidate queues is published as the synchronized sensor data. Therefore, the technical solution of this application can synchronize the time of multi-sensor data without interfering with the multi-sensor data, make up for the delay caused by data transmission of each sensor, and ensure the efficiency and accuracy of multi-sensor data time synchronization. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0023] Figure 1 is a flowchart of a multi-sensor data synchronization method provided in an embodiment of this application;

[0024] Figure 2 is a flowchart of a multi-sensor data synchronization method provided in an embodiment of this application;

[0025] Figure 3 is a schematic diagram of the current data queue in the multi-sensor data synchronization method provided in the embodiments of this application;

[0026] Figure 4 is a schematic diagram of the candidate queue in the multi-sensor data synchronization method provided in the embodiments of this application;

[0027] Figure 5 is a schematic diagram of the overall logic of the multi-sensor data synchronization method provided in the embodiment of this application;

[0028] Figure 6 is a schematic diagram of the main flow of the time synchronization thread of the multi-sensor data synchronization method provided in the embodiments of this application;

[0029] Figure 7 is a schematic diagram of the time synchronization processing flow of the multi-sensor data synchronization method provided in the embodiments of this application;

[0030] Figure 8 is a schematic diagram of the structure of the multi-sensor data synchronization device provided in an embodiment of this application;

[0031] Figure 9 is a schematic diagram of the structure of the electronic device provided in the embodiment of this application.

[0032] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0033] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0034] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first" and "second" are used only to distinguish descriptions and should not be construed as indicating or implying relative importance. The acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations.

[0035] Figure 1 is a flowchart of a multi-sensor data synchronization method provided in an embodiment of this application. This embodiment is applicable to the time synchronization of data from multiple sensors in autonomous vehicles to ensure the efficient operation of the vehicle's perception system. This method can be executed by a multi-sensor data synchronization device, which can be implemented in software and / or hardware, preferably configured in an electronic device, such as an autonomous vehicle or a server. As shown in Figure 1, the method specifically includes:

[0036] S101. Store the sensor data in each sensor's receiving queue into each sensor's current data queue according to their respective receiving order.

[0037] S102. When all current data queues are not empty, copy the earliest arriving sensor data from each current data queue to the candidate queue.

[0038] S103. Take the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and take the latest arrival timestamp of the sensor data in the current data queue of the hub as the hub time.

[0039] S104. Store the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor.

[0040] S105. If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time, then the sensor data in the candidate queue will be published as the synchronized sensor data.

[0041] Autonomous vehicles typically include multiple sensors, such as cameras, LiDAR, or 4D millimeter-wave radar. These sensors collect data about the surrounding environment, and then the data is fused and used to guide the vehicle's operation. Therefore, the accuracy of multi-sensor time synchronization is a crucial factor in ensuring the accuracy of the perception results.

[0042] In this embodiment, each sensor corresponds to a thread for acquiring and receiving sensor data. The acquired sensor data is placed into its respective receiving queue. Simultaneously, a dedicated thread is also started for data synchronization processing. When the receiving queue is not empty, the sensor's thread notifies the data synchronization processing thread, which then stores the sensor data in the receiving queue into the current data queue corresponding to each sensor according to their respective receiving order. Figure 3 is a schematic diagram of the current data queue in the multi-sensor data synchronization method provided in this embodiment. The figure schematically shows the current data queues corresponding to sensors 1, 2, and 3, with sensor data stored in each current data queue according to the receiving order.

[0043] When all current data queues are not empty, the earliest arriving sensor data in each current data queue is copied to the candidate queue. If the sensor data in the candidate queue is time-synchronized, then the data in the candidate queue can be published to the sensing system for fusion sensing. If the sensor data in the candidate queue is not time-synchronized, then candidates need to be acquired again. Figure 4 is a schematic diagram of the candidate queue in the multi-sensor data synchronization method provided in this application embodiment. Taking the above three sensors as an example, data 1-1, 2-1, and 3-1 located at the head of the current data queues of sensors 1, 2, and 3 are obtained as the earliest arriving sensor data and placed into the candidate queue.

[0044] In this embodiment, the method used to determine whether the data in the candidate queue is time-matched synchronized data is as follows: The sensor name corresponding to the latest arriving sensor data in each current data queue is taken as the hub, and the latest arrival timestamp of the sensor data in the hub's current data queue is taken as the hub time. The earliest arriving sensor data in each current data queue is stored in each sensor's respective historical data queue. If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time, then the sensor data in the candidate queue is published as synchronized sensor data. If publication is successful, the data in the historical data queue is deleted; if publication fails, the data in the historical data queue is returned to the current data queue to re-obtain candidates.

[0045] The data synchronization method of this application embodiment can be executed cyclically at a certain frequency. If no sensor experiences delays in data acquisition or transmission, under normal circumstances, each time the data synchronization method is executed, each sensor's current data queue will contain one sensor data point. These sensor data points are also copied to the candidate queue as the earliest arriving sensor data in each current data queue. Simultaneously, these candidates are time-matched synchronization data and will be successfully published. However, if any sensor experiences a delay, the number of data points in each sensor's current data queue will be different. Furthermore, during the cyclic execution of the data synchronization method, sensor data is continuously acquired and received according to their respective acquisition frequencies, and the amount of data in the current data queue gradually increases. Therefore, it is necessary to determine and process whether the candidate data is time-matched in abnormal situations.

[0046] Specifically, on the one hand, if the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, meaning the hub's message has become the earliest arriving data, this indicates that the current hub's sensor data has not matched the sensor data of the previous frame, but can match subsequent sensor data. Since there is no data in the current data queue that can match the data in the candidate queue in terms of time, the sensor data in the current candidate queue is the optimal time-synchronized data and can therefore be published as synchronized sensor data. On the other hand, if the hub's message is not the earliest arriving data, but other sensor data arrives in each current data queue before and after the hub's time, this means that the sensor data corresponding to the current hub has been proven to be optimal in terms of time matching, meaning the data in the current candidate queue is also the optimal time-synchronized data and can be published.

[0047] In summary, the technical solution of this application embodiment involves each sensor receiving sensor data through its own thread. Furthermore, a separate thread is opened for data synchronization. Therefore, this application does not interfere with the data acquisition and reception of each sensor. After obtaining candidates, this application determines whether the sensor data in the candidate queue matches in time. If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub's time, then the match is successful, and the sensor data in the candidate queue is published as the synchronized sensor data. Thus, the technical solution of this application can synchronize the time of multiple sensor data without interfering with multiple sensors, compensating for delays caused by data transmission between sensors. It also does not require a strict order of arrival for each sensor, thus accommodating slight delays in some sensors at certain times. This ensures the efficiency and accuracy of multi-sensor data time synchronization, guarantees the efficient operation of the perception system, and improves the driving safety of autonomous vehicles.

[0048] Figure 2 is a flowchart of a multi-sensor data synchronization method provided in an embodiment of this application. This embodiment further optimizes the above embodiment. As shown in Figure 2, the method specifically includes:

[0049] S201. Store the sensor data in each sensor's receiving queue into each sensor's current data queue according to their respective receiving order.

[0050] S202. When all current data queues are not empty, determine the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue.

[0051] S203. Copy the earliest arriving sensor data from each current data queue to the candidate queue.

[0052] S204. Obtain the earliest arrival timestamp T3 and the latest arrival timestamp T4 of each sensor data in the candidate queue.

[0053] S205. Take the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and take the latest arrival timestamp of the sensor data in the current data queue of the hub as the hub time.

[0054] S206. Store the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor.

[0055] S207. Determine whether the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub. If the determination is no, execute S208. If the determination is yes, execute S209.

[0056] S208. Determine whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3. If the determination is yes, then execute S209.

[0057] S209. Publish the sensor data in the candidate queue as synchronized sensor data.

[0058] In this embodiment, the earliest arrival timestamp T1 and latest arrival timestamp T2 of all sensor data in each current data queue are further determined, and the earliest arrival timestamp T3 and latest arrival timestamp T4 of each sensor data in the candidate queue are obtained. Then, when performing time matching on the data in the candidate queue, it is first determined whether the earliest arriving sensor data in each current data queue corresponds to the hub. If yes, the sensor data in the candidate queue is directly published. If no, it is determined whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3. If yes, it indicates that sensor data arrived in each current data queue before and after the hub time; therefore, the sensor data in the candidate queue is optimal and can be published.

[0059] Therefore, this application embodiment determines whether the sensor data in the current candidate queue is time-matched by analyzing the relationship between the timestamps of the hub sensor and the hub time in all sensor data, thereby achieving the goal of efficiently and accurately performing time synchronization processing on multi-sensor data.

[0060] In one embodiment, the method further includes:

[0061] If the difference between T2 and T4 is less than the difference between the hub time and T3, then calculate the theoretical value of the earliest arrival timestamp T5 and the theoretical value of the latest arrival timestamp T6 in the current data queue of each sensor based on the data acquisition frequency and the frequency of receiving sensor data for each sensor.

[0062] Determine whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3;

[0063] If the determination is yes, then the sensor data in the candidate queue will be published as synchronized sensor data.

[0064] Specifically, if the difference between T2 and T4 is less than the difference between the hub time and T3, meaning that for the current candidate queue, the earliest arriving sensor data in each current data queue is not the sensor data corresponding to the hub, and it also does not meet the condition that sensor data arrives in each current data queue before and after the hub time. For example, under normal working conditions, each sensor trigger should only send and receive one sensor data. Therefore, for each sensor working normally, at the current moment, each sensor only has one data transmitted to the current data queue. The data in the candidate queue is the earliest (and unique) data in each sensor data queue, while the hub is the latest data in the candidate queue. Therefore, the difference between T2 and T4 is generally 0, while the difference between the hub time (i.e., T2) and T3 is generally not 0. In this case, the current judgment is negative. At this time, this embodiment will continue to calculate the theoretical value of the earliest arrival timestamp T5 and the theoretical value of the latest arrival timestamp T6 in the current data queue of each sensor based on the data acquisition frequency and the frequency of receiving sensor data for each sensor. For example, if the sensor collects data at 100ms intervals, and 50ms is set as the maximum difference in trigger times for each sensor at any given moment, then theoretically, the latest arrival time of each sensor's data = the latest arrival time of the currently collected data + 50ms limit difference. Then, based on this theoretical value, further judgment is made: that is, whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3. If the judgment is yes, it indicates that the data in the current candidate queue is time-matched, and the sensor data in the candidate queue can be published as synchronized sensor data. The advantage of this approach is that, at a certain moment, if the trigger time of one or several sensors is slightly delayed, but the overall time difference is within ± the limit difference at that moment, the messages in that candidate queue can still be considered to belong to the same frame.

[0065] Furthermore, if the difference between T6 and T4 is less than the difference between the hub time and T3, then it is further determined whether the difference between T6 and T4 is less than the difference between T5 and T3. If the determination is yes, it indicates that the time difference of the current data queue is less than the time difference of the candidate queue, therefore the optimal time matching of the data in the current candidate queue cannot be proven. Thus, CC is set to zero, the sensor data in each sensor's historical data queue is stored in the corresponding current data queue, and the current data synchronization process is exited. Here, CC represents the number of non-empty data in each sensor's current data queue. If the determination is no, the current data synchronization process is directly exited, and the determination continues in the next round. For example, if the current candidate queue no longer meets the premise of the maximum time difference limit, it means that the candidate messages no longer meet the condition of belonging to the same frame, that is, the data in the candidate queue does not belong to the same frame, so the process is directly exited.

[0066] Furthermore, in the above embodiments, after publishing the sensor data in the candidate queue as synchronized sensor data, the method further includes: clearing the sensor data in the candidate queue; setting CC to 0; storing the sensor data in each sensor's historical data queue into the corresponding current data queue; deleting the earliest arriving sensor data in each sensor's current data queue; and counting the number of non-empty data in each sensor's current data queue and assigning it to CC. In other words, after publication, the next round of time synchronization processing begins, so the value of CC needs to be reset. The historical data queue stores the sensor data previously copied to the candidate queue, so it can be returned to the current data queue and then deleted.

[0067] Based on the above embodiments, before copying the earliest arriving sensor data from each current data queue to the candidate queue, the method of this application also needs to determine whether there is already sensor data in the candidate queue. As mentioned above, sensor data is continuously collected and received; therefore, data is constantly being added to the current data queue. Simultaneously, the data synchronization method is executed iteratively according to a certain period and frequency. Furthermore, in different situations, data in the candidate queue may fail the time matching judgment and not be published to the sensing module. Therefore, when executing the time synchronization method, it is necessary to first determine whether there is already sensor data in the candidate queue.

[0068] Specifically, before copying the earliest arriving sensor data from each current data queue to the candidate queue, the method further includes: determining whether there is already sensor data in the candidate queue; if the determination is no, and the difference between T2 and T1 does not exceed a preset threshold, then the step of copying the earliest arriving sensor data from each current data queue to the candidate queue is executed. Wherein, if the difference between T2 and T1 does not exceed the preset threshold, it indicates that the arrival time of messages in the current data queue has not exceeded the limit, and therefore the next step can proceed. That is, when there is no sensor data in the current candidate queue, and the difference between T2 and T1 does not exceed the preset threshold, a candidate is directly generated.

[0069] If sensor data already exists in the candidate queue, it is necessary to determine whether there are more suitable candidates in the current data queue. In this embodiment, the determination is made based on the time difference between sensor data. That is, it is determined whether the difference between T2 and T4 is greater than or equal to the difference between T1 and T3. If the determination is yes, it indicates that the time difference of the current data queue exceeds the time difference of the candidate queue. Therefore, the candidate queue is better, and the step of storing the earliest arriving sensor data in each current data queue into the historical data queue of each sensor is performed without updating the candidates.

[0070] If there is already sensor data in the candidate queue, and the difference between T2 and T4 is less than the difference between T1 and T3, it indicates that the current data queue is better. Therefore, the sensor data in the candidate queue is deleted, and the earliest arriving sensor data in each current data queue is copied to the candidate queue, i.e., a new candidate is generated. Then, T3 and T4 are updated according to the new candidate queue, and the step of storing the earliest arriving sensor data in each current data queue into the historical data queue of each sensor is executed.

[0071] Based on the above embodiments, before determining the earliest arrival timestamp T1 and latest arrival timestamp T2 of all sensor data in each current data queue, the method further includes: determining whether CC is equal to the number of sensors; if so, then performing the step of determining the earliest arrival timestamp T1 and latest arrival timestamp T2 of all sensor data in each current data queue. Therefore, the value of CC can be verified again before time synchronization processing, ensuring the effective execution of the process.

[0072] Meanwhile, in the above embodiments, when there is no sensor data in the candidate queue, it is necessary to further determine whether the difference between T2 and T1 exceeds a preset threshold. In this embodiment, if the difference between T2 and T1 exceeds the preset threshold, it indicates that the arrival time of the data in the current data queue has exceeded the limit. Therefore, the earliest arriving sensor data among all sensor data is deleted, and the process returns to the step of determining whether CC is equal to the number of sensors.

[0073] In one implementation, before entering the time synchronization process, it is necessary to handle some possible abnormal situations to ensure the accuracy of the time synchronization process.

[0074] On the one hand, before determining whether CC equals the number of sensors, the method further includes: if more than one sensor data from the same sensor is stored in the corresponding current data queue at the same time, then the time order between the more than one sensor data is determined, and an alarm is issued. Afterwards, the process can return to reacquire sensor data. This avoids situations where two or more sensor data are acquired at the same time due to sensor malfunctions, thereby preventing errors in subsequent time synchronization processing.

[0075] On the other hand, before determining whether CC equals the number of sensors, the method further includes: if any sensor has a sum of sensor data in its current data queue and historical data queue that exceeds a preset capacity threshold, it indicates that the sensor's data has not been entered into time synchronization processing, resulting in data backlog. In this case, the sensor data in its historical data queue needs to be stored in its current data queue, the CC value updated, and a discard flag added to the current sensor data in its current data queue before entering the time synchronization processing flow. Conversely, if the sum of sensor data in the current data queue and historical data queue for any sensor does not exceed the capacity threshold, the method returns to the step of storing the sensor data in each sensor's respective receiving queue into each sensor's respective current data queue according to their respective receiving order. That is, as long as the sum does not exceed the capacity threshold, sensor data acquisition continues, and the time synchronization processing flow is temporarily suspended.

[0076] Accordingly, before determining whether there is already sensor data in the candidate queue, the method further includes: traversing all sensor data in each current data queue; if there is sensor data with a discard mark located at the tail of the current data queue, then the discard mark of the sensor data is maintained; if the sensor data with a discard mark is not the last frame of data in all sensor data, then the discard mark is cleared.

[0077] Accordingly, before copying the earliest arriving sensor data in each current data queue to the candidate queue, the method further includes: determining whether a discard flag exists; if the determination is no, then performing the step of copying the earliest arriving sensor data in each current data queue to the candidate queue; if the determination is yes, then deleting the earliest arriving sensor data among all sensor data, and returning to perform the step of determining whether CC is equal to the number of sensors.

[0078] Specifically, as mentioned above, a discarded data queue indicates a data backlog. In this embodiment, before determining whether there is already sensor data in the candidate queue, the sensor data in each current data queue is traversed. If the sensor data with a discarded mark is not the last frame of all data, it indicates that new data has entered the current data queue. Since this sensor data did not match in the previous matching, it can be matched with the latest arriving data, and therefore the discarded mark can be cleared. If the sensor data with a discarded mark is located at the end of the current data queue, it means that no new data has been added to the current data queue, that is, the current data queue has not been updated, so the discarded mark of this sensor data is maintained. Then, before copying the earliest arriving sensor data in each current data queue to the candidate queue, it is also necessary to determine whether there is a discarded mark. If the determination is negative, the step of copying the earliest arriving sensor data in each current data queue to the candidate queue is executed; if the determination is positive, it indicates that there is a data backlog. Therefore, the earliest arriving sensor data in all sensor data is deleted because the data has become invalid. Then, the step of determining whether CC is equal to the number of sensors is executed, which means re-entering the time synchronization processing flow.

[0079] Figure 5 is a schematic diagram of the overall logic of the multi-sensor data synchronization method provided in this application embodiment. As shown in Figure 5, the vehicle-side sensors may include cameras, LiDAR, 4D millimeter-wave radar, or ultrasonic sensors, etc. Each sensor has its own thread for reading sensor data and stores the data in its own receiving queue. When the receiving queue is not empty, the thread for reading sensor data will notify the time synchronization thread to store the sensor data in the current data queue of each sensor. Whenever a current data queue is not empty, CC is incremented by 1. When CC equals the number of sensors, the time synchronization process can be entered. If sensor data with a matching time is found, the data is published to the vehicle-side perception system for fusion perception.

[0080] Figure 6 is a schematic diagram of the main flow of the time synchronization thread of the multi-sensor data synchronization method provided in this application embodiment. As shown in Figure 6, after the process starts, it is determined whether the sensor receiving queue is empty. If it is empty, the thread waits; if it is not empty, the sensor data in the receiving queue is copied. Then, it is determined whether the copied sensor data is valid. Specifically, it is first determined whether the corresponding current data queue is not empty. If it is not empty, the copied sensor data is read, and it is determined whether there is data of the same type as the sensor data in each current data queue. If it exists, it means that the copied sensor data is valid and can be stored in the current data queue of the corresponding sensor; otherwise, it means that it is invalid, that is, unknown sensor data has been received.

[0081] Furthermore, on one hand, it determines whether the same sensor receives data from another sensor at the same time. If not, it checks the time sequence of data from multiple sensors and issues an alarm. Otherwise, it increments CC by 1 and checks if CC equals the number of sensors. If it does, it proceeds to the time synchronization process. On the other hand, if time synchronization is not initiated, it checks if the sum of sensor data in the current and historical data queues of the sensor exceeds a preset threshold. If not, it returns; otherwise, it sets CC to zero, stores the sensor data from the historical data queue into the current data queue, updates the value of CC, adds a discard flag to the current data queue, and then proceeds to the time synchronization process.

[0082] Figure 7 is a schematic diagram of the time synchronization processing flow of the multi-sensor data synchronization method provided in this application embodiment. As shown in Figure 7, after the process starts, it is first determined whether CC is equal to the number of sensors. If it is equal, the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue are determined. All sensor data in each current data queue are traversed. If there is sensor data at the tail of the current data queue with a discard mark, which is not the last frame of data in all sensor data, the discard mark is cleared.

[0083] Next, it checks if there is already sensor data in the candidate queue. If there is no sensor data in the candidate queue, it first checks if the difference between T2 and T1 exceeds a preset threshold. If it does not exceed the threshold, it further checks if there is a discard flag. If there is no discard flag, a candidate is generated. If the threshold is exceeded, or a discard flag exists, the earliest arriving data in all sensor data is deleted, and the process returns to the step of checking if CC equals the number of sensors. After generating candidates, the step of storing the earliest arriving sensor data in each current data queue into the historical data queue of each sensor is executed. If there is sensor data in the candidate queue, it means that no candidate was published last time. Therefore, it checks if the difference between T2 and T4 is greater than or equal to the difference between T1 and T3. If the difference is yes, it means that the candidate is better, and the step of storing the earliest arriving sensor data in each current data queue into the historical data queue of each sensor is executed. If the difference is no, it means that the data in the current data queue is better, so a new candidate is generated, and T3 and T4 are updated.

[0084] After storing the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor, it is determined whether the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub. If yes, a candidate is directly released. Otherwise, it is determined whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3. If yes, a candidate is directly released. Otherwise, based on the data acquisition frequency and the frequency of receiving sensor data for each sensor, the theoretical value of the earliest arrival timestamp T5 and the theoretical value of the latest arrival timestamp T6 in the current data queue of each sensor are calculated. Then, it is determined whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3. If yes, a candidate is directly released. Otherwise, it is further determined whether the difference between T6 and T4 is less than the difference between T5 and T3. If less, CC is set to zero, and the sensor data in the respective historical data queue of each sensor is stored into the corresponding current data queue, and the current data synchronization process is exited. Otherwise, the current data synchronization process is exited directly.

[0085] In summary, to more accurately and precisely meet the perception requirements of intelligent driving, the technical solution of this application performs unified time synchronization processing after acquiring data from various types of sensors on the vehicle side. This eliminates the impact of sensor transmission delays and data loss due to sensor downtime, ensuring the synchronization of various types of sensor data entering the perception system. This allows for the acquisition of effective information from various sensors at the same time, enabling the autonomous driving system to correctly perceive external environmental information and ensuring the fusion, planning, and decision-making of multi-sensor data for autonomous driving. Specifically, the embodiments of this application have the following technical effects:

[0086] 1. The program's concurrency requirements are guaranteed by employing N+1 threads. N threads continuously acquire data from N sensors on the vehicle, with one thread assigned to each sensor. An additional thread receives and stores the sensor data, performs time synchronization processing on data that meets certain conditions, and then sends the synchronized data to the vehicle's perception system. This ensures that the multiple sensors do not interfere with each other and that the communication channels remain open.

[0087] 2. It can achieve time synchronization of data from different sensors, which can compensate for the delay caused by data transmission between different sensors, and solve the adverse consequences caused by the failure of some sensors, thus ensuring the robustness of the sensing system, on the basis of ensuring the hardware or software synchronization of sensors.

[0088] 3. It can efficiently ensure the efficiency and accuracy of data time matching when the vehicle system acquires data from multiple sensors. For the same type of sensor, the main sensor data only needs to arrive at the time synchronizer in sequence, without having to guarantee that a certain sensor must arrive first. This allows for compatibility with the situation where some sensors are slightly delayed at certain times, ensuring the robustness of the time synchronizer and guaranteeing the efficient operation of the perception system.

[0089] Figure 8 is a schematic diagram of the structure of the multi-sensor data synchronization device provided in an embodiment of this application. As shown in Figure 8, the multi-sensor data synchronization device 80 includes:

[0090] The data acquisition module 810 is used to store the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to their respective receiving order.

[0091] The candidate acquisition module 820 is used to copy the earliest arriving sensor data in each current data queue to the candidate queue when each current data queue is not empty.

[0092] The hub determination module 830 is used to take the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and take the latest arrival timestamp of the sensor data in the current data queue of the hub as the hub time.

[0093] The historical data storage module 840 is used to store the earliest arriving sensor data in each current data queue into the historical data queue of each sensor.

[0094] The matching processing module 850 is used to publish the sensor data in the candidate queue as synchronized sensor data if the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time.

[0095] In some embodiments, the apparatus further includes:

[0096] The first timestamp determination module is used to determine the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue before the candidate acquisition module 820 copies the earliest arriving sensor data in each current data queue to the candidate queue.

[0097] Accordingly, the device also includes:

[0098] The second timestamp determination module is used to obtain the earliest arrival timestamp T3 and the latest arrival timestamp T4 of each sensor data in the candidate queue before the hub determination module 830 uses the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub.

[0099] Accordingly, the matching processing module 850 is also used for:

[0100] If the earliest arriving sensor data in each of the current data queues is not the sensor data corresponding to the hub, then it is determined whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3. If the determination is yes, it is determined that sensor data has arrived in each of the current data queues before and after the hub time.

[0101] In some embodiments, the apparatus further includes:

[0102] The third timestamp determination module is used to calculate the theoretical value of the earliest arrival timestamp T5 and the theoretical value of the latest arrival timestamp T6 in the current data queue of each sensor, based on the data acquisition frequency and the frequency of receiving sensor data of each sensor, if the difference between T2 and T4 is less than the difference between the hub time and T3.

[0103] The first judgment processing module is used to determine whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3; if the determination is yes, the sensor data in the candidate queue is published as synchronized sensor data.

[0104] In some embodiments, the apparatus further includes a second determination processing module, specifically used for:

[0105] If the difference between T6 and T4 is less than the difference between the hub time and T3, then determine whether the difference between T6 and T4 is less than the difference between T5 and T3.

[0106] If the determination is yes, then CC is set to zero, the sensor data in the historical data queue of each sensor is stored into the corresponding current data queue, and the current data synchronization process is exited. Here, CC represents the number of non-empty data in the current data queue of each sensor.

[0107] If the result is negative, the current data synchronization process will be terminated.

[0108] In some embodiments, the apparatus further includes a publishing processing module, specifically used for:

[0109] Clear the sensor data in the candidate queue;

[0110] Set CC to 0;

[0111] Store the sensor data from the historical data queue of each sensor into the corresponding current data queue;

[0112] Delete the earliest arriving sensor data from the current data queue of each sensor;

[0113] The number of non-empty data queues in the current data queue of each sensor is counted and assigned to CC.

[0114] In some embodiments, the apparatus further includes a third determination and processing module, specifically used for:

[0115] Before the candidate acquisition module 820 copies the earliest arriving sensor data from each current data queue to the candidate queue, it determines whether there is already sensor data in the candidate queue.

[0116] If the determination is negative, and the difference between T2 and T1 does not exceed the preset threshold, then the candidate acquisition module 820 copies the earliest arriving sensor data from each current data queue to the candidate queue.

[0117] In some embodiments, the apparatus further includes a fourth determination and processing module, specifically used for:

[0118] If sensor data is already available in the candidate queue, determine whether the difference between T2 and T4 is greater than or equal to the difference between T1 and T3.

[0119] If the determination is yes, then the historical data storage module 840 stores the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor.

[0120] In some embodiments, the apparatus further includes a fifth determination and processing module, specifically used for:

[0121] If the difference between T2 and T4 is less than the difference between T1 and T3, then the sensor data in the candidate queue is deleted, and the earliest arriving sensor data in each current data queue is copied to the candidate queue.

[0122] Update T3 and T4, and the historical data storage module 840 stores the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor.

[0123] In some embodiments, the apparatus further includes a sixth determination processing module, specifically used for:

[0124] Before the first timestamp determination module determines the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue, it is determined whether CC is equal to the number of sensors.

[0125] If the determination is yes, then the first timestamp determination module determines the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue.

[0126] In some embodiments, the apparatus further includes a seventh determination and processing module, specifically used for:

[0127] If the difference between T2 and T1 exceeds a preset threshold, the earliest arriving sensor data among all sensor data is deleted, and the process is returned to the sixth judgment and processing module to determine whether CC is equal to the number of sensors.

[0128] In some embodiments, the apparatus further includes:

[0129] The alarm module is used to determine the time order of the data from the same sensor if more than one sensor data is stored in the corresponding current data queue at the same time, and to issue an alarm before the sixth judgment and processing module determines whether CC is equal to the number of sensors.

[0130] In some embodiments, the apparatus further includes:

[0131] The discard marker module is used to, before the sixth judgment and processing module determines whether CC is equal to the number of sensors, if there is any sensor whose sensor data in the current data queue and the historical data queue is greater than the preset capacity threshold, store the sensor data in its historical data queue into its current data queue, update the value of CC, and add a discard marker to its current data queue.

[0132] Accordingly, the device also includes:

[0133] The traversal module is used to traverse all sensor data in each current data queue before the third judgment processing module determines whether there is already sensor data in the candidate queue. If there is sensor data with a discard mark located at the end of the current data queue, the discard mark of the sensor data is kept; if the sensor data with a discard mark is not the last frame of data among all the sensor data, the discard mark is cleared.

[0134] Accordingly, the device further includes an eighth judgment and processing module, specifically used for:

[0135] Before the candidate acquisition module 820 copies the earliest arriving sensor data from each current data queue to the candidate queue, it determines whether the discard mark exists.

[0136] If the determination is negative, the candidate acquisition module 820 copies the earliest arriving sensor data from each current data queue to the candidate queue.

[0137] If the determination is yes, then delete the earliest arriving sensor data among all the sensor data, and return to the sixth determination processing module to determine whether CC is equal to the number of sensors.

[0138] In some embodiments, the apparatus further includes:

[0139] The threshold exceeding processing module is used to return the data acquisition module 810 to store the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to their respective receiving order if the sum of the sensor data in the current data queue and the historical data queue corresponding to any sensor is not greater than the threshold.

[0140] In some embodiments, the data acquisition module 810 includes:

[0141] The storage unit is used to obtain sensor data from the respective thread of each sensor and store it into the respective receiving queue of each sensor.

[0142] The copying unit is used to copy the sensor data in the receiving queue to the current data queue of each sensor in their respective receiving order when the receiving queue is not empty.

[0143] The multi-sensor data synchronization device provided in this application embodiment can be used to execute the technical solution of the multi-sensor data synchronization method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0144] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the matching processing module 850 can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its functions can be called and executed by a processing element of the above device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0145] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0146] Figure 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 9, the electronic device may include: a transceiver 121, a processor 122, and a memory 123.

[0147] Processor 122 executes computer execution instructions stored in memory, causing processor 122 to perform the scheme in the above embodiments. Processor 122 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0148] The memory 123 is connected to the processor 122 via the system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0149] Transceiver 121 can be used to obtain the task to be run and its configuration information.

[0150] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0151] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0152] This application also provides a computer-readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the technical solution of the multi-sensor data synchronization method described above.

[0153] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the multi-sensor data synchronization method in the above embodiments.

[0154] In the implementation of the computer program product, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] Note that the above are merely preferred embodiments and technical principles of this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for synchronizing data of multiple sensors, the method comprising: include: The sensor data in the receiving queue of each sensor is stored into the current data queue of each sensor in the order of receiving. If none of the current data queues are empty, then copy the earliest arriving sensor data from each current data queue to the candidate queue. The sensor name corresponding to the latest arriving sensor data in each of the current data queues is taken as the hub, and the latest arrival timestamp of the sensor data in the current data queue of the hub is taken as the hub time. The earliest arriving sensor data in each current data queue is stored in the respective historical data queue of each sensor; If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time, then the sensor data in the candidate queue will be published as the synchronized sensor data.

2. The method of claim 1, wherein, Before copying the earliest arriving sensor data from each current data queue to the candidate queue, the method further includes: Determine the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue; Accordingly, before using the sensor name corresponding to the latest arriving sensor data in each of the current data queues as the hub, the method further includes: Obtain the earliest arrival timestamp T3 and the latest arrival timestamp T4 of each sensor data in the candidate queue; Accordingly, the method further includes: If the earliest arriving sensor data in each of the current data queues is not the sensor data corresponding to the hub, then it is determined whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3. If the determination is yes, it is determined that sensor data has arrived in each of the current data queues before and after the hub time.

3. The method of claim 2, wherein, The method further includes: If the difference between T2 and T4 is less than the difference between the hub time and T3, then the theoretical values ​​of the earliest arrival timestamp T5 and the latest arrival timestamp T6 in the current data queue of each sensor are calculated based on the data acquisition frequency and the frequency of receiving sensor data for each sensor. Determine whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3; If the determination is yes, then the sensor data in the candidate queue will be published as synchronized sensor data.

4. The method of claim 3, wherein, The method further includes: If the difference between T6 and T4 is less than the difference between the hub time and T3, then determine whether the difference between T6 and T4 is less than the difference between T5 and T3. If the determination is yes, then CC is set to zero, the sensor data in the historical data queue of each sensor is stored into the corresponding current data queue, and the current data synchronization process is exited. Here, CC represents the number of non-empty data in the current data queue of each sensor. If the result is negative, the current data synchronization process will be terminated.

5. The method of claim 4, wherein, After publishing the sensor data from the candidate queue as synchronized sensor data, the method further includes: Clear the sensor data in the candidate queue; Set CC to 0; Store the sensor data from the historical data queue of each sensor into the corresponding current data queue; Delete the earliest arriving sensor data from the current data queue of each sensor; The number of non-empty data queues in the current data queue of each sensor is counted and assigned to CC.

6. The method of claim 5, wherein, Before copying the earliest arriving sensor data from each current data queue to the candidate queue, the method further includes: Determine whether sensor data is already in the candidate queue; If the determination is negative, and the difference between T2 and T1 does not exceed the preset threshold, then the step of copying the earliest arriving sensor data in each current data queue to the candidate queue is executed.

7. The method of claim 6, wherein, The method further includes: If sensor data is already available in the candidate queue, determine whether the difference between T2 and T4 is greater than or equal to the difference between T1 and T3. If the determination is yes, then the step of storing the earliest arriving sensor data in each of the current data queues into the respective historical data queue of each sensor is executed.

8. The method of claim 7, wherein, The method further includes: If the difference between T2 and T4 is less than the difference between T1 and T3, then the sensor data in the candidate queue is deleted, and the earliest arriving sensor data in each current data queue is copied to the candidate queue. Update T3 and T4, and perform the step of storing the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor.

9. The method of claim 8, wherein, Before determining the earliest arrival timestamp T1 and latest arrival timestamp T2 of all sensor data in each of the current data queues, the method further includes: Determine if CC equals the number of sensors; If the determination is yes, then the step of determining the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue is executed.

10. The method of claim 9, wherein, The method further includes: If the difference between T2 and T1 exceeds a preset threshold, delete the earliest arriving sensor data from all sensor data and return to the step of determining whether CC is equal to the number of sensors.

11. The method of claim 10, wherein, Before determining whether CC equals the number of sensors, the method further includes: If more than one sensor data is stored in the corresponding current data queue at the same time, the time order between the more than one sensor data is determined and an alarm is issued.

12. The method of claim 11, wherein, Before determining whether CC equals the number of sensors, the method further includes: If any sensor has a total sensor data in its current data queue and historical data queue that is greater than a preset threshold, then the sensor data in its historical data queue is stored in its current data queue, the CC value is updated, and the current sensor data in its current data queue is marked as discarded. Accordingly, before determining whether sensor data already exists in the candidate queue, the method further includes: Iterate through all sensor data in each current data queue. If a sensor data with a discard mark is located at the end of the current data queue, keep the discard mark of that sensor data. If the sensor data with a discard mark is not the last frame of data among all the sensor data, clear the discard mark. Accordingly, before copying the earliest arriving sensor data from each current data queue to the candidate queue, the method further includes: Determine whether the discard marker exists; If the determination is negative, then the step of copying the earliest arriving sensor data from each current data queue to the candidate queue is executed. If the determination is yes, then delete the earliest arriving sensor data among all the sensor data, and return to the step of determining whether CC is equal to the number of sensors.

13. The method according to claim 12, characterized in that, If the sum of sensor data in the current data queue and historical data queue corresponding to any sensor is not greater than the capacity threshold, then return to the step of storing the sensor data in the respective receiving queue of each sensor into the respective current data queue of each sensor according to their respective receiving order.

14. The method of claim 1, wherein, The step of storing the sensor data in each sensor's respective receiving queue into each sensor's respective current data queue according to their respective receiving order includes: Sensor data is acquired from the respective thread of each sensor and stored in the respective receive queue of each sensor; If the receiving queue is not empty, the sensor data in the receiving queue is copied to the current data queue of each sensor according to their respective receiving order.

15. A multi-sensor data synchronization apparatus, comprising: include: The data acquisition module is used to store the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to their respective receiving order; The candidate acquisition module is used to copy the earliest arriving sensor data from each current data queue to the candidate queue when each current data queue is not empty. The hub determination module is used to take the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and take the latest arrival timestamp of the sensor data in the current data queue of the hub as the hub time. The historical data storage module is used to store the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor. The matching processing module is used to publish the sensor data in the candidate queue as synchronized sensor data if the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time.

16. An electronic device, comprising: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the multi-sensor data synchronization method as described in any one of claims 1-14.

17. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the multi-sensor data synchronization method of any one of claims 1-14.

18. A computer program product, characterised in that, The computer program, when executed by a processor, implements the multi-sensor data synchronization method of any one of claims 1-14.

Images