Multi-sensor data synchronization method, device, equipment and medium

Abstract

The present application provides a multi-sensor data synchronization method, device, equipment and medium, which relates to the field of autonomous driving technology. The method includes: when the current data queue of each sensor is not empty, copying the earliest arriving sensor data in each current data queue to the candidate queue; using the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and using the latest arrival timestamp of the sensor data in the current data queue of the hub as the hub time; storing the earliest arriving sensor data in each current data queue into each sensor's respective historical data queue; if the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or before and after the hub time, sensor data arrives in each current data queue, and publishing the sensor data in the candidate queue as synchronized sensor data. The present application ensures the efficiency and accuracy of multi-sensor data time synchronization.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a multi-sensor data synchronization method, device, equipment and medium. Background Art

[0002] Autonomous vehicles generally need to be equipped with multiple sensors, such as cameras, lidar or 4D millimeter radar waves, to collect data about the surrounding environment. Autonomous vehicles perform data fusion perception based on the perception data collected by multiple sensors to obtain perception information about the surrounding environment. In this process, since multi-sensor data needs to be fused, the accuracy of multi-sensor time synchronization becomes a key factor to ensure the accuracy of the perception results.

[0003] However, there will be a link delay when each sensor on the vehicle triggers data collection, and there will also be a time delay in the process of each sensor transmitting data to the autonomous driving system. This makes the autonomous driving system receive data sent by different sensors at inconsistent times, or a temporary failure of a sensor causes time misalignment, which may cause the autonomous driving system to fail to work properly. Summary of the invention

[0004] The present application provides a multi-sensor data synchronization method, device, equipment and medium, which are used to synchronize the time of multi-sensor data without interfering with the multi-channel sensors, and ensure the efficiency and accuracy of the multi-sensor data time synchronization.

[0005] In a first aspect, the present application provides a multi-sensor data synchronization method, comprising:

[0006] The sensor data in the receiving queue of each sensor are stored in the current data queue of each sensor according to the receiving order;

[0007] When all current data queues are not empty, the earliest arriving sensor data in each current data queue is copied to the candidate queue;

[0008] The sensor name corresponding to the latest sensor data in each current data queue is used as the hub, and the latest arrival timestamp of the sensor data in the current data queue of the hub is used as the hub time;

[0009] Storing the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor;

[0010] If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or sensor data arrives in each current data queue before and after the hub time, the sensor data in the candidate queue is published as synchronized sensor data.

[0011] In a second aspect, the present application provides a multi-sensor data synchronization device, comprising:

[0012] A data acquisition module, used for storing the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to the receiving order;

[0013] A candidate acquisition module is used to copy the earliest arriving sensor data in each current data queue to the candidate queue when each current data queue is not empty;

[0014] A hub determination module, configured to use the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and use the latest arriving timestamp of the sensor data in the current data queue of the hub as the hub time;

[0015] A historical data storage module, used for storing the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor;

[0016] A matching processing module is used to publish the sensor data in the candidate queue as synchronized sensor data if the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time.

[0017] In a third aspect, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multi-sensor data synchronization method as described in any one of the embodiments of the present application is implemented.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a multi-sensor data synchronization method as described in any one of the embodiments of the present application.

[0019] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the multi-sensor data synchronization method as described in any one of the embodiments of the present application.

[0020] The multi-sensor data synchronization method, device, equipment and medium provided by the present application, each sensor receives sensor data through its own thread, and in addition, a separate thread is opened to perform data synchronization. Specifically, the sensor data is first obtained from the receiving queue of each sensor and stored in the current data queue of each sensor, and then the earliest arriving sensor data in each current data queue is copied to the pre-established candidate queue, and the sensor name corresponding to the latest arriving sensor data in each current data queue is used as the hub, and the latest arrival timestamp of the sensor data in the current data queue of the hub is used as the hub time, and then the earliest arriving sensor data in each current data queue is stored in the historical data queue of each sensor. Afterwards, it is judged whether the sensor data in the candidate queue matches in time. If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or before and after the hub time, sensor data arrives in each current data queue, then the match is successful, and the sensor data in the candidate queue is published as the synchronized sensor data. Therefore, the technical solution of the present application can time synchronize multi-sensor data without interfering with the multi-channel sensors, compensate for the delay caused by data transmission of each sensor, and ensure the efficiency and accuracy of multi-sensor data time synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] Figure 1 A flowchart of a multi-sensor data synchronization method provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a multi-sensor data synchronization method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a current data queue in a multi-sensor data synchronization method provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of a candidate queue in a multi-sensor data synchronization method provided in an embodiment of the present application;

[0026] Figure 5 An overall logic diagram of a multi-sensor data synchronization method provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the main process of the time synchronization thread of the multi-sensor data synchronization method provided in an embodiment of the present application;

[0028] Figure 7 A schematic diagram of a time synchronization process of a multi-sensor data synchronization method provided in an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the structure of a multi-sensor data synchronization device provided in an embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0031] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0032] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings.

[0033] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0034] Figure 1 Flow chart of the multi-sensor data synchronization method provided in the embodiment of the present application. This embodiment can be applied to the situation where the time synchronization of the data of multiple sensors in an autonomous driving vehicle is performed to ensure the efficient operation of the vehicle perception system. The method can be executed by a multi-sensor data synchronization device, which can be implemented in software and / or hardware, and is preferably configured in an electronic device, such as an autonomous driving vehicle or a server. Figure 1 As shown, the method specifically includes:

[0035] S101 , storing the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to the receiving order.

[0036] S102 : When all current data queues are not empty, the earliest arriving sensor data in each current data queue is copied to the candidate queue.

[0037] S103 , taking the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and taking the latest arriving timestamp of the sensor data in the current data queue of the hub as the hub time.

[0038] S104: Store the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor.

[0039] S105. If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or sensor data arrives in each current data queue before and after the hub time, the sensor data in the candidate queue is published as synchronized sensor data.

[0040] Autonomous vehicles usually include multiple sensors, such as cameras, lidar or 4D millimeter radar waves, etc. These sensors can be used to collect data about the surrounding environment, and then perform data fusion perception based on the data collected by multiple sensors, and then guide vehicle operation based on the perception results. Therefore, in order to ensure the accuracy of the perception results, the accuracy of multi-sensor time synchronization becomes a key factor.

[0041] In the embodiment of the present application, each sensor corresponds to a thread for collecting and receiving sensor data, and the collected sensor data will be placed in their respective receiving queues. At the same time, a special thread is started in the embodiment of the present application to perform data synchronization processing. When the receiving queue is not empty, the sensor thread will notify the data synchronization processing thread, and the data synchronization processing thread will store the sensor data in the receiving queue into the current data queue corresponding to each sensor according to their respective receiving order. Figure 3 A schematic diagram of the current data queue in the multi-sensor data synchronization method provided in an embodiment of the present application. The figure schematically shows the current data queues corresponding to the three sensors 1, 2, and 3, and each current data queue stores sensor data in the order in which it is received.

[0042] When all current data queues are not empty, the earliest arriving sensor data in each current data queue is copied to the candidate queue. If the sensor data in the candidate queue is time-synchronized data, the data in the candidate queue can be published to the perception system for fusion perception. If the sensor data in the candidate queue is not time-synchronized data, the candidate needs to be re-acquired. Figure 4A schematic diagram of a candidate queue in a multi-sensor data synchronization method provided in an embodiment of the present application. Still taking the above three sensors as an example, data 1-1, 2-1, and 3-1 at the head of the queue are obtained from the current data queues of sensors 1, 2, and 3, respectively, as the earliest arriving sensor data, and are put into the candidate queue.

[0043] In the embodiment of the present application, when judging whether the data in the candidate queue is time-matched synchronization data, the specific method used is as follows: the sensor name corresponding to the latest arriving sensor data in each current data queue is used as the hub, and the latest arrival timestamp of the sensor data in the current data queue of the hub is used as the hub time. The earliest arriving sensor data in each current data queue is stored in the respective historical data queue of each sensor. If the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or sensor data arrives in each current data queue before and after the hub time, the sensor data in the candidate queue is published as synchronized sensor data. If it can be published successfully, the data in the historical data queue can be deleted. If it is not published successfully, the data in the historical data queue needs to be put back into the current data queue to re-acquire the candidate.

[0044] The data synchronization method of the embodiment of the present application can be executed cyclically at a certain frequency. If no sensor has a delay in data collection or data transmission, then under normal circumstances, each time the data synchronization method is executed, there will be a sensor data in the current data queue of each sensor, and these sensor data will also be copied to the candidate queue as the earliest sensor data arriving in each current data queue. At the same time, these candidates are also time-matched synchronization data and will be published smoothly. However, when any sensor is delayed, the number of data in the current data queue of each sensor will be different. At the same time, in the process of cyclically executing the data synchronization method, the sensor data is continuously collected and received according to their respective collection frequencies, and the data in the current data queue will gradually increase. Therefore, it is necessary to judge and process whether the candidate is time-matched data for abnormal situations.

[0045] Specifically, on the one hand, if the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, that is, the message of the hub has become the earliest arriving data, this indicates that the sensor data corresponding to the current hub does not match the sensor data of the previous frame, but can match the subsequent sensor data, and there is no data in the current data queue that can match the data in the candidate queue in time. The sensor data in the current candidate queue is the optimal time synchronization data, and therefore, it can be published as synchronized sensor data. On the other hand, if the message of the hub does not become the earliest arriving data, but other sensor data arrives in each current data queue before and after the hub time, this means that the sensor data corresponding to the current hub has been proven to be optimal in terms of time matching, that is, the data in the current candidate queue is also the optimal time synchronization data and can be published.

[0046] In summary, in the technical solution of the embodiment of the present application, each sensor receives sensor data through its own thread. In addition, a separate thread is opened for data synchronization. Therefore, the present application will not interfere with the data collection and reception of each sensor. After obtaining the candidate, the present application judges whether the sensor data in the candidate queue matches in time. If the earliest sensor data arriving in each current data queue is the sensor data corresponding to the hub, or before and after the hub time, sensor data arrives in each current data queue, then the match is successful, and the sensor data in the candidate queue is published as synchronized sensor data. Therefore, the technical solution of the present application can synchronize the time of multi-sensor data without interfering with multi-channel sensors, make up for the delay caused by data transmission of each sensor, and at the same time, there is no need to impose mandatory requirements on the order of arrival of each sensor, so as to be compatible with the situation where some sensors are slightly delayed at some moments, ensure the efficiency and accuracy of multi-sensor data time synchronization, ensure the efficient operation of the perception system, and improve the driving safety of autonomous driving vehicles.

[0047] Figure 2 Flow chart of the multi-sensor data synchronization method provided in the embodiment of the present application. This embodiment is further optimized on the basis of the above embodiment. Figure 2 As shown, the method specifically includes:

[0048] S201 , storing the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to the receiving order.

[0049] S202 : When all current data queues are not empty, determine the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue.

[0050] S203 : Copy the earliest arriving sensor data in each current data queue to the candidate queue.

[0051] S204 , obtaining the earliest arrival timestamp T3 and the latest arrival timestamp T4 of each sensor data in the candidate queue.

[0052] S205 , taking the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and taking the latest arriving timestamp of the sensor data in the current data queue of the hub as the hub time.

[0053] S206: Store the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor.

[0054] S207, judging whether the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, if it is judged as no, executing S208, if it is judged as yes, executing S209.

[0055] S208, determine whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3, if so, execute S209.

[0056] S209: Publish the sensor data in the candidate queue as synchronized sensor data.

[0057] In this embodiment, the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue are further determined, and the earliest arrival timestamp T3 and the latest arrival timestamp T4 of each sensor data in the candidate queue are obtained. Afterwards, when the data in the candidate queue is time-matched, it is first determined whether the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub. If it is determined to be yes, the sensor data in the candidate queue is directly released. If it is determined to be no, it is determined whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3. If it is determined to be yes, it indicates that sensor data has arrived in each current data queue before and after the hub time. Therefore, the sensor data in the candidate queue is optimal and can be released.

[0058] Therefore, the embodiment of the present application determines whether the sensor data in the current candidate queue is time-matched data by comparing the relationship between the hub sensor and the hub time in all sensor data timestamps, thereby achieving the purpose of efficient and accurate time synchronization processing of multi-sensor data.

[0059] In one embodiment, the method further comprises:

[0060] If the difference between T2 and T4 is less than the difference between the hub time and T3, then the earliest arrival timestamp theoretical value T5 and the latest arrival timestamp theoretical value T6 in the current data queue of each sensor are calculated according to the data collection frequency of each sensor and the frequency of receiving sensor data;

[0061] Determine whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3;

[0062] If the judgment is yes, the sensor data in the candidate queue is published as synchronized sensor data.

[0063] Specifically, if the difference between T2 and T4 is less than the difference between the hub time and T3, that is, for the current candidate queue, the earliest sensor data in each current data queue is not the sensor data corresponding to the hub, and at the same time, it does not conform to the situation where sensor data arrives in each current data queue before and after the hub time. For example, under normal working conditions, each time a sensor is triggered, only one sensor data should be sent and received. Then, for the normal working of each sensor, at the current moment, each sensor has only one data transmitted to the current data queue. The data in the candidate queue is the earliest data (and only) in each sensor data queue, and the hub is the latest data in the candidate queue. Therefore, the difference between T2 and T4 is generally 0, and the difference between the hub time (i.e., T2) and T3 is generally not 0. At this time, the current judgment is no. At this time, the embodiment of the present application will continue to calculate the earliest arrival timestamp theoretical value T5 and the latest arrival timestamp theoretical value T6 in the current data queue of each sensor according to the data acquisition frequency of each sensor and the frequency of receiving sensor data. For example, the interval between each sensor's data collection is 100ms, and 50ms is set as the maximum difference in the triggering collection time of each sensor at each moment, that is, in theory, the latest arrival time of each sensor data is equal to the latest time of the currently collected data plus the 50ms difference. Then continue to judge in combination with the theoretical value, that is, continue to judge whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3. If the judgment is yes, it indicates that the data in the current candidate queue is time-matched data, and the sensor data in the candidate queue can be published as synchronized sensor data. The advantage of this is that for a certain moment, the trigger time of a certain sensor or several sensors is slightly delayed, but the overall time difference is within the range of plus or minus the difference at that moment, and the messages in the candidate queue can also be considered to belong to the same frame.

[0064] Furthermore, if the difference between T6 and T4 is less than the difference between the hub time and T3, then determine whether the difference between T6 and T4 is less than the difference between T5 and T3. If the judgment is yes, it means that the time difference of the current data queue is less than the time difference of the candidate queue, so it is impossible to prove the optimality of the time matching of the data in the current candidate queue, so CC is set to zero, the sensor data in each sensor's respective historical data queue is stored in the corresponding current data queue, and the current round of data synchronization process is exited, where CC represents the number of non-empty data queues in the current data queue of each sensor. If the judgment is no, then exit the current round of data synchronization process directly, and continue to judge in the next round. For example, the current candidate queue no longer meets the premise of the maximum time difference, indicating that the candidate message no longer meets the condition of belonging to the same frame, that is, the data in the candidate queue does not belong to the same frame, so it exits directly.

[0065] In addition, in the above embodiment, after the sensor data in the candidate queue is published as synchronized sensor data, the method further includes: clearing the sensor data in the candidate queue; setting CC to 0; storing the sensor data in each sensor's respective historical data queue into the corresponding current data queue; deleting the earliest arriving sensor data in each sensor's respective current data queue; and counting the number of non-empty data in each sensor's current data queue and assigning it to CC. That is, after publishing, the next round of time synchronization processing begins, so the value of CC needs to be reset, and the sensor data previously copied to the candidate queue is stored in the historical data queue, so it can be deleted after being put back into the current data queue.

[0066] On the basis of the above-mentioned embodiments, before copying the earliest arriving sensor data in each current data queue to the candidate queue, the method of the present application also needs to determine whether there is already sensor data in the candidate queue. As described above, sensor data is constantly being collected and received, so there will always be data stored in the current data queue. At the same time, the data synchronization method is also iteratively executed according to a certain period and frequency. In addition, in different situations, the data in the candidate queue may not pass the time matching judgment and is not released to the perception module. Therefore, when executing the time synchronization method, it is necessary to first determine whether there is already sensor data in the candidate queue.

[0067] Specifically, before copying the earliest arriving sensor data in each current data queue to the candidate queue, the method further includes: determining whether there is already sensor data in the candidate queue; if the determination is no, and the difference between T2 and T1 does not exceed the preset threshold, executing the step of copying the earliest arriving sensor data in each current data queue to the candidate queue. Wherein, if the difference between T2 and T1 does not exceed the preset threshold, it means that the time of the messages arriving before and after in the current data queue does not exceed the limit, so the next step of processing can be performed. That is, when there is no sensor data in the current candidate queue, and the difference between T2 and T1 does not exceed the preset threshold, the candidate is directly generated.

[0068] If there is already sensor data in the candidate queue, it is necessary to determine whether there is a more suitable candidate in the current data queue. In the embodiment of the present application, the judgment is made by the time difference between the sensor data. That is, it is determined whether the difference between T2 and T4 is greater than or equal to the difference between T1 and T3. If the judgment is yes, it indicates that the time difference of the current data queue exceeds the time difference of the candidate queue. Therefore, the candidate queue is better, and the step of storing the earliest arriving sensor data in each current data queue into each sensor's respective historical data queue is executed without updating the candidate.

[0069] If there is sensor data in the candidate queue, and the difference between T2 and T4 is less than the difference between T1 and T3, it indicates that the current data queue is better, so the sensor data in the candidate queue is deleted, and the earliest arriving sensor data in each current data queue is copied to the candidate queue again, that is, a new candidate is generated, and then T3 and T4 are updated according to the new candidate queue, and the step of storing the earliest arriving sensor data in each current data queue into each sensor's respective historical data queue is executed.

[0070] Based on the above embodiments, before determining the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue, the method further includes: determining whether CC is equal to the number of sensors; if the determination is yes, then executing the step of determining the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue. Thus, the value of CC can be verified again before performing the time synchronization process to ensure the effective execution of the process.

[0071] Meanwhile, in the above embodiment, when there is no sensor data in the candidate queue, it is necessary to further determine whether the difference between T2 and T1 exceeds the preset threshold. In this embodiment, if the difference between T2 and T1 exceeds the preset threshold, it indicates that the time of the data arriving before and after in the current data queue has exceeded the limit, therefore, the earliest arriving sensor data among all sensor data is deleted, and the process returns to the step of determining whether CC is equal to the number of sensors.

[0072] In one implementation, before entering the time synchronization process, some possible abnormal situations need to be processed to ensure the accuracy of the time synchronization process.

[0073] On the one hand, before determining whether CC is equal to the number of sensors, the method further includes: if more than one sensor data of the same sensor is stored in the corresponding current data queue at the same time, then determining the time sequence between the more than one sensor data and giving an alarm. After that, the sensor data can be returned to be acquired again. Thus, the situation where two or more sensor data are acquired at the same time due to sensor abnormality can be avoided, thereby avoiding errors in subsequent time synchronization processing.

[0074] On the other hand, before determining whether CC is equal to the number of sensors, the method further includes: if there is any sensor, the sum of the sensor data in the current data queue and the historical data queue corresponding to it is greater than the preset accommodating threshold, which indicates that the data of the sensor has not entered the time synchronization processing, and a backlog of data has occurred. It is necessary to store the sensor data in its historical data queue into its current data queue, update the value of CC, and add a discard mark to the current sensor data in its current data queue before entering the time synchronization processing flow. Correspondingly, if the sum of the sensor data in the current data queue and the historical data queue corresponding to any sensor is not greater than the accommodating threshold, the method returns to execute the step of storing the sensor data in each sensor's respective receiving queue into each sensor's respective current data queue in the order of their respective reception. That is, as long as it is not greater than the accommodating threshold, the sensor data will continue to be acquired, and the time synchronization processing flow will not be entered temporarily.

[0075] Correspondingly, before determining whether there is already sensor data in the candidate queue, the method also includes: traversing all sensor data in each current data queue, and if there is sensor data with a discard mark at the end of the current data queue, keeping the discard mark of the sensor data; if the sensor data with a discard mark is not the last frame of data among all sensor data, clearing the discard mark.

[0076] Correspondingly, before copying the earliest arriving sensor data in each current data queue to the candidate queue, the method also includes: determining whether there is a discard mark; if the determination is no, executing the step of copying the earliest arriving sensor data in each current data queue to the candidate queue; if the determination is yes, deleting the earliest arriving sensor data among all sensor data, and returning to execute the step of determining whether CC is equal to the number of sensors.

[0077] Specifically, as described above, the current data queue with a discard mark indicates that there is a backlog of data. In this embodiment, before determining whether there is already sensor data in the candidate queue, the sensor data in each current data queue is first traversed. As long as the sensor data with a discard mark is not the last frame of all data, it indicates that new data has entered the current data queue. In this case, the sensor data was not matched in the previous match and can be matched with the latest data, so the discard mark can be cleared. If the sensor data with a discard mark is at the end of the current data queue, it means that no new data has been added to the current data queue, that is, the current data queue has not been updated, so the discard mark of the sensor data is maintained. Then, before copying the earliest arriving sensor data in each current data queue to the candidate queue, it is also necessary to determine whether there is a discard mark. If the judgment is no, the step of copying the earliest arriving sensor data in each current data queue to the candidate queue is executed; if the judgment is yes, it means that there is a data backlog, therefore, the earliest arriving sensor data among all sensor data is deleted because the data has become invalid data, and then the step of determining whether CC is equal to the number of sensors is returned to execute, that is, re-entering the time synchronization processing flow.

[0078] Figure 5 This is an overall logic diagram of the multi-sensor data synchronization method provided in the embodiment of the present application. Figure 5 As shown, the vehicle-side sensors may include cameras, lidars, 4D millimeter waves or ultrasonic waves, etc. Each sensor has its own corresponding thread for reading sensor data, and stores the data in each sensor's respective receiving queue. When the receiving queue is not empty, the thread that reads the sensor data will notify the time synchronization thread to store the sensor data in each sensor's respective current data queue. Whenever the current data queue is not empty, CC is incremented by 1. When CC is equal to the number of sensors, the time synchronization process can be entered. If sensor data with time matching is found, the data will be published to the vehicle-side perception system for fusion perception.

[0079] Figure 6 This is a schematic diagram of the main process of the time synchronization thread of the multi-sensor data synchronization method provided in the embodiment of the present application. Figure 6As shown, after the process starts, it is determined whether the sensor receiving queue is empty. If it is empty, the thread waits. If it is not empty, the sensor data in the receiving queue is copied. Then it is determined whether the copied sensor data is legal. Specifically, it is first determined whether the corresponding current data queue is not empty. If it is not empty, the copied sensor data is read and it is determined whether there is data of the same type as the sensor data in each current data queue. If so, it means that the copied sensor data is legal data and can be stored in the current data queue of the corresponding sensor. Otherwise, it means that it is illegal, that is, unknown sensor data is received.

[0080] Furthermore, on the one hand, it is determined whether the same sensor receives a sensor data at the same time. If it is not, the time sequence between more than one sensor data is determined and an alarm is issued. Otherwise, CC is increased by 1 to determine whether CC is equal to the number of sensors. If it is equal, the time synchronization process is entered. On the other hand, if the time synchronization process has not been entered, it is also necessary to determine whether the sum of the sensor data in the current data queue and the historical data queue of the sensor is greater than the preset accommodating threshold. If it is not, it returns. Otherwise, CC is set to zero, the sensor data in its historical data queue is stored in its current data queue, the value of CC is updated, and a discard mark is added to its current data queue, and then the time synchronization process is entered.

[0081] Figure 7 Schematic diagram of the time synchronization process of the multi-sensor data synchronization method provided in the embodiment of the present application. Figure 7 As shown, after the process starts, first determine whether CC is equal to the number of sensors. If so, determine the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue. Traverse all sensor data in each current data queue. If there is a discarded sensor data at the end of the queue in the current data queue, which is not the last frame of all sensor data, clear the discarded mark.

[0082] Afterwards, it is determined whether there is sensor data in the candidate queue. If there is no sensor data in the candidate queue, it is first determined whether the difference between T2 and T1 exceeds the preset threshold, and if it does not exceed, it is further determined whether there is a discard mark. If there is no discard mark, a candidate is generated. If it exceeds the threshold, or there is a discard mark, the earliest arriving data among all sensor data is deleted, and then the step of determining whether CC is equal to the number of sensors is returned. After the candidate is generated, the step of storing the earliest arriving sensor data in each current data queue into each sensor's respective historical data queue is executed. If there is sensor data in the candidate queue, it means that the candidate was not released last time, so it is determined whether the difference between T2 and T4 is greater than or equal to the difference between T1 and T3. If it is determined to be yes, it means that the candidate is better, and the step of storing the earliest arriving sensor data in each current data queue into each sensor's respective historical data queue is executed. If it is determined to be no, it means that the data in the current data queue is better, then a new candidate is generated, and T3 and T4 are updated.

[0083] After storing the earliest arriving sensor data in each current data queue into each sensor's respective historical data queue, determine whether the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub. If so, directly publish the candidate. Otherwise, determine whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3. If so, directly publish the candidate. Otherwise, calculate the earliest arrival timestamp theoretical value T5 and the latest arrival timestamp theoretical value T6 in the current data queue of each sensor according to the data acquisition frequency of each sensor and the frequency of receiving sensor data, and then determine whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3. If so, directly publish the candidate. Otherwise, further determine whether the difference between T6 and T4 is less than the difference between T5 and T3. If so, set CC to zero, store the sensor data in each sensor's respective historical data queue into the corresponding current data queue, and exit this round of data synchronization process. Otherwise, directly exit this round of data synchronization process.

[0084] In summary, in order to more accurately and precisely realize the perception needs of intelligent driving, the technical solution of the embodiment of the present application, after acquiring various types of sensor data on the vehicle side, uniformly performs time synchronization processing to eliminate the influence of sensor transmission delay, sensor downtime causing data loss, etc., to ensure the synchronization of various types of sensor data entering the perception system, so as to obtain effective information from various sensors at the same time, so that the autonomous driving system can correctly perceive the external environment information, and ensure the fusion, planning and decision-making of multi-sensor data of autonomous driving. Specifically, the embodiment of the present application has the following technical effects:

[0085] 1. The concurrent requirements of the program are guaranteed by using N+1 threads. N threads continuously obtain data from N sensors on the vehicle side. Each sensor is equipped with 1 thread. Then another thread is started to receive and store the sensor data. The sensor data that meets the conditions is time synchronized and the synchronized data is sent to the vehicle-side perception system. Therefore, it is ensured that the multi-channel sensors do not interfere with each other and the channels are unobstructed.

[0086] 2. It can achieve time synchronization for data from different sensors, which can ensure that on the basis of good sensor hardware or software synchronization, it can compensate for the delay caused by data transmission of various sensors, solve the adverse consequences caused by the downtime of some sensors, and ensure the robustness of the perception system.

[0087] 3. It can efficiently ensure the efficiency and accuracy of data time matching when the vehicle-mounted system obtains multi-sensor data. For the same type of sensors, the main sensor data can arrive at the time synchronizer one after another, and there is no need to ensure that a certain sensor must arrive first, thereby being compatible with the situation where some sensors are slightly delayed at certain times, ensuring the robustness of the time synchronizer and ensuring the efficient operation of the perception system.

[0088] Figure 8 This is a schematic diagram of the structure of the multi-sensor data synchronization device provided in the embodiment of the present application. Figure 8 As shown, the multi-sensor data synchronization device 80 includes:

[0089] The data acquisition module 810 is used to store the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to the receiving order;

[0090] The candidate acquisition module 820 is used to copy the earliest arriving sensor data in each current data queue to the candidate queue when each current data queue is not empty;

[0091] A hub determination module 830, configured to use the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub, and use the latest arriving timestamp of the sensor data in the current data queue of the hub as the hub time;

[0092] A historical data storage module 840 is used to store the earliest arriving sensor data in each current data queue into the historical data queue of each sensor;

[0093] The matching processing module 850 is used to publish the sensor data in the candidate queue as synchronized sensor data if the earliest arriving sensor data in each current data queue is the sensor data corresponding to the hub, or if sensor data arrives in each current data queue before and after the hub time.

[0094] In some embodiments, the apparatus further comprises:

[0095] A first timestamp determination module, configured to determine the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue before the candidate acquisition module 820 copies the earliest arriving sensor data in each current data queue to the candidate queue;

[0096] Accordingly, the device further comprises:

[0097] A second timestamp determination module, configured to obtain the earliest arrival timestamp T3 and the latest arrival timestamp T4 of each sensor data in the candidate queue before the hub determination module 830 uses the sensor name corresponding to the latest arriving sensor data in each current data queue as the hub;

[0098] Accordingly, the matching processing module 850 is also used for:

[0099] If the earliest arriving sensor data in each current data queue is not the sensor data corresponding to the hub, determine whether the difference between T2 and T4 is greater than or equal to the difference between the hub time and T3, and when the judgment is yes, determine that sensor data has arrived in each current data queue before and after the hub time.

[0100] In some embodiments, the apparatus further comprises:

[0101] A third timestamp determination module is used to calculate the earliest arrival timestamp theoretical value T5 and the latest arrival timestamp theoretical value T6 in the current data queue of each sensor according to the data collection frequency of each sensor and the frequency of receiving sensor data if the difference between T2 and T4 is less than the difference between the hub time and T3;

[0102] The first judgment processing module is used to judge whether the difference between T6 and T4 is greater than or equal to the difference between the hub time and T3; if it is judged to be yes, the sensor data in the candidate queue is published as synchronized sensor data.

[0103] In some embodiments, the device further includes a second judgment processing module, specifically configured to:

[0104] If the difference between T6 and T4 is less than the difference between the hub time and T3, then determine whether the difference between T6 and T4 is less than the difference between T5 and T3;

[0105] If the judgment is yes, CC is set to zero, the sensor data in the historical data queue of each sensor is stored in the corresponding current data queue, and the current round of data synchronization process is exited, wherein CC represents the number of non-empty data in the current data queue of each sensor;

[0106] If the answer is no, exit this round of data synchronization process directly.

[0107] In some embodiments, the apparatus further includes a publishing processing module, specifically configured to:

[0108] Clearing the sensor data in the candidate queue;

[0109] Set CC to 0;

[0110] Storing the sensor data in the respective historical data queue of each sensor into the corresponding current data queue;

[0111] Deleting the earliest arriving sensor data in the current data queue of each sensor;

[0112] The number of non-empty data in the current data queue of each sensor is counted and assigned to CC.

[0113] In some embodiments, the device further includes a third judgment processing module, specifically configured to:

[0114] Before the candidate acquisition module 820 copies the earliest arriving sensor data in each current data queue to the candidate queue, it is determined whether there is already sensor data in the candidate queue;

[0115] If the judgment is no, and the difference between T2 and T1 does not exceed the preset threshold, the candidate acquisition module 820 copies the earliest arriving sensor data in each current data queue to the candidate queue.

[0116] In some embodiments, the device further includes a fourth judgment processing module, specifically configured to:

[0117] If there is sensor data in the candidate queue, determine whether the difference between T2 and T4 is greater than or equal to the difference between T1 and T3;

[0118] If the answer is yes, the historical data storage module 840 stores the earliest arriving sensor data in each current data queue into the historical data queue of each sensor.

[0119] In some embodiments, the device further includes a fifth judgment processing module, specifically configured to:

[0120] If the difference between T2 and T4 is smaller than the difference between T1 and T3, the sensor data in the candidate queue is deleted, and the earliest arriving sensor data in each current data queue is copied to the candidate queue again;

[0121] T3 and T4 are updated, and the historical data storage module 840 stores the earliest arriving sensor data in each current data queue into the historical data queue of each sensor.

[0122] In some embodiments, the device further includes a sixth judgment processing module, specifically configured to:

[0123] Before the first timestamp determination module determines the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue, determining whether CC is equal to the number of sensors;

[0124] If the answer is yes, the first timestamp determination module determines the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in the current data queues.

[0125] In some embodiments, the device further includes a seventh judgment processing module, specifically configured to:

[0126] If the difference between T2 and T1 exceeds the preset threshold, the earliest arriving sensor data among all sensor data is deleted, and the sixth judgment processing module is returned to judge whether CC is equal to the number of sensors.

[0127] In some embodiments, the apparatus further comprises:

[0128] The alarm module is used to determine the time sequence between the more than one sensor data and issue an alarm before the sixth judgment processing module determines whether CC is equal to the number of sensors if more than one sensor data of the same sensor is stored in the corresponding current data queue at the same time.

[0129] In some embodiments, the apparatus further comprises:

[0130] A discarding marking module is used for storing the sensor data in the historical data queue into the current data queue, updating the value of CC, and adding a discarding mark to the current data queue if the sum of the sensor data in the current data queue and the historical data queue corresponding to any sensor is greater than a preset accommodating threshold before the sixth judgment processing module determines whether CC is equal to the number of sensors;

[0131] Accordingly, the device further comprises:

[0132] a traversal module, configured to traverse all sensor data in each current data queue before the third judgment processing module determines whether there is already sensor data in the candidate queue, and if the sensor data with a discard mark is located at the end of the current data queue, keep the discard mark of the sensor data; if the sensor data with a discard mark is not the last frame of data among all the sensor data, clear the discard mark;

[0133] Accordingly, the device further includes an eighth judgment processing module, which is specifically configured to:

[0134] Before the candidate acquisition module 820 copies the earliest arriving sensor data in each current data queue to the candidate queue, determining whether the discard mark exists;

[0135] If the judgment is no, the candidate acquisition module 820 copies the earliest arriving sensor data in each current data queue to the candidate queue;

[0136] If the answer is yes, the earliest arriving sensor data among all the sensor data is deleted, and the process returns to the sixth judgment processing module to judge whether CC is equal to the number of sensors.

[0137] In some embodiments, the apparatus further comprises:

[0138] The accommodative threshold exceeding processing module is used to return the sensor data in each sensor's respective receiving queue as described by the data acquisition module 810 and store them into each sensor's respective current data queue in their respective receiving order if the sum of the sensor data in the current data queue and the historical data queue corresponding to any sensor is not greater than the accommodative threshold.

[0139] In some embodiments, the data acquisition module 810 includes:

[0140] A storage unit, used for acquiring sensor data from the respective threads of each sensor, and storing the data in the respective receiving queue of each sensor;

[0141] The copy unit is used for copying the sensor data in the receiving queue to the current data queue of each sensor according to the respective receiving order when the receiving queue is not empty.

[0142] The multi-sensor data synchronization device provided in the embodiment of the present application can be used to execute the technical solution of the multi-sensor data synchronization method in the above embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0143] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the matching processing module 850 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above matching processing module 850. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0144] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0145] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0146] The processor 122 executes the computer-executable instructions stored in the memory, so that the processor 122 executes the scheme in the above embodiment. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital signal processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0147] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0148] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0149] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0150] The electronic device provided in the embodiment of the present application may be the terminal device of the above embodiment.

[0151] An embodiment of the present application further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on a computer, the computer executes the technical solution of the multi-sensor data synchronization method of the above embodiment.

[0152] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, the technical solution of the multi-sensor data synchronization method in the above embodiment can be implemented.

[0153] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present application, and the programming language includes an object-oriented programming language, such as Java, Smalltalk, C++, and also includes a conventional procedural programming language, such as "C" language or similar programming language. The program code can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0154] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A multi-sensor data synchronization method, It is characterized in that include: The sensor data in the receiving queue of each sensor are stored in the current data queue of each sensor according to the receiving order; When all current data queues are not empty, the first round of synchronization processing flow is as follows: The earliest arriving sensor data in each current data queue is copied to the candidate queue, and the sensor name corresponding to the latest arriving sensor data in the candidate queue is used as the hub, and the latest arrival timestamp of the sensor data in the current data queue of the hub is used as the hub time, and the earliest arrival timestamp T3 and the latest arrival timestamp T4 of each sensor data in the candidate queue are obtained; Storing the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor; If the sensor data in the candidate queue is not time-synchronized data, the sensor data in the candidate queue is not published, and the first round of synchronization processing flow is exited; Continue to store the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to the receiving order; If there is any sensor whose corresponding sum of sensor data in the current data queue and historical data queue is greater than the preset accommodating threshold, the sensor data in the historical data queue is stored in the current data queue, the current sensor data in the current data queue is marked as discarded, and the earliest arriving sensor data among all sensor data is deleted; the value of CC is updated, where CC represents the number of non-empty data in the current data queue of each sensor; The second round of synchronization process is as follows: Determine the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue of each sensor; Determine whether there is sensor data in the candidate queue; If there is sensor data in the candidate queue, determine whether the difference between T2 and T4 is greater than or equal to the difference between T1 and T3; If not, update the candidate queue; If the sensor data of the hub becomes the earliest arriving sensor data, the sensor data in the updated candidate queue is published as synchronized sensor data.

2. The method according to claim 1, It is characterized in that The second round of synchronization process also includes: storing the earliest arriving sensor data in each current data queue into the respective historical data queue of each sensor; and After publishing the sensor data in the updated candidate queue as synchronized sensor data, clearing the sensor data in the updated candidate queue; Set CC to 0; delete the data in the historical data queue.

3. The method according to claim 2, It is characterized in that The second round of synchronization process also includes: Before determining the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue, determining whether CC is equal to the number of sensors; If the answer is yes, the step of determining the earliest arrival timestamp T1 and the latest arrival timestamp T2 of all sensor data in each current data queue is performed.

4. The method according to claim 1, It is characterized in that The step of storing the sensor data in the receiving queue of each sensor into the current data queue of each sensor according to the receiving order of each sensor includes: Acquire sensor data from the respective threads of each sensor and store the data in the respective receiving queue of each sensor; When the receiving queue is not empty, the sensor data in the receiving queue are copied to the current data queue of each sensor according to the respective receiving order.

5. An electronic device, It is characterized in that include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the multi-sensor data synchronization method according to any one of claims 1 to 4.

6. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the multi-sensor data synchronization method according to any one of claims 1 to 4.

7. A computer program product, It is characterized in that The invention comprises a computer program, which, when executed by a processor, implements the multi-sensor data synchronization method according to any one of claims 1 to 4.

Images