Time synchronization method and device for sensing data and unmanned vehicle

Abstract

The invention provides a time synchronization method and device for sensing data and an unmanned vehicle, and the method comprises the steps: determining a reference sensing data frame of reference sensing equipment in a plurality of pieces of sensing equipment, based on the time interval between the trigger timestamp of the reference sensing data frame and the trigger timestamps of the sensing data frames of other sensing devices in the plurality of sensing devices, determining first synchronization frames corresponding to the plurality of sensing devices respectively; respectively and sequentially acquiring sensing data frames after the corresponding first synchronization frame acquired by a plurality of sensing devices; and for the current sensing data frames respectively corresponding to the plurality of sensing devices obtained each time, under the condition that each current sensing data frame accords with a preset time sequence rule, determining that the sensing data synchronization of the time sequence corresponding to each current sensing data frame is completed. According to the invention, the time synchronization of the sensing data collected by multiple sensing devices is realized efficiently and precisely.

Description

Technical Field

[0001] The present disclosure relates to the fields of smart mines, autonomous driving, and perception technology, and in particular to a method and device for time synchronization of perception data and an unmanned vehicle. Background Art

[0002] Currently in the field of autonomous driving, autonomous driving systems usually rely on data from different sensors to achieve environmental perception. For example, lidar may provide accurate 3D point cloud data, cameras provide visual images, and radar can help provide distance and speed measurements. In order to perform effective data fusion, the data of all sensors need to be aligned under the same time reference. Time synchronization ensures that the time scales of these sensor data are consistent to avoid data misalignment. Secondly, the autonomous driving system needs to perceive the surrounding environment in real time and make driving decisions based on these perception results. Any deviation in timing will lead to inaccurate control instructions or response delays, which may affect the driving safety of the vehicle. For example, the time asynchrony of sensor data may cause the vehicle to react laggingly to road conditions, or the calculated position of surrounding objects is inaccurate. In addition, perception methods based on deep learning have been widely used in the field of unmanned driving for accurate perception of the vehicle's surrounding environment. Perception methods based on deep learning have high requirements for raw data when used for different tasks. For example, the time series model requires time series data input and sensor calibration parameters and vehicle posture information at the corresponding time, and the multimodal model requires multi-sensor data, sensor calibration parameters and vehicle posture information at the corresponding time, and the matching accuracy of this information in time is also required to be high. Therefore, it is necessary to perform high-precision time synchronization on the sensor data. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, device and unmanned vehicle for time synchronization of perception data, so as to solve the related problems existing in the existing solutions.

[0004] Based on the above problem, in a first aspect, a time synchronization method for sensing data is provided, including:

[0005] Determine a reference perception data frame of a reference perception device among the multiple perception devices, and determine first synchronization frames corresponding to the multiple perception devices respectively based on time intervals between trigger timestamps of the reference perception data frame and trigger timestamps of perception data frames of other perception devices among the multiple perception devices;

[0006] Sequentially acquiring the perception data frames collected by the plurality of perception devices after the corresponding first synchronization frame respectively;

[0007] For the current perception data frames corresponding to the multiple perception devices obtained each time, when each current perception data frame complies with the preset timing rules, it is determined that the perception data synchronization corresponding to the timing of each current perception data frame is completed; wherein the preset timing rules represent that the time interval between the trigger timestamps of each two adjacent perception data frames in the perception data frame sequence corresponding to each perception device is within a preset range.

[0008] In combination with the first aspect, in a possible implementation, determining the first synchronization frames corresponding to the multiple perception devices respectively based on the time interval between the trigger timestamp of the reference perception data frame and the trigger timestamp of the perception data frames of other perception devices in the multiple perception devices includes:

[0009] Acquire the sensing data frames collected by the multiple sensing devices;

[0010] Traversing the perception data frames collected by the other perception devices among the multiple perception devices except the reference perception device, and determining a first perception data frame whose time interval between a trigger timestamp and a trigger timestamp of the reference perception data frame is less than a first preset time threshold;

[0011] When each other perception device determines the corresponding first perception data frame, the first perception data frame is determined as the first synchronization frame corresponding to the other perception device, and the reference perception data frame is determined as the first synchronization frame corresponding to the reference perception device.

[0012] In combination with the first aspect, in a possible implementation manner, acquiring the perception data frames collected by the multiple perception devices includes:

[0013] Acquiring the perception data frames collected by the plurality of perception devices and storing them in a perception data set includes:

[0014] If the acquired perception data frame is not the reference perception data frame collected by the reference perception device, storing the acquired perception data frame into the perception data set; or,

[0015] When the acquired perception data frame is a reference perception data frame collected by the reference perception device and the reference perception data frame does not exist in the current perception data set, the acquired reference perception data frame is stored in the perception data set, and the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set are updated.

[0016] In combination with the first aspect, in a possible implementation manner, acquiring the perception data frames collected by the multiple perception devices includes:

[0017] Acquiring the perception data frames collected by the plurality of perception devices and storing them in a perception data set includes:

[0018] In the case where the acquired perception data frame is a reference perception data frame collected by the reference perception device, and there is a historical reference perception data frame in the current perception data set, traverse all perception data frames in the perception data set, remove perception data frames whose time interval between a trigger timestamp and a trigger timestamp of a newly acquired reference perception data frame is greater than a second preset time threshold from the perception data set, store the newly acquired reference perception data frame in the perception data set, and update the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set;

[0019] The determining of a reference sensing data frame of a reference sensing device among the plurality of sensing devices comprises:

[0020] When it is determined that the perception data set stores the perception data frames collected by each perception device, a reference perception data frame of a reference perception device is determined from the perception data set according to a storage position of a most recently updated reference perception data frame in the perception data set.

[0021] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0022] After determining the first synchronization frames respectively corresponding to the multiple sensing devices, the reference sensing data frame corresponding to the reference sensing device and the corresponding first sensing data frames determined for the other sensing devices are removed from the sensing data set.

[0023] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0024] When any other perception device fails to determine the corresponding first perception data frame, determine the reference perception data frame that is most recently updated after the currently used reference perception data frame; and continue to traverse the perception data frames currently collected by the other perception devices to determine the first perception data frame whose time interval between a trigger timestamp and a trigger timestamp of the reference perception data frame that is most recently updated after the currently used reference perception data frame is less than a first preset time threshold.

[0025] In combination with the first aspect, in a possible implementation, it also includes: in the event that any other perception device determines at least two corresponding first perception data frames, the first perception data frame among the at least two corresponding first perception data frames whose time interval with the trigger timestamp of the reference perception data frame is the smallest is determined as the first perception data frame corresponding to the other perception device.

[0026] In combination with the first aspect, in a possible implementation, for each current perception data frame corresponding to the multiple perception devices obtained respectively, when each current perception data frame meets a preset timing rule, determining to complete the perception data synchronization of the timing corresponding to each current perception data frame includes:

[0027] For each current perception data frame corresponding to the multiple perception devices respectively acquired each time, determine whether a difference between a first trigger timestamp of the current perception data frame and a second trigger timestamp of a previous perception data frame of the current perception data frame of the corresponding perception device is less than a third preset time threshold;

[0028] When the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time threshold, it is determined that the perception data synchronization of the corresponding timing of each current perception data frame is completed.

[0029] In combination with the first aspect, in a possible implementation manner, the method further includes:

[0030] In the case where the difference between the first trigger timestamp and the corresponding second trigger timestamp of the current perception data frame of at least one first perception device is greater than or equal to the third preset time threshold, determining that the perception data synchronization of the time sequence corresponding to the current perception data frame fails;

[0031] Taking the current perception data frame corresponding to the at least one first perception device as the latest acquired new current perception data frame, and determining a corresponding second trigger timestamp for the new current perception data frame;

[0032] Determine a second sensing device among the multiple sensing devices, for which a difference between a first trigger timestamp of a current sensing data frame and a corresponding second trigger timestamp is less than the third preset time threshold;

[0033] Acquire a new current perception data frame for the second perception device, and continue to execute the step of determining whether the difference between the first trigger timestamp of the new current perception data frame and the corresponding second trigger timestamp is less than the third preset time threshold for each new current perception data frame corresponding to the first perception device and the second perception device respectively acquired most recently.

[0034] In combination with the first aspect, in a possible implementation manner, before acquiring a new current perception data frame for the second perception device, the method further includes:

[0035] Determine a minimum trigger timestamp among first trigger timestamps respectively corresponding to current perception data frames of the at least one first perception device;

[0036] Compare the first trigger timestamp of the current perception data frame corresponding to the second perception device with the minimum trigger timestamp;

[0037] The acquiring a new current perception data frame for the second perception device includes:

[0038] If there is a second perception device that satisfies that the difference between the first trigger timestamp of the current perception data frame corresponding to the second perception device and the minimum trigger timestamp is greater than or equal to the fourth preset time threshold, obtain a new current perception data frame for the second perception device;

[0039] The method further comprises:

[0040] When there is a second perception device, and the difference between the first trigger timestamp of the current perception data frame corresponding to the second perception device and the minimum trigger timestamp is less than the fourth preset time threshold, the current perception data frame corresponding to the second perception device is used as the new current perception data frame most recently obtained by the corresponding second perception device.

[0041] In combination with the first aspect, in a possible implementation, taking the current perception data frame corresponding to the at least one first perception device as a new current perception data frame to be acquired next time, and determining a corresponding second trigger timestamp for the new current perception data frame, includes:

[0042] The current perception data frame corresponding to the at least one first perception device is used as the new current perception data frame to be acquired next time, and the first trigger timestamp of the current perception data frame corresponding to the second perception device is determined as the second trigger timestamp corresponding to the new current perception data frame of the at least one first perception device.

[0043] In combination with the first aspect, in a possible implementation, when the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time interval threshold, before determining that the perception data frame synchronization of the corresponding timing of each current perception data frame is completed, it also includes:

[0044] Respectively determine a difference between a first trigger timestamp of a first current perception data frame other than a current reference perception data frame corresponding to the reference perception device in each current perception data frame and a first trigger timestamp of the current reference perception data frame;

[0045] The step of determining to complete the synchronization of the perception data frames corresponding to the timing of each current perception data frame includes:

[0046] When the difference between the first trigger timestamp of the first current perception data frame and the first trigger timestamp of the current reference perception data frame is less than a fifth preset time threshold, determining that the perception data frame synchronization of the corresponding timing of each current perception data frame is completed; or,

[0047] The method further comprises:

[0048] When there is a difference between at least one first trigger timestamp of the first current perception data frame and the first trigger timestamp of the current reference perception data frame that is greater than or equal to a fifth preset time threshold, it is determined that the perception data frame synchronization of the timing corresponding to the current perception data frame fails.

[0049] In combination with the first aspect, in a possible implementation manner, when the current perception data frames respectively corresponding to the multiple perception devices are the first synchronization frames, the method further includes:

[0050] According to the first trigger timestamps respectively corresponding to the first synchronization frames and the acquisition frequency of the sensing device to which the first synchronization frames belong, the second trigger timestamps respectively corresponding to the first synchronization frames are determined.

[0051] In a second aspect, a time synchronization device for sensing data is provided, including:

[0052] A first frame synchronization module, used to determine a reference perception data frame of a reference perception device among multiple perception devices, and determine the first synchronization frames corresponding to the multiple perception devices respectively based on the time intervals between the trigger timestamps of the reference perception data frame and the trigger timestamps of the perception data frames of other perception devices among the multiple perception devices;

[0053] A data acquisition module, used to sequentially acquire the perception data frames collected by the plurality of perception devices after the corresponding first synchronization frame;

[0054] A time synchronization module is used to determine the completion of perception data synchronization corresponding to the timing of each current perception data frame corresponding to the multiple perception devices obtained each time, when each current perception data frame complies with a preset timing rule; wherein the preset timing rule indicates that the time interval between the trigger timestamps of each two adjacent perception data frames in the perception data frame sequence corresponding to each perception device is within a preset range.

[0055] In a third aspect, an unmanned vehicle is provided, comprising a time synchronization device for perception data according to the second aspect, or in combination with any possible implementation of the second aspect.

[0056] The beneficial effects of the embodiments of the present disclosure include:

[0057] The embodiment of the present disclosure provides a method, device and unmanned vehicle for time synchronization of perception data, which are applied to the scene of autonomous driving. When the quality of data collected by different perception devices during the driving of the unmanned vehicle is poor, such as the inconsistent triggering time of each perception device and / or the jump of the triggering time of each perception device, how to achieve time synchronization of different perception data. The time synchronization method of perception data provided by the present disclosure firstly performs first frame synchronization based on the reference perception data frame to obtain the first synchronization frame corresponding to each perception device, and determines the reference data frame for subsequent frame synchronization for each perception device, so that the synchronization is more efficient; then, the perception data frames collected by the perception device after the first synchronization frame corresponding to each perception device are sequentially acquired; finally, it is judged whether the time interval between the triggering timestamps of each two adjacent perception data frames in the perception data frame sequence corresponding to each perception device is within the preset range, and if it is in accordance with the range, it is determined that the current perception data frame acquired this time can complete the synchronization. First, the first synchronization data frame as the synchronization reference is determined for multiple perception devices, and for each perception device, the subsequent data frames are synchronized based on their respective first synchronization data frames, so as to efficiently and accurately realize the time synchronization of the perception data collected by multiple perception devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A flowchart of a method for time synchronization of perception data provided by an embodiment of the present disclosure;

[0059] Figure 2a to Figure 2b A schematic diagram of the sensing device assembly and coverage provided by an embodiment of the present disclosure;

[0060] Figure 3 A schematic diagram of a sensing device that experiences frame skipping according to an embodiment of the present disclosure;

[0061] Figure 4 A flowchart of first frame synchronization provided by an embodiment of the present disclosure;

[0062] Figure 5 A schematic diagram of frame loss and subsequent synchronization of a presence sensing device provided in an embodiment of the present disclosure;

[0063] Figure 6 A schematic diagram of frame skipping and subsequent synchronization of a presence sensing device provided in an embodiment of the present disclosure;

[0064] Figure 7 A flowchart of data time synchronization provided for an embodiment of the present disclosure;

[0065] Figure 8 A schematic diagram of a time synchronization device for sensing data provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0066] The embodiments of the present disclosure provide a method, device and unmanned vehicle for time synchronization of perception data. The preferred embodiments of the present disclosure are described below in conjunction with the drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0067] The present disclosure provides a method for time synchronization of perception data, such as Figure 1 As shown, including:

[0068] S101, determining a reference perception data frame of a reference perception device among multiple perception devices, and determining first synchronization frames corresponding to the multiple perception devices respectively based on the time intervals between the trigger timestamps of the reference perception data frame and the trigger timestamps of the perception data frames of other perception devices among the multiple perception devices;

[0069] S102, sequentially acquiring the sensing data frames collected by the plurality of sensing devices after the corresponding first synchronization frame;

[0070] S103. For the current perception data frames corresponding to the multiple perception devices obtained each time, when each current perception data frame complies with the preset timing rule, determine that the perception data synchronization corresponding to the timing of each current perception data frame is completed; wherein the preset timing rule indicates that the time interval between the trigger timestamps of each two adjacent perception data frames in the perception data frame sequence corresponding to each perception device is within a preset range.

[0071] In the embodiments of the present disclosure, when the autonomous driving unmanned vehicle is driving, each perception device will collect data about the surrounding environment to obtain corresponding perception data frames. During the process of perception device collection, since the trigger times of different perception devices will fluctuate, there will be a problem of time asynchrony of the corresponding perception data frames. Therefore, the present disclosure provides a time synchronization method for perception data, which realizes efficient and high-precision time synchronization of data from multiple perception devices when there is frame loss, frame skipping, and frequency instability in the data of multiple perception devices. During implementation, first determine the reference sensing device among multiple sensing devices, and use the sensing data frame of the reference device as the reference sensing data frame, and select the first synchronization frame corresponding to each sensing device based on the trigger timestamp of the reference sensing data frame, complete the first frame synchronization with strict judgment logic, remove the sensing data frames collected by each sensing device in an unstable running state when the sensing device starts running, and speed up the synchronization; then, for each sensing device, sequentially obtain the sensing data frames after the first synchronization frame for subsequent synchronization, so that the synchronization is gradually advanced; finally, for each current sensing data frame obtained each time, determine whether the time interval between the trigger timestamps of each two adjacent sensing data frames in the sensing data frame sequence corresponding to each sensing device is within the preset range, and if so, determine to complete the sensing data synchronization of the corresponding time sequence of each current sensing data frame. In the process of autonomous driving, efficient and high-precision time synchronization of multi-sensing device data is achieved.

[0072] In actual applications, self-driving cars are equipped with a variety of sensing devices, including: laser radar, visual cameras, millimeter-wave radar, ultrasonic radar, etc. During the driving process of the unmanned vehicle, a variety of sensing devices will collect surrounding environmental information in real time and generate corresponding sensing data. Due to the unstable triggering of the sensing devices, frame skipping and frame loss may occur, and the coverage of each sensing device is different. It is necessary to summarize the sensing data collected by multiple sensing devices for subsequent analysis, so the sensing data needs to be synchronized in time. Taking the example of an unmanned vehicle equipped with 4 laser radars and 7 visual cameras (including fisheye, mid-focus and telephoto), each laser radar or visual camera is a sensing device. The installation and coverage of the laser radar are as follows: Figure 2a As shown, the visual camera installation and coverage Figure 2b As shown, during implementation, the main radar can be determined as the reference sensing device, and other sensing devices can also be determined as reference sensing devices according to actual needs, which is not limited here.

[0073] In another embodiment provided by the present disclosure, the above S101, determining the first synchronization frames corresponding to the multiple sensing devices respectively based on the time intervals between the trigger timestamps of the reference sensing data frame and the trigger timestamps of the sensing data frames of other sensing devices in the multiple sensing devices, can be implemented as the following steps:

[0074] Step 1: Acquire perception data frames collected by multiple perception devices;

[0075] Step 2: traverse the perception data frames collected by the other perception devices except the reference perception device among the multiple perception devices, and determine the first perception data frame whose time interval between the trigger timestamp and the trigger timestamp of the reference perception data frame is less than the first preset time threshold;

[0076] Step three: when each other perception device has determined the corresponding first perception data frame, the first perception data frame is determined as the first synchronization frame corresponding to the other perception device, and the reference perception data frame is determined as the first synchronization frame corresponding to the reference perception device.

[0077] In the disclosed embodiment, the sensing device has a corresponding operating frequency (taking a laser radar as an example, its operating frequency may be, for example, 10 Hz, that is, 10 scans are completed in 1 second), the trigger time of each scan may be used as a trigger timestamp, and the sensing data obtained in each scan may be a data frame. When the operating frequencies of multiple sensing devices differ slightly (they are the same, or the difference is within a preset range), or when the operating frequencies of multiple sensing devices differ slightly after pre-processing, as long as the first synchronization frame of multiple sensing devices is determined, and the first synchronization frame corresponding to each sensing device is used as a reference, subsequent data frames are collected according to the corresponding operating frequency, then it can be basically ensured that subsequent data frames of each sensing device can also be synchronized.

[0078] Therefore, in the embodiment of the present disclosure, the reference perception data frame of the reference perception device is first determined, and the perception data frames of other perception acquisition devices are traversed based on the reference perception data frame. When the time interval between the trigger timestamp of the perception data frame (i.e., the first perception data frame) of the other perception devices and the trigger timestamp of the reference perception data frame is less than a first preset time threshold, each first perception data frame is respectively determined as the first synchronization frame corresponding to the other perception device. Similarly, the reference perception data frame is determined as the first synchronization frame corresponding to the reference perception device.

[0079] In another embodiment provided by the present disclosure, the above step 1, obtaining the sensing data frames collected by multiple sensing devices, can be implemented as follows:

[0080] Acquiring the perception data frames collected by multiple perception devices and storing them in a perception data set can be implemented as follows:

[0081] If the acquired perception data frame is not a reference perception data frame collected by the reference perception device, storing the acquired perception data frame into the perception data set; or,

[0082] When the acquired perception data frame is a reference perception data frame collected by a reference perception device and there is no reference perception data frame in the current perception data set, the acquired reference perception data frame is stored in the perception data set, and the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set are updated.

[0083] In the disclosed embodiment, when the perception data frames corresponding to multiple perception devices are continuously acquired, the perception data frames can be stored in the perception data set, and the first synchronization frame corresponding to each perception device can be compared and screened in the perception data set. In order to reduce the complexity of subsequent traversal, it is necessary to reduce the redundancy of data in the perception data set, so the perception data set is initialized, and it is judged whether and / or how the perception data frames corresponding to the acquired multiple perception devices are stored in the perception data set. Therefore, according to the different types of perception data frames acquired (non-reference perception data frames or reference perception data frames), it is determined how to store the acquired perception data frames in the perception data set, including the following implementation methods:

[0084] The first implementation mode: the acquired perception data frame is a non-reference perception data frame. No matter whether the reference perception data frame already exists in the current perception data set, or whether the non-reference perception data frame corresponding to the same perception device as the acquired perception data frame already exists in the current perception data set, the acquired perception data frame can be directly stored in the perception data set.

[0085] The second implementation method: the acquired perception data frame is a reference perception data frame, and the reference perception data frame does not exist in the current perception data set, the acquired reference perception data frame is stored in the perception data set, and the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set are updated.

[0086] In another embodiment provided by the present disclosure, the above step 1, obtaining the sensing data frames collected by multiple sensing devices, can be implemented as follows:

[0087] Acquiring perception data frames collected by multiple perception devices and storing them in a perception data set includes the following implementation methods:

[0088] In the case where the acquired perception data frame is a reference perception data frame collected by a reference perception device, and there are historical reference perception data frames in the current perception data set, traverse all perception data frames in the perception data set, remove perception data frames whose time interval between a trigger timestamp and a trigger timestamp of a newly acquired reference perception data frame is greater than a second preset time threshold from the perception data set, store the newly acquired reference perception data frame in the perception data set, and update the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set;

[0089] In the above S101, determining the reference sensing data frame of the reference sensing device among the multiple sensing devices may be implemented as follows:

[0090] When it is determined that the perception data set stores the perception data frames collected by each perception device, the reference perception data frame of the reference perception device is determined from the perception data set according to the storage position of the most recently updated reference perception data frame in the perception data set.

[0091] In the embodiment of the present disclosure, following the above description, for different types of acquired perception data frames, determining how to store the perception data set also includes:

[0092] The third implementation method: the acquired perception data frame is a reference perception data frame, and there are historical reference perception data frames in the current perception data set. At this time, it is necessary to traverse all the perception data frames in the perception data set, remove the perception data frames whose time interval between the trigger timestamp and the trigger timestamp of the newly acquired reference perception data frame is greater than the second preset time threshold from the perception data set, complete the update of the reference perception data frame and remove other perception data frames corresponding to other perception devices with too early timestamps in the perception data set, so as to achieve the purpose of removing redundant data with earlier timestamps and reduce the complexity of traversal. Store the newly acquired reference perception data frame with updated timestamp in the perception data set, and update the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set.

[0093] In the embodiment of the present disclosure, before performing first frame synchronization on the perception data frame in the perception data set, it is also necessary to ensure that each perception device has corresponding perception data frames stored in the perception data set, and then determine the reference perception data frame of the reference perception device from the perception data set based on the storage position of the latest updated reference perception data frame in the perception data set.

[0094] In another embodiment provided by the present disclosure, the above method further includes the following steps, which may be subsequent to the above step three:

[0095] Step 4: after determining the first synchronization frames corresponding to the multiple sensing devices respectively, remove the reference sensing data frame corresponding to the reference sensing device and the corresponding first sensing data frames determined for other sensing devices from the sensing data set.

[0096] In the disclosed embodiment, the perception data set is used to compare and screen the first synchronization frame corresponding to each perception device. After the first frame synchronization is completed and the first synchronization frame corresponding to each perception device is determined, the perception data frame in the perception data set can be removed. During implementation, the first synchronization frame corresponding to each perception device can also be moved into a synchronization data set for subsequent data frame synchronization comparison, and the data frames collected by each perception device after the first synchronization frame continue to be received in the synchronization data set, which is not limited here.

[0097] In another embodiment provided by the present disclosure, the above method further includes:

[0098] When any other perception device fails to determine the corresponding first perception data frame, determine the reference perception data frame that is most recently updated after the currently used reference perception data frame; and continue to execute the above-mentioned step 2, traverse the perception data frames currently collected by other perception devices, and determine the first perception data frame whose time interval between the trigger timestamp and the trigger timestamp of the reference perception data frame that is most recently updated after the currently used reference perception data frame is less than the first preset time threshold.

[0099] In an embodiment of the present disclosure, in the above-mentioned step two, during the execution of traversing the perception data frames collected by other perception devices among multiple perception devices except the reference perception device to determine the first perception data frame whose time interval between a trigger timestamp and a trigger timestamp of the reference perception data frame is less than a first preset time threshold, if any other perception device fails to determine the corresponding first perception data frame, it is necessary to determine the reference perception data frame that is most recently updated after the currently used reference perception data frame and repeat the traversal operation of the above-mentioned step two until step three is met and each other perception device determines the corresponding first perception data frame.

[0100] In another embodiment provided by the present disclosure, the above method further includes:

[0101] In the event that any other perception device determines at least two corresponding first perception data frames, the first perception data frame with the smallest time interval between the trigger timestamp of the reference perception data frame and the at least two corresponding first perception data frames is determined as the first perception data frame corresponding to the other perception device.

[0102] In the embodiment of the present disclosure, in the above step 2, traversing the perception data frames collected by other perception devices except the reference perception device among the multiple perception devices, and determining the first perception data frame whose time interval between the trigger timestamp and the trigger timestamp of the reference perception data frame is less than the first preset time threshold, it is possible that any other perception device determines at least two corresponding first perception data frames, such as Figure 3As shown, the reference perception device and the perception device A respectively collect perception data frames at the time sequence of 0s to 0.3s. For example, due to frame skipping and other phenomena, the perception device A collects perception data frame a (0.08s) and perception data frame b (0.13s) before and after 0.1s, so that the perception device A has two first perception data frames (a and b), which appear before and after the reference perception data frame c, respectively, and the time interval between the trigger timestamps of the two first perception data frames and the trigger timestamp of the reference perception data frame is less than the first preset time threshold. At this time, the timestamps of the at least two corresponding first perception data frames can be compared with the timestamp of the reference perception data frame, and the first perception data frame (a) with the smallest time interval between the trigger timestamp of the reference perception data frame is determined as the first perception data frame corresponding to the other perception device. The perception data frame a with the closest timestamp to the reference perception data frame is retained, and the perception data frame b with the farther timestamp is removed.

[0103] like Figure 4 As shown, for the above S101, determining a reference perception data frame of a reference perception device among multiple perception devices, and determining the first synchronization frames corresponding to the multiple perception devices respectively based on the time intervals between the trigger timestamps of the reference perception data frame and the trigger timestamps of the perception data frames of other perception devices among the multiple perception devices, a possible implementation method includes the following steps:

[0104] S401, acquiring perception data frames collected by multiple perception devices;

[0105] S402, determining whether the acquired perception data frame is a reference perception data frame, if so, proceeding to step S403, if not, proceeding to step S404;

[0106] S403: Whether there is a reference perception data frame in the current perception data set, if yes, proceed to step S405, if not, proceed to step S406;

[0107] S404, storing the acquired perception data frame into the perception data set, and proceeding to step S407;

[0108] S405, traversing all the perception data frames in the perception data set, removing the perception data frames whose time interval between the trigger timestamp and the trigger timestamp of the newly acquired reference perception data frame is greater than the second preset time threshold from the perception data set, storing the newly acquired reference perception data frame into the perception data set, and updating the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set; and proceeding to step S407;

[0109] S406, storing the acquired reference perception data frame into the perception data set, and updating the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set;

[0110] S407, determining whether each sensing device has a sensing data frame in the sensing data set, if so, proceeding to step S408, if not, proceeding to step S401;

[0111] S408, traversing the perception data frames collected by the other perception devices except the reference perception device among the multiple perception devices, determining whether the time interval between the trigger timestamp and the trigger timestamp of the reference perception data frame is less than the first preset time threshold of the first perception data frame, if so, proceeding to step S409, if not, proceeding to step S401;

[0112] S409, determining whether any other sensing device has determined at least two corresponding first sensing data frames, if so, proceeding to step S410, if not, proceeding to step S411;

[0113] S410: Determine, among at least two corresponding first perception data frames, a first perception data frame having a minimum time interval between a triggering timestamp of a reference perception data frame and the frame as the first perception data frame corresponding to the other perception device;

[0114] S411, determining the first perception data frame as the first synchronization frame corresponding to the other perception devices, and determining the reference perception data frame as the first synchronization frame corresponding to the reference perception device;

[0115] S412: After determining the first synchronization frames corresponding to the multiple perception devices respectively, remove the reference perception data frame corresponding to the reference perception device and the corresponding first perception data frames determined for other perception devices from the perception data set.

[0116] In another embodiment provided by the present disclosure, the above S103, for the current perception data frames corresponding to the multiple perception devices respectively obtained each time, when each current perception data frame conforms to the preset timing rule, determines that the perception data synchronization corresponding to the timing of each current perception data frame is completed, which can be implemented as follows:

[0117] Step 1: for each current perception data frame corresponding to the multiple perception devices acquired each time, determine whether the difference between the first trigger timestamp of the current perception data frame and the second trigger timestamp of the previous perception data frame of the current perception data frame of the corresponding perception device is less than a third preset time threshold;

[0118] Step 2: When the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time threshold, it is determined that the perception data synchronization of the corresponding timing of each current perception data frame is completed.

[0119] In the disclosed embodiment, during the execution of the above step S103, for each current perception data frame acquired each time, the trigger timestamp of the current perception data frame and the trigger timestamp of the previous perception data frame of the same perception device are determined, wherein the trigger timestamp of the current perception data frame is the first trigger timestamp, and the trigger timestamp of the previous perception data frame of the same perception device is the second trigger timestamp. It is determined whether the difference between the two timestamps is less than a third preset threshold, wherein the third preset threshold is set according to the trigger frequency of the corresponding perception device, and if the difference is less than the third preset threshold, it means that there is no frame loss in the current perception data frame, and if the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time threshold, that is, there is no frame loss, then it can be preliminarily determined that the perception data synchronization of the corresponding timing of each current perception data frame is completed.

[0120] In a possible implementation, the current sensing data frame acquired by each sensing device each time may be stored in the following structure:

[0121] {

[0122] topic:{

[0123] "msg":message,

[0124] "timestamp":timestamp,

[0125] "timestamp_last":timestamp_last,

[0126] "lost_flag":False

[0127] }

[0128] }

[0129] Among them, topic represents the name of the perception device, msg represents the perception data, timestamp represents the trigger timestamp of the current perception data frame (ie, the first trigger timestamp), and timestamp_last represents the trigger timestamp of the previous perception data frame (ie, the second trigger timestamp).

[0130] The value of the above-mentioned lost_flag is determined by the difference between timestamp and timestamp_last. If the difference is less than the third preset threshold, lost_flag is set to False. If the difference is greater than or equal to the third preset threshold, there may be frame loss between the current perception data frame and the previous perception data frame, and lost_flag is set to True. When the current perception data frame is the first synchronization frame, lost_flag is initialized to False. In addition, when obtaining a new current perception data frame, only msg and timestamp in the structure can be updated, timestamp_last can be the value of timestamp corresponding to the previous perception data frame, and lost_flag can be determined based on the above-mentioned difference, which will not be repeated here.

[0131] In another embodiment provided by the present disclosure, the above method can also be implemented as the following steps:

[0132] Step 1: When the difference between the first trigger timestamp and the corresponding second trigger timestamp of the current perception data frame of at least one first perception device is greater than or equal to the third preset time threshold, it is determined that the perception data synchronization of the time sequence corresponding to the current perception data frame fails;

[0133] Step 2: taking the current perception data frame corresponding to the at least one first perception device as the latest acquired new current perception data frame, and determining a corresponding second trigger timestamp for the new current perception data frame;

[0134] Step 3: determine a second sensing device among multiple sensing devices for which the difference between the first trigger timestamp of the current sensing data frame and the corresponding second trigger timestamp is less than a third preset time threshold;

[0135] Step 4: Obtain a new current perception data frame for the second perception device, and continue to execute the above step 1: for each new current perception data frame corresponding to the newly acquired first perception device and the second perception device, determine whether the difference between the first trigger timestamp of the new current perception data frame and the corresponding second trigger timestamp is less than the third preset time threshold.

[0136] In the embodiment of the present disclosure, during the execution of the above-mentioned step S103, if there is a situation where the difference between the first trigger timestamp and the corresponding second trigger timestamp of the current perception data frame of at least one first perception device is greater than or equal to the third preset time threshold, then there may be frame loss between the current perception data frame and the previous perception data frame, and it can be determined that the perception data synchronization of the timing corresponding to the current perception data frame has failed.

[0137] In a possible implementation, regardless of whether frame loss occurs, each perception device stores the collected data frames in sequence, and the disclosed embodiment obtains the data frames stored by each perception device in sequence according to the storage order. Each time the current perception data frame of each perception device is obtained, for a perception device that works normally, the time interval between its stored current perception data frame and the stored previous perception data frame meets its operating frequency (i.e., the difference is less than the third preset time threshold), and for a perception device that has frame loss, one or more previous perception data frames of its stored current perception data frame may have experienced frame loss, so the time interval between its stored previous perception data frame and the stored current perception data frame does not meet its operating frequency (i.e., the difference is greater than or equal to the third preset time threshold). Therefore, by comparing the time interval between the current perception data frame and the previous perception data frame with the third preset time threshold, it can be known whether there may be frame loss before the current perception data frame. For a first perception device that may have frame loss, although there is a second perception device that has not experienced frame loss in the same synchronization, due to the frame loss of the first perception device, the first perception device does not have a perception data frame that can be synchronized with the current perception data frame of the second perception device. Therefore, it is determined that the synchronization of the current perception data frame has failed. The current perception data frame of the second perception device can be discarded, while the current perception data frame of the first perception device can be retained, because the subsequent perception data frame stored in the second perception device may be successfully synchronized with the current data frame of the first perception device due to a later trigger timestamp.

[0138] Therefore, in the embodiment of the present disclosure, first, the current perception data frame corresponding to at least one first perception device, for which the difference between the first trigger timestamp of the above-mentioned current perception data frame and the corresponding second trigger timestamp is greater than or equal to the third preset time threshold, is taken as the latest acquired new current perception data frame, and the second trigger timestamp of the latest acquired new current perception data frame is updated. Then, a second perception device is determined for which the difference between the first trigger timestamp of the current perception data frame and the corresponding second trigger timestamp is less than the third preset time threshold among multiple perception devices. Finally, a new current perception data frame is acquired for the second perception device and the above step one is performed, that is, for each new current perception data frame corresponding to the multiple perception devices acquired this time, it is determined whether the difference between the first trigger timestamp of the new current perception data frame and the second trigger timestamp of the previous perception data frame of the new current perception data frame of the corresponding perception device is less than the third preset time threshold.

[0139] like Figure 5As shown, taking the case where the perception device is triggered once every 0.1 seconds and the third preset time threshold is 0.1s, the first data synchronization corresponds to a timing of 0.1s, and the difference between the first trigger timestamp (0.1s) of the current perception data frame of perception devices 1, 2, 3, and 4 and the second trigger timestamp (0s) corresponding to the previous perception data frame is less than or equal to the third preset time threshold (0.1s). It can be preliminarily determined that the perception data synchronization corresponding to the timing of each current perception data frame is completed.

[0140] In the second perception data synchronization, the next perception data frame is obtained in sequence. The timing corresponding to this data synchronization is 0.2s. The current perception data frame obtained this time is labeled a. For perception device 1, the first trigger timestamp of the current perception data frame a is 0.2s, and the second trigger timestamp is 0.1s; for perception device 2, the first trigger timestamp of the current perception data frame a is 0.4s, and the second trigger timestamp is 0.1s; for perception device 3, the first trigger timestamp of the current perception data frame a is 0.3s, and the second trigger timestamp is 0.1s; for perception device 4, the first trigger timestamp of the current perception data frame a is 0.2s, and the second trigger timestamp is 0.1s. It can be seen that perception device 2 and perception device 3 are the first perception devices with frame loss, and perception device 1 and perception device 4 are the second perception devices without frame loss. For perception device 2 and perception device 3, the perception data frames synchronized with the current perception data frame a corresponding to perception device 1 and perception device 4 respectively have been lost, and this data synchronization fails. In order to maximize the use of the perception data frames, the current perception data frames a corresponding to the perception device 1 and the perception device 4 may be discarded.

[0141] In the third perception data synchronization, the current perception data frames of the two first perception devices determined last time, perception device 2 and perception device 3 in the second synchronization (the perception data frame of 0.4s corresponding to perception device 2 and the data frame of 0.3s corresponding to perception device 3, which are marked as b in this synchronization) are used as the new current perception data frames of the third perception data synchronization, and the current data frames of the second perception devices without frame loss, namely perception device 1 and perception device 4, are obtained, that is, the data frames of the two with a mark of b at 0.3s. In the third perception data synchronization, data synchronization is performed on the four current data frames marked as b. It can be seen that perception device 2 is the first perception device with frame loss in this data synchronization, and perception devices 1, 3, and 4 are the second perception devices without frame loss. There is a first perception device with frame loss in this data synchronization, and it can be determined that the perception data synchronization corresponding to the timing of the current perception data frame has failed. Similarly, the current perception data frames b corresponding to perception devices 1, 2, and 4, respectively, can be discarded.

[0142] In the fourth perception data synchronization, the perception data frame of 0.4s corresponding to the first perception device determined last time, that is, perception device 2, continues to be used as the new current perception data frame of the fourth perception data synchronization (marked as c in this synchronization), and the current data frames of the second perception devices without frame loss, that is, perception device 1, perception device 2 and perception device 4, are obtained, that is, the data frames of the two marked as c at 0.4s. In the fourth perception data synchronization, data synchronization is performed on the four current data frames marked as c.

[0143] During implementation, the discarding operation of the perception data frame corresponding to the second perception device can be achieved by clearing the content of the topic structure corresponding to the perception data frame.

[0144] In another embodiment provided by the present disclosure, before obtaining a new current perception data frame for the second perception device, the method further includes the following steps:

[0145] Step 1: determining the minimum trigger timestamp among the first trigger timestamps corresponding to the current perception data frames of the at least one first perception device;

[0146] Step 2: Compare the first trigger timestamp of the current perception data frame corresponding to the second perception device with the minimum trigger timestamp;

[0147] The above step 4, obtaining a new current perception data frame for the second perception device, can be implemented as follows:

[0148] If there is a second perception device that satisfies that the difference between the first trigger timestamp and the minimum trigger timestamp of the current perception data frame corresponding to the second perception device is greater than or equal to the fourth preset time threshold, obtain a new current perception data frame for the second perception device;

[0149] The above step 4, obtaining a new current perception data frame for the second perception device, may also be implemented as follows:

[0150] When there is a second perception device that satisfies the fact that the difference between the first trigger timestamp and the minimum trigger timestamp of the current perception data frame corresponding to the second perception device is less than a fourth preset time threshold, the current perception data frame corresponding to the second perception device is used as the new current perception data frame most recently obtained by the corresponding second perception device.

[0151] In the above embodiment, due to the frame loss of the first perception device, the current perception data frame of the second perception device cannot be synchronized successfully, so a discard operation is performed; however, if the time interval between the current data frame of the second perception device and the earliest current perception data frame in the current perception data frame of the first perception device is very small (less than the fourth preset time threshold), the current perception data frame of the second perception device may have frame skipping, and the current perception data frame of the second perception device may be retained until the next perception data synchronization, and the synchronization comparison may continue.

[0152] Therefore, in an embodiment of the present disclosure, before obtaining a new current perception data frame for the second perception device, the minimum trigger timestamp among the first trigger timestamps corresponding to the current perception data frames of the at least one first perception device is first determined, and by comparing the first trigger timestamp of the current perception data frame corresponding to the second perception device with the minimum trigger timestamp, it is determined whether a new current perception data frame needs to be obtained for the corresponding second perception device.

[0153] In the embodiment of the present disclosure, when a new current perception data frame is obtained for the second perception device, how to obtain the new current perception data frame is determined based on the first trigger timestamp of the current perception data frame corresponding to the second perception device and the above-mentioned minimum trigger timestamp, including the following situations:

[0154] Case 1: if there is a second perception device that satisfies the condition that the difference between the first trigger timestamp and the minimum trigger timestamp of the current perception data frame corresponding to the second perception device is greater than or equal to the fourth preset time threshold, a new current perception data frame is obtained for the second perception device;

[0155] The second situation: if there is a second perception device that satisfies the situation that the difference between the first trigger timestamp and the minimum trigger timestamp of the current perception data frame corresponding to the second perception device is less than the fourth preset time threshold, the current perception data frame corresponding to the second perception device is used as the new current perception data frame most recently obtained by the corresponding second perception device.

[0156] like Figure 6 As shown, taking the perception device as triggered once every 0.1 seconds, the third preset time threshold is 0.15s, and the fourth preset time threshold is 0.09s as an example, the first perception data synchronization (corresponding to data synchronization with a timing of 0.1s) is successful. Please refer to the above for the reasons and will not repeat them here.

[0157] Sequentially obtain the next perception data frame and perform data synchronization for the next time sequence. The time sequence corresponding to the second data synchronization is 0.2s. The current perception data frame participating in this data synchronization is numbered a. For perception device 1, the first trigger timestamp of the current perception data frame a is 0.2s, and the second trigger timestamp is 0.1s; for perception device 2, the first trigger timestamp of the current perception data frame a is 0.22s, and the second trigger timestamp is 0.1s; for perception device 3, the first trigger timestamp of the current perception data frame a is 0.3s, and the second trigger timestamp is 0.1s; for perception device 4, the first trigger timestamp of the current perception data frame a is 0.2s, and the second trigger timestamp is 0.1s. Therefore, it is determined that perception device 3 is the first perception device with frame loss in this data synchronization, and perception device 1, perception device 2 and perception device 4 are the second perception devices without frame loss. Therefore, there is a first perception device with frame loss in this data synchronization, and it can be determined that the perception data synchronization of the time sequence corresponding to the current perception data frame has failed. Perform the following steps:

[0158] The first trigger timestamp (0.3s) of the sensing device 3 is used as the minimum trigger timestamp, and the first trigger timestamps (0.2s, 0.22s and 0.2s) of the sensing devices 1, 2 and 4 are respectively compared with the first trigger timestamp (0.3s). It can be determined that the difference (0.08s) between the first trigger timestamp corresponding to the sensing device 2 belonging to the second sensing device and the minimum trigger timestamp is less than the fourth preset time threshold (0.09s). It can be seen that the current sensing data frame a corresponding to the sensing device 2 may be a skipped frame. Then the current sensing data frames of the sensing devices 1 and 4 are discarded, and the current sensing data frames of the sensing devices 2 and 3 are retained until the next data synchronization.

[0159] In the third data synchronization, the corresponding timing is 0.3s, and the current perception data frame a corresponding to the last synchronization of perception device 2 and perception device 3 is used as the latest new current perception data frame acquired in this data synchronization (identified as b in this synchronization), and the current data frame is acquired for perception device 1 and perception device 4, that is, the perception data frame identified as b at 0.3s. Continue to compare whether it is synchronized, which will not be repeated here.

[0160] In another embodiment provided by the present disclosure, in the above step 2, the current perception data frame corresponding to the at least one first perception device is used as the new current perception data frame to be acquired next time, and the corresponding second trigger timestamp is determined for the new current perception data frame, which can be implemented as follows:

[0161] The current perception data frame corresponding to the at least one first perception device is used as the new current perception data frame to be acquired next time, and the first trigger timestamp of the current perception data frame corresponding to the second perception device is determined as the second trigger timestamp corresponding to the new current perception data frame of the at least one first perception device.

[0162] In the embodiment of the present disclosure, when there is a first perception device, when the current data frame corresponding to the first perception device determined in the previous perception synchronization is retained to the current perception synchronization, since the first perception device has experienced frame loss, as the new current data frame in the current perception synchronization, it is impossible to directly obtain the corresponding second trigger timestamp for it. Therefore, the second trigger timestamp of the new current data frame in the current perception synchronization can be determined with the help of the first trigger timestamp of the current data frame in the previous perception synchronization of the second perception device that did not experience frame loss in the previous perception synchronization.

[0163] Continue to see Figure 5 In the third perception data synchronization (corresponding timing is 0.3s), the current perception data frames of perception device 2 and perception device 3 in the second perception data synchronization (perception data frame a of 0.4s corresponding to perception device 2, and data frame a of 0.3s corresponding to perception device 3) are used as the new current perception data frame b of the third perception data synchronization. When determining the second trigger timestamp corresponding to the new current perception data frame for perception device 2 and perception device 3, the first trigger timestamp (0.2s) corresponding to the current perception data frame a of the second perception device determined in the second perception data synchronization (i.e., perception device 1 and perception device 4) in the second perception data synchronization can be determined as the second trigger timestamp corresponding to the new current perception data frame of perception device 2 and perception device 3 in the third perception data synchronization.

[0164] In another embodiment provided by the present disclosure, when the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time interval threshold, before determining that the perception data frame synchronization of the corresponding timing of each current perception data frame is completed, the above method can also be implemented as the following steps:

[0165] Step 1: respectively determine the difference between the first trigger timestamp of the first current perception data frame other than the current reference perception data frame corresponding to the reference perception device in each current perception data frame and the first trigger timestamp of the current reference perception data frame;

[0166] Step 2: Determine and complete the synchronization of the sensing data frames corresponding to the timing of each current sensing data frame, which can be implemented as follows:

[0167] When the difference between the first trigger timestamp of the first current perception data frame and the first trigger timestamp of the current reference perception data frame is less than the fifth preset time threshold, it is determined that the perception data frame synchronization of the corresponding timing of each current perception data frame is completed; or,

[0168] The above method can also be implemented as follows:

[0169] When there is a difference between at least one first trigger timestamp of the first current perception data frame and the first trigger timestamp of the current reference perception data frame that is greater than or equal to a fifth preset time threshold, it is determined that the perception data frame synchronization of the timing corresponding to the current perception data frame fails.

[0170] In the disclosed embodiment, during a synchronization process, although the difference between the first trigger timestamp of each current perception data frame and the corresponding second trigger timestamp is less than the third preset time interval threshold, it is also possible that the first trigger timestamp between each current perception data frame that completes this synchronization, or the first trigger timestamp of each current perception data frame is too far away from the timing of this synchronization. Therefore, it is necessary to further judge the first current data frame other than the reference perception data frame in each current perception data frame, and again take the reference perception data frame as the benchmark, by determining whether the difference between the first trigger timestamp of each first current perception data frame and the first trigger timestamp of the current reference data frame is less than the fifth preset threshold, determine whether the timing difference between each current perception data frame is too large; if they are all less than the fifth preset threshold, it can be determined that this synchronization is successful; if there is at least one greater than or equal to the fifth preset threshold, it can be determined that this synchronization has failed. Among them, the fifth preset threshold can be obtained through implementation experience.

[0171] In another embodiment provided by the present disclosure, when the current perception data frames respectively corresponding to the multiple perception devices are the first synchronization frames, the method further includes:

[0172] According to the first trigger timestamps corresponding to the first synchronization frames and the acquisition frequency of the sensing device to which the first synchronization frames belong, the second trigger timestamps corresponding to the first synchronization frames are determined.

[0173] In the disclosed embodiment, the current perception data frames respectively corresponding to the multiple perception devices participating in the synchronization may not only be the perception data frames after the first synchronization frame, but also the first synchronization frame. In a synchronization process, when the current perception data frame is the first synchronization frame, since there is no previous perception data frame for each first synchronization frame, the second trigger timestamp of each first synchronization frame cannot be determined by the previous perception data frame of the current perception data frame that is not the first synchronization frame to determine the second trigger timestamp of the current perception data frame. Therefore, the second trigger timestamp respectively corresponding to each first synchronization frame can be calculated by the trigger frequency of the perception device respectively corresponding to itself and the first trigger timestamp of the first synchronization frame to obtain its theoretical value. For example, for each first synchronization frame, taking the trigger frequency of the perception device as 10HZ as an example, the perception device completes 10 scans in 1 second, and the trigger time interval of the perception device is 0.1s. The first trigger timestamp of the first synchronization frame is subtracted from the trigger time interval of the perception device corresponding to the first synchronization frame to obtain the second trigger timestamp of the first synchronization frame. Taking the first trigger timestamp of 0.1s as an example, the second trigger timestamp is 0s.

[0174] In another embodiment provided by the present disclosure, because the method provided by the present disclosure involves input determination of perception data corresponding to a plurality of perception devices, there will not be too much data in the perception data set for traversal synchronization. For example, in the above S101, the data set size of the first frame synchronization can be set and will not exceed 20 data items; and in the above S103, the size of the data set during time synchronization is related to the number of perception devices and will not exceed the number of perception devices, thereby ensuring high efficiency.

[0175] In another embodiment provided by the present disclosure, when a new perception device needs to be added to the synchronization, the relevant information and the perception device name are added to the corresponding queue in the form of the above-mentioned structure according to the data characteristics of the perception device and stored in the perception data set, which has very convenient scalability.

[0176] like Figure 7 As shown, for the above S103, for the current perception data frames corresponding to the multiple perception devices respectively obtained each time, when each current perception data frame meets the preset timing rule, it is determined that the perception data synchronization of the timing corresponding to each current perception data frame is completed. In a possible implementation, the following steps may be included:

[0177] S701, for each current perception data frame corresponding to the multiple perception devices obtained each time, determine whether the difference between the first trigger timestamp of the current perception data frame and the second trigger timestamp of the previous perception data frame of the current perception data frame of the corresponding perception device is less than the third preset time threshold; if so, proceed to step S708, if not, proceed to step S702;

[0178] S702, determining that the synchronization of the perception data of the time sequence corresponding to the current perception data frame fails;

[0179] S703, determining the minimum trigger timestamp among the first trigger timestamps respectively corresponding to the current perception data frames of the first perception device; the first perception device includes a perception device in which the difference between the first trigger timestamp of the current perception data frame and the second trigger timestamp of the previous perception data frame of the current perception data frame of the corresponding perception device is greater than or equal to the third preset time threshold;

[0180] S704, determine whether the difference between the first trigger timestamp and the minimum trigger timestamp of the current perception data frame corresponding to the second perception device is less than the fourth preset threshold value, if so, proceed to step S705, if not, proceed to step S706; the second perception device includes a perception device in which the difference between the first trigger timestamp of the current perception data frame and the second trigger timestamp of the previous perception data frame of the current perception data frame of the corresponding perception device is less than the third preset time threshold value;

[0181] S705: The current perception data frame corresponding to the second perception device is used as a new current perception data frame newly acquired by the corresponding second perception device, and the process proceeds to step S707;

[0182] S706, obtaining a new current perception data frame for the second perception device;

[0183] S707, taking the current perception data frame corresponding to the first perception device as the latest acquired new current perception data frame, and determining the first trigger timestamp of the current perception data frame corresponding to the second perception device as the second trigger timestamp corresponding to the new current perception data frame of the first perception device, performing the next synchronization, and entering step S701;

[0184] S708, respectively determine whether the difference between the first current perception data frame and the current reference perception data frame other than the current reference perception data frame corresponding to the reference perception device in each current perception data frame is less than a fifth preset threshold; if so, execute step S709; if not, execute step S710;

[0185] S709, complete the perception data frame synchronization of the corresponding timing of each current perception data frame; this process ends.

[0186] S710, determining that the synchronization of the perception data frame corresponding to the timing of the current perception data frame fails, performing the next synchronization, and entering step S701.

[0187] Based on the same public concept, the embodiments of the present disclosure also provide a time synchronization device for perception data and an unmanned vehicle. Since the principles of the problems solved by these devices and the unmanned vehicle are similar to those of the aforementioned time synchronization method for perception data, the implementation of the device and the unmanned vehicle can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0188] With the above Figure 1 Corresponding to the method shown, the embodiment of the present disclosure also provides a time synchronization device for sensing data, such as Figure 8 As shown, including:

[0189] The first frame synchronization module 801 is used to determine a reference perception data frame of a reference perception device among multiple perception devices, and determine the first synchronization frames corresponding to the multiple perception devices respectively based on the time interval between the trigger timestamp of the reference perception data frame and the trigger timestamps of the perception data frames of other perception devices among the multiple perception devices;

[0190] The data acquisition module 802 is used to sequentially acquire the sensing data frames collected by multiple sensing devices after the corresponding first synchronization frame;

[0191] The time synchronization module 803 is used to determine the completion of the perception data synchronization corresponding to the timing of each current perception data frame corresponding to each of the multiple perception devices acquired each time, when each current perception data frame complies with the preset timing rule; wherein the above-mentioned preset timing rule indicates that the time interval between the trigger timestamps of each two adjacent perception data frames in the perception data frame sequence corresponding to each perception device is within a preset range.

[0192] In another embodiment provided by the present disclosure, the first frame synchronization module 801 is used to:

[0193] Acquire perception data frames collected by multiple perception devices;

[0194] Traversing the perception data frames collected by the other perception devices except the reference perception device among the multiple perception devices, and determining a first perception data frame whose time interval between a trigger timestamp and a trigger timestamp of the reference perception data frame is less than a first preset time threshold;

[0195] When each other perception device determines the corresponding first perception data frame, the first perception data frame is determined as the first synchronization frame corresponding to the other perception device, and the reference perception data frame is determined as the first synchronization frame corresponding to the reference perception device.

[0196] In another embodiment provided by the present disclosure, the first frame synchronization module 801 is used to:

[0197] Acquire the perception data frames collected by multiple perception devices and store them in the perception data set, where:

[0198] If the acquired perception data frame is not a reference perception data frame collected by the reference perception device, storing the acquired perception data frame into the perception data set; or,

[0199] When the acquired perception data frame is a reference perception data frame collected by a reference perception device and there is no reference perception data frame in the current perception data set, the acquired reference perception data frame is stored in the perception data set, and the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set are updated.

[0200] In another embodiment provided by the present disclosure, following the first frame synchronization module 801, it is used to:

[0201] Acquire the perception data frames collected by multiple perception devices and store them in the perception data set, where:

[0202] When the acquired perception data frame is a reference perception data frame collected by a reference perception device and there are historical reference perception data frames in the current perception data set, all perception data frames in the current perception data set are traversed, and perception data frames whose time intervals between a trigger timestamp and a trigger timestamp of a newly acquired reference perception data frame are greater than a second preset time threshold are removed from the perception data set, the newly acquired reference perception data frame is stored in the perception data set, and the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set are updated.

[0203] The first frame synchronization module 801 is used for:

[0204] When it is determined that the perception data set stores the perception data frames collected by each perception device, the reference perception data frame of the reference perception device is determined from the perception data set according to the storage position of the most recently updated reference perception data frame in the perception data set.

[0205] In another embodiment provided by the present disclosure, the first frame synchronization module 801 is further used to:

[0206] After determining the first synchronization frames corresponding to the multiple sensing devices respectively, the reference sensing data frame corresponding to the reference sensing device and the corresponding first sensing data frames determined for other sensing devices are removed from the sensing data set.

[0207] In another embodiment provided by the present disclosure, the first frame synchronization module 801 is further used to:

[0208] When any other perception device fails to determine the corresponding first perception data frame, determine the reference perception data frame that is most recently updated after the currently used reference perception data frame; and traverse the perception data frames currently collected by other perception devices again to determine the first perception data frame whose time interval between a trigger timestamp and a trigger timestamp of a reference perception data frame that is most recently updated after the currently used reference perception data frame is less than a first preset time threshold.

[0209] In another embodiment provided by the present disclosure, the first frame synchronization module 801 is also used to: in the event that there are any other perception devices that determine at least two corresponding first perception data frames, determine the first perception data frame among the at least two corresponding first perception data frames whose time interval with the trigger timestamp of the reference perception data frame is the smallest as the first perception data frame corresponding to the other perception device.

[0210] In another embodiment provided by the present disclosure, the time synchronization module 803 is used to:

[0211] For each current perception data frame corresponding to the multiple perception devices respectively acquired each time, determine whether a difference between a first trigger timestamp of the current perception data frame and a second trigger timestamp of a previous perception data frame of the current perception data frame of the corresponding perception device is less than a third preset time threshold;

[0212] When the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time threshold, it is determined that the perception data synchronization of the corresponding timing of each current perception data frame is completed.

[0213] In another embodiment provided by the present disclosure, the time synchronization module 803 is further used to:

[0214] In the case where the difference between the first trigger timestamp and the corresponding second trigger timestamp of the current perception data frame of at least one first perception device is greater than or equal to the third preset time threshold, determining that the perception data synchronization of the time sequence corresponding to the current perception data frame fails;

[0215] Using the current perception data frame corresponding to the at least one first perception device as the latest acquired new current perception data frame, and determining a corresponding second trigger timestamp for the new current perception data frame;

[0216] Determine a second sensing device among the multiple sensing devices, wherein the difference between the first trigger timestamp of the current sensing data frame and the corresponding second trigger timestamp is less than a third preset time threshold;

[0217] Obtain a new current perception data frame for the above-mentioned second perception device, and again for each new current perception data frame corresponding to the most recently acquired first perception device and the second perception device, determine whether the difference between the first trigger timestamp of the new current perception data frame and the corresponding second trigger timestamp is less than a third preset time threshold.

[0218] In another embodiment provided by the present disclosure, before acquiring a new current perception data frame for the second perception device, the time synchronization module 803 is further used to:

[0219] Determine a minimum trigger timestamp among the first trigger timestamps respectively corresponding to the current perception data frames of the at least one first perception device;

[0220] Compare the first trigger timestamp of the current perception data frame corresponding to the second perception device with the minimum trigger timestamp;

[0221] If there is a second perception device that satisfies that the difference between the first trigger timestamp and the minimum trigger timestamp of the current perception data frame corresponding to the second perception device is greater than or equal to the fourth preset time threshold, obtain a new current perception data frame for the second perception device;

[0222] The time synchronization module 803 is further used for:

[0223] When there is a second perception device, and the difference between the first trigger timestamp and the minimum trigger timestamp of the current perception data frame corresponding to the second perception device is less than a fourth preset time threshold, the current perception data frame corresponding to the second perception device is used as the new current perception data frame most recently obtained by the corresponding second perception device.

[0224] In another embodiment provided by the present disclosure, the time synchronization module 803 is used to:

[0225] The current perception data frame corresponding to the at least one first perception device is used as the new current perception data frame to be acquired next time, and the first trigger timestamp of the current perception data frame corresponding to the second perception device is determined as the second trigger timestamp corresponding to the new current perception data frame of the at least one first perception device.

[0226] In another embodiment provided by the present disclosure, when the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time interval threshold, before determining that the perception data frame synchronization of the corresponding timing of each current perception data frame is completed, the time synchronization module 803 is further used to:

[0227] respectively determining a difference between a first current perception data frame other than a current reference perception data frame corresponding to a reference perception device and the current reference perception data frame in each current perception data frame;

[0228] When the difference between the first current perception data frame and the current reference perception data frame is less than the fifth preset time threshold, it is determined that the perception data frame synchronization of the corresponding timing of each current perception data frame is completed; or,

[0229] When there is a difference between at least one first current perception data frame and the current reference perception data frame that is greater than or equal to a fifth preset time threshold, it is determined that the perception data frame synchronization of the timing corresponding to the current perception data frame fails.

[0230] In another embodiment provided by the present disclosure, when the current perception data frames respectively corresponding to the multiple perception devices are the first synchronization frames, the method further includes:

[0231] According to the first trigger timestamps corresponding to the first synchronization frames and the acquisition frequency of the sensing device to which the first synchronization frames belong, the second trigger timestamps corresponding to the first synchronization frames are determined.

[0232] With the above Figure 1 Corresponding to the method shown, an embodiment of the present disclosure also provides an unmanned vehicle, including a time synchronization device for perception data provided by any of the above embodiments.

[0233] Through the description of the above implementation methods, those skilled in the art can clearly understand that the embodiments of the present disclosure can be implemented by hardware, or by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present disclosure.

[0234] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the accompanying drawings are not necessarily required for implementing the present disclosure.

[0235] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the present embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0236] The serial numbers of the above-mentioned embodiments of the present disclosure are only for description and do not represent the advantages or disadvantages of the embodiments.

[0237] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for time synchronization of perception data, characterized in that: include: Determine a reference perception data frame of a reference perception device among the multiple perception devices, and determine first synchronization frames corresponding to the multiple perception devices respectively based on time intervals between trigger timestamps of the reference perception data frame and trigger timestamps of perception data frames of other perception devices among the multiple perception devices; Sequentially acquiring the perception data frames collected by the plurality of perception devices after the corresponding first synchronization frame respectively; For the current perception data frames corresponding to the multiple perception devices obtained each time, when each current perception data frame complies with the preset timing rules, it is determined that the perception data synchronization corresponding to the timing of each current perception data frame is completed; wherein the preset timing rules represent that the time interval between the trigger timestamps of each two adjacent perception data frames in the perception data frame sequence corresponding to each perception device is within a preset range.

2. The method according to claim 1, characterized in that The determining, based on the time intervals between the trigger timestamps of the reference perception data frame and the trigger timestamps of the perception data frames of other perception devices among the multiple perception devices, first synchronization frames corresponding to the multiple perception devices respectively include: Acquire the sensing data frames collected by the multiple sensing devices; Traversing the perception data frames collected by the other perception devices among the multiple perception devices except the reference perception device, and determining a first perception data frame whose time interval between a trigger timestamp and a trigger timestamp of the reference perception data frame is less than a first preset time threshold; When each other perception device determines the corresponding first perception data frame, the first perception data frame is determined as the first synchronization frame corresponding to the other perception device, and the reference perception data frame is determined as the first synchronization frame corresponding to the reference perception device.

3. The method according to claim 2, characterized in that The acquiring of the perception data frames collected by the plurality of perception devices comprises: Acquiring the perception data frames collected by the plurality of perception devices and storing them in a perception data set includes: If the acquired perception data frame is not the reference perception data frame collected by the reference perception device, storing the acquired perception data frame into the perception data set; or, When the acquired perception data frame is a reference perception data frame collected by the reference perception device and the reference perception data frame does not exist in the current perception data set, the acquired reference perception data frame is stored in the perception data set, and the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set are updated.

4. The method according to claim 2 or 3, characterized in that The acquiring of the perception data frames collected by the plurality of perception devices comprises: Acquiring the perception data frames collected by the plurality of perception devices and storing them in a perception data set includes: In the case where the acquired perception data frame is a reference perception data frame collected by the reference perception device, and there is a historical reference perception data frame in the current perception data set, traverse all perception data frames in the perception data set, remove perception data frames whose time interval between a trigger timestamp and a trigger timestamp of a newly acquired reference perception data frame is greater than a second preset time threshold from the perception data set, store the newly acquired reference perception data frame in the perception data set, and update the trigger timestamp of the reference perception data frame and the storage position of the reference perception data frame in the perception data set; The determining of a reference sensing data frame of a reference sensing device among the plurality of sensing devices comprises: When it is determined that the perception data set stores the perception data frames collected by each perception device, a reference perception data frame of a reference perception device is determined from the perception data set according to a storage position of a most recently updated reference perception data frame in the perception data set.

5. The method according to claim 3, characterized in that The method further comprises: After determining the first synchronization frames respectively corresponding to the multiple sensing devices, the reference sensing data frame corresponding to the reference sensing device and the corresponding first sensing data frames determined for the other sensing devices are removed from the sensing data set.

6. The method according to claim 2, characterized in that Also includes: In the case that any other sensing device fails to determine the corresponding first sensing data frame, determining a reference sensing data frame that is most recently updated after the currently used reference sensing data frame; And continue to traverse the perception data frames currently collected by the other perception devices, and determine the first perception data frame whose time interval between the trigger timestamp and the trigger timestamp of the reference perception data frame most recently updated after the currently used reference perception data frame is less than the first preset time threshold.

7. The method according to claim 2, characterized in that Also includes: When there is any other perception device that determines at least two corresponding first perception data frames, the first perception data frame with the smallest time interval between the trigger timestamp of the reference perception data frame and the at least two corresponding first perception data frames is determined as the first perception data frame corresponding to the other perception device.

8. The method according to claim 1, characterized in that For each current perception data frame corresponding to the plurality of perception devices acquired respectively, when each current perception data frame complies with a preset timing rule, determining to complete the perception data synchronization of the timing corresponding to each current perception data frame includes: For each current perception data frame corresponding to the multiple perception devices respectively acquired each time, determine whether a difference between a first trigger timestamp of the current perception data frame and a second trigger timestamp of a previous perception data frame of the current perception data frame of the corresponding perception device is less than a third preset time threshold; When the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time threshold, it is determined that the perception data synchronization of the corresponding timing of each current perception data frame is completed.

9. The method according to claim 8, characterized in that Also includes: In the case where the difference between the first trigger timestamp and the corresponding second trigger timestamp of the current perception data frame of at least one first perception device is greater than or equal to the third preset time threshold, determining that the perception data synchronization of the time sequence corresponding to the current perception data frame fails; Taking the current perception data frame corresponding to the at least one first perception device as the latest acquired new current perception data frame, and determining a corresponding second trigger timestamp for the new current perception data frame; Determine a second sensing device among the multiple sensing devices, for which a difference between a first trigger timestamp of a current sensing data frame and a corresponding second trigger timestamp is less than the third preset time threshold; Acquire a new current perception data frame for the second perception device, and continue to execute the step of determining whether the difference between the first trigger timestamp of the new current perception data frame and the corresponding second trigger timestamp is less than the third preset time threshold for each new current perception data frame corresponding to the first perception device and the second perception device respectively acquired most recently.

10. The method according to claim 9, characterized in that Before acquiring a new current perception data frame for the second perception device, the method further includes: Determine a minimum trigger timestamp among first trigger timestamps respectively corresponding to current perception data frames of the at least one first perception device; Comparing a first trigger timestamp of a current perception data frame corresponding to the second perception device with the minimum trigger timestamp; The acquiring a new current perception data frame for the second perception device includes: If there is a second perception device that satisfies that the difference between the first trigger timestamp of the current perception data frame corresponding to the second perception device and the minimum trigger timestamp is greater than or equal to the fourth preset time threshold, obtain a new current perception data frame for the second perception device; The method further comprises: When there is a second perception device, and the difference between the first trigger timestamp of the current perception data frame corresponding to the second perception device and the minimum trigger timestamp is less than the fourth preset time threshold, the current perception data frame corresponding to the second perception device is used as the new current perception data frame most recently obtained by the corresponding second perception device.

11. The method according to claim 9 or 10, characterized in that The method of using the current perception data frame corresponding to the at least one first perception device as a new current perception data frame to be acquired next time, and determining a corresponding second trigger timestamp for the new current perception data frame, includes: The current perception data frame corresponding to the at least one first perception device is used as the new current perception data frame to be acquired next time, and the first trigger timestamp of the current perception data frame corresponding to the second perception device is determined as the second trigger timestamp corresponding to the new current perception data frame of the at least one first perception device.

12. The method according to claim 8, characterized in that When the difference between the first trigger timestamp and the corresponding second trigger timestamp of each current perception data frame is less than the third preset time interval threshold, before determining that the perception data frame synchronization of the corresponding timing of each current perception data frame is completed, the method further includes: Respectively determine a difference between a first trigger timestamp of a first current perception data frame other than a current reference perception data frame corresponding to the reference perception device in each current perception data frame and a first trigger timestamp of the current reference perception data frame; The step of determining to complete the synchronization of the perception data frames corresponding to the timing of each current perception data frame includes: When the difference between the first trigger timestamp of the first current perception data frame and the first trigger timestamp of the current reference perception data frame is less than a fifth preset time threshold, determining that the perception data frame synchronization of the corresponding timing of each current perception data frame is completed; or, The method further comprises: When there is a difference between at least one first trigger timestamp of the first current perception data frame and the first trigger timestamp of the current reference perception data frame that is greater than or equal to a fifth preset time threshold, it is determined that the perception data frame synchronization of the timing corresponding to the current perception data frame fails.

13. The method according to claim 8, characterized in that In the case where the current perception data frames respectively corresponding to the multiple perception devices are the first synchronization frames, the method further includes: According to the first trigger timestamps respectively corresponding to the first synchronization frames and the acquisition frequency of the sensing device to which the first synchronization frames belong, the second trigger timestamps respectively corresponding to the first synchronization frames are determined.

14. A time synchronization device for sensing data, characterized in that: include: A first frame synchronization module, used to determine a reference perception data frame of a reference perception device among multiple perception devices, and determine the first synchronization frames corresponding to the multiple perception devices respectively based on the time intervals between the trigger timestamps of the reference perception data frame and the trigger timestamps of the perception data frames of other perception devices among the multiple perception devices; A data acquisition module, used to sequentially acquire the perception data frames collected by the plurality of perception devices after the corresponding first synchronization frame; A time synchronization module is used to determine the completion of perception data synchronization corresponding to the timing of each current perception data frame corresponding to the multiple perception devices obtained each time, when each current perception data frame complies with a preset timing rule; wherein the preset timing rule indicates that the time interval between the trigger timestamps of each two adjacent perception data frames in the perception data frame sequence corresponding to each perception device is within a preset range.

15. An unmanned vehicle, characterized in that: It comprises a time synchronization device for sensing data as claimed in claim 14.

Images