Time synchronization method and related device for heterogeneous sensors

Abstract

This invention provides a time synchronization method and related apparatus for heterogeneous sensors, relating to the field of autonomous driving technology. The heterogeneous sensor time synchronization method employs a hierarchical time synchronization mechanism. First, for multiple cameras acquiring data at the same time, a unified target image timestamp is determined and the timestamp of each frame is updated, effectively resolving minor time deviations caused by trigger delays and hardware clock drift between multiple cameras, thereby achieving time synchronization of multiple cameras within the visual domain. Furthermore, using the target image timestamp as a reference, the sensing data acquired by non-camera sensors is synchronized, achieving high-precision time synchronization of heterogeneous sensors on a visual time reference, thus realizing spatiotemporal consistency of end-to-end perception data.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, specifically to a time synchronization method and related apparatus for heterogeneous sensors. Background Technology

[0002] With the rapid development of autonomous driving technology, single sensors are no longer sufficient to meet the safety and reliability requirements of complex traffic scenarios in high-level autonomous driving perception systems. Currently, autonomous driving perception systems typically adopt a heterogeneous sensor fusion architecture, which uses heterogeneous sensors such as cameras, LiDAR, millimeter-wave radar, and inertial measurement units (IMUs) to work together to improve the robustness and detection accuracy of the autonomous driving perception system.

[0003] However, heterogeneous sensors generally suffer from time synchronization issues. If high-precision time synchronization of heterogeneous sensor data is not achieved, spatiotemporal misalignment can occur during the data fusion process in autonomous driving perception systems. This can lead to phenomena such as moving target position deviation, obstacle misjudgment, and trajectory prediction distortion, severely reducing the system's perception accuracy and stability, and even posing safety hazards. Therefore, a high-precision time synchronization method for heterogeneous sensor data is urgently needed. Summary of the Invention

[0004] The purpose of this invention is to provide a time synchronization method and related apparatus for heterogeneous sensors, so as to solve the problem of inaccurate data fusion of heterogeneous sensors caused by time asynchrony in related technologies.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A time synchronization method for heterogeneous sensors includes: when the heterogeneous sensors are time-synchronized to the master clock of a vehicle domain controller, acquiring multiple frames of images captured simultaneously by multiple cameras in the heterogeneous sensors, and sensing data collected by non-camera sensors in the heterogeneous sensors, wherein each frame of the multiple images is marked with a first global timestamp, the first global timestamp being the midpoint of the effective imaging time of the image, and the sensing data is marked with a second global timestamp; if the first global timestamp of each frame of the image is timestamped, then a target image timestamp is determined based on the first global timestamp of each frame of the image, and the timestamp of each frame of the image is updated to the target image timestamp; if the first global timestamp of at least one frame of the image is not timestamped, then each frame of the image is marked as an invalid image, and time synchronization with non-camera sensors is not performed; and using the target image timestamp as a reference, the sensing data is time-synchronized according to the second global timestamp.

[0007] Furthermore, based on the target image timestamp, the sensor data is time-synchronized according to the second global timestamp, including: determining the differentiated time synchronization window corresponding to the non-camera sensor based on the target image timestamp; determining the target data corresponding to the differentiated time synchronization window in the sensor data according to the second global timestamp; and interpolating the target data to obtain the time synchronization data of the non-camera sensor at the target image timestamp.

[0008] Furthermore, based on the target image timestamp, a differentiated time synchronization window corresponding to the non-camera sensor is determined, including: based on the target image timestamp, the differentiated time synchronization window is determined according to the sampling frequency of the non-camera sensor.

[0009] Furthermore, based on the second global timestamp, the target data corresponding to the differentiated time synchronization window is determined in the sensing data, including: searching for all data in the sensing data whose second global timestamp is within the differentiated time synchronization window; for each data in all data, calculating the time difference between the second global timestamp of the data and the timestamp of the target image, and determining the target data based on the time difference.

[0010] Furthermore, the target image timestamp is determined based on the first global timestamp of each frame image, including: selecting a first global timestamp from the first global timestamps of each frame image as the target image timestamp.

[0011] Furthermore, the time stamp alignment of the first global timestamp of each frame is determined by the following method: determining whether the first global timestamp of each frame is within the image time synchronization window; if the first global timestamp of each frame is within the image time synchronization window, then the time stamp alignment of each frame is determined.

[0012] Furthermore, before acquiring multiple frames of images captured simultaneously by multiple cameras in a heterogeneous sensor, the method further includes: sending a frame trigger pulse signal to the multiple cameras, the frame trigger pulse signal carrying the trigger time, the frame trigger pulse signal being used to trigger the multiple cameras to acquire images at the trigger time.

[0013] Furthermore, the first global timestamp is obtained in the following way: for each camera in the multi-camera setup, the first global timestamp is calculated based on the camera's exposure time and trigger time.

[0014] Furthermore, synchronizing the time of the heterogeneous sensors to the master clock of the vehicle domain controller includes: acquiring the reference time of the master clock, which is referenced to the time after the vehicle is powered on; broadcasting a time synchronization signal to the heterogeneous sensors, which is used to control the heterogeneous sensors to generate a global timestamp of the heterogeneous sensors based on the local timestamp and time offset, and the global timestamp is synchronized with the master clock.

[0015] Furthermore, after synchronizing the sensor data in time, the process also includes packaging and outputting the time synchronization data corresponding to the heterogeneous sensors.

[0016] A time synchronization device for heterogeneous sensors, comprising:

[0017] The acquisition module is used to acquire multiple frames of images collected by multiple cameras in the heterogeneous sensors at the same time, and sensing data collected by non-camera sensors in the heterogeneous sensors, when the heterogeneous sensors are synchronized to the master clock of the vehicle domain controller. Each frame of the multiple images is marked with a first global timestamp, which is the midpoint of the effective imaging time of the image. The sensing data is marked with a second global timestamp.

[0018] The determination module is used to determine the target image timestamp based on the first global timestamp of each frame image when the first global timestamp of each frame image is aligned with the timestamp, and update the timestamp of each frame image to the target image timestamp; if the first global timestamp of at least one frame image is not aligned with the timestamp, then each frame image is marked as an invalid image and time synchronization with non-camera sensors is not performed.

[0019] The time synchronization module is used to synchronize the sensor data based on the target image timestamp and the second global timestamp.

[0020] A time synchronization system for heterogeneous sensors includes: a vehicle domain controller and heterogeneous sensors, wherein the vehicle domain controller is equipped with a time synchronization module and a main control module;

[0021] The time synchronization module is used to synchronize the time of heterogeneous sensors to the master clock;

[0022] The main control module is used to execute the time synchronization method for heterogeneous sensors provided above.

[0023] A vehicle includes: a vehicle body and a time synchronization system for heterogeneous sensors as described above.

[0024] A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement a time synchronization method for heterogeneous sensors as provided in any of the preceding claims.

[0025] A computer program product includes: a computer program that, when executed by a processor, implements a time synchronization method for heterogeneous sensors as provided in any of the above.

[0026] The beneficial effects of this invention are:

[0027] This invention provides a time synchronization method and related apparatus for heterogeneous sensors. The method involves acquiring multiple frames of images captured simultaneously by multiple cameras within the heterogeneous sensors, along with sensor data collected by non-camera sensors, when the heterogeneous sensors are synchronized to the master clock of the vehicle domain controller. Each frame is marked with a first global timestamp, which is the midpoint of the effective imaging time of the image. The sensor data is marked with a second global timestamp. If the first global timestamp of each frame is already timestamped, a target image timestamp is determined based on the first global timestamp of each frame, and the timestamp of each frame is updated to the target image timestamp. Using the target image timestamp as a reference, the sensor data is time-synchronized according to the second global timestamp. This invention employs a layered time synchronization mechanism. First, for multiple cameras acquiring data at the same time, it determines a unified target image timestamp and updates the timestamp of each frame, effectively resolving minor time discrepancies caused by trigger delays and hardware clock drift between multiple cameras, thereby achieving time synchronization of multiple cameras within the visual domain. Furthermore, using the target image timestamp as a reference, it synchronizes the sensing data acquired by non-camera sensors, achieving high-precision time synchronization of heterogeneous sensors on a visual time reference, thus realizing spatiotemporal consistency of end-to-end sensing data. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the time synchronization system for heterogeneous sensors provided in an embodiment of the present invention;

[0029] Figure 2 This is a diagram of a heterogeneous sensor time synchronization architecture provided in an embodiment of the present invention;

[0030] Figure 3 A flowchart illustrating the time synchronization method for heterogeneous sensors provided in this embodiment of the invention. Figure 1 ;

[0031] Figure 4 A flowchart illustrating the time synchronization method for heterogeneous sensors provided in this embodiment of the invention. Figure 2 ;

[0032] Figure 5 This is a schematic diagram illustrating the time synchronization of multiple frames of images acquired by multiple cameras with sensor data acquired by non-camera sensors, provided in an embodiment of the present invention.

[0033] Figure 6 A schematic diagram illustrating the frame image timestamp alignment process provided in an embodiment of the present invention;

[0034] Figure 7 This is a schematic diagram of the time synchronization device for heterogeneous sensors provided in an embodiment of the present invention.

[0035] The accompanying drawings have illustrated specific embodiments of the invention, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the invention in any way, but rather to illustrate the concept of the invention to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0036] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0037] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0038] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0039] The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes elements is not excluded. For example, the use of terms such as "first," "second," etc., to indicate names does not imply any particular order.

[0040] Currently, the synchronization of heterogeneous sensor data typically employs hardware-triggered synchronization and software timestamp alignment. Hardware-triggered synchronization achieves synchronization by uniformly controlling the acquisition timing of heterogeneous sensors. For example, it pre-establishes a correspondence between LiDAR scanning times and camera exposure times, or controls the camera's initial scanning time based on parameters such as the LiDAR's scanning angle and rotation speed, ensuring spatiotemporal consistency between point cloud data and images. However, this approach has significant limitations: firstly, it is complex, costly, and limited to specific application scenarios, making it difficult to cover all types of sensors; secondly, it does not fully consider the scanning characteristics of each sensor (such as differences in camera exposure times), and given the unavoidable transmission link and processing delays, its synchronization error is relatively large.

[0041] Software timestamp alignment relies on the system clock to correct the timestamps of heterogeneous sensor data. However, this method is susceptible to network latency, operating system scheduling, and other factors, resulting in insufficient synchronization accuracy and large errors. In image-based autonomous driving perception systems, the visual network composed of multiple image sensors distributed around the vehicle is the foundation for environmental perception and understanding. The accuracy deficiencies of existing synchronization methods have become a key bottleneck restricting perception performance.

[0042] To address the aforementioned issues, this invention employs a layered time synchronization mechanism. First, for multiple cameras acquiring data at the same time, a unified image timestamp is determined and the timestamp of each frame is updated. This effectively resolves minor time discrepancies caused by trigger delays and hardware clock drift between multiple cameras, thereby achieving time synchronization of multiple cameras within the visual domain. Furthermore, using the image timestamp as a reference, time synchronization is performed on sensor data acquired by non-camera sensors, achieving high-precision time synchronization of heterogeneous sensors on a visual time reference, thus realizing spatiotemporal consistency of end-to-end sensing data.

[0043] The application scenarios of the embodiments of the present invention will be described below first.

[0044] The time synchronization method for heterogeneous sensors provided in this invention is applicable to high-level autonomous driving perception systems (such as L2 and above), and is especially suitable for high-precision collaborative perception of heterogeneous sensors in dynamic driving scenarios.

[0045] The application scenarios mentioned above are only some examples. Those skilled in the art can expand the applications according to specific needs and scenarios. The embodiments of the present invention do not impose specific limitations on this.

[0046] Figure 1 This is a schematic diagram of the time synchronization system for heterogeneous sensors provided in an embodiment of the present invention. Figure 1As shown, the time synchronization system for the heterogeneous sensor includes a vehicle domain controller and a heterogeneous sensor. The vehicle domain controller is equipped with a time synchronization module and a main control module.

[0047] The time synchronization module is used to synchronize the time of heterogeneous sensors to the master clock.

[0048] The main control module is used to execute the time synchronization method for heterogeneous sensors provided in the embodiments of the present invention.

[0049] For example, the time synchronization module can be a high-precision time synchronization module, such as a hardware clock supported by the Generalized Precision Time Protocol (gPTP) or a Global Navigation Satellite System (GNSS) timing module. Specifically, when the time synchronization module is a gPTP-supported hardware clock, the corresponding master clock is the Vehicle Interface Unit (VIU) clock; when the time synchronization module is a GNSS timing module, the corresponding master clock is the Coordinated Universal Time (UTC) clock.

[0050] For example, heterogeneous sensors include, but are not limited to, multi-camera systems, LiDAR, millimeter-wave radar, and IMU.

[0051] Figure 2 This is a diagram illustrating a heterogeneous sensor time synchronization architecture provided in an embodiment of the present invention. Figure 2 As shown, this heterogeneous sensor time synchronization architecture uses a real-time clock (Real) The Time Clock (RTC) is the clock source, which synchronizes the time synchronization signal to heterogeneous sensors through different buses or interfaces.

[0052] Among them, RTC can be the master clock mentioned above, serving as the time reference source for the time synchronization system of heterogeneous sensors, and is used to provide a unified high-precision timestamp.

[0053] like Figure 2As shown, multiple cameras communicate with the RTC via Pulse Per Second (PPS) and Inter-Process Communication (IPC) to achieve time synchronization between the multiple cameras and the vehicle domain controller; the LiDAR communicates with the RTC via Ethernet to achieve time synchronization between the LiDAR and the vehicle domain controller; the millimeter-wave radar communicates with the vehicle domain controller via the Controller Area Network (CAN) bus to achieve time synchronization between the millimeter-wave radar and the vehicle domain controller; and the IMU communicates with the RTC via the Serial Peripheral Interface (SPI) to achieve time synchronization between the IMU and the vehicle domain controller.

[0054] The following is based on Figure 1 The time synchronization system of the heterogeneous sensor shown is the main execution body. The specific implementation of the time synchronization method of the heterogeneous sensor provided in the embodiments of the present invention will be described in detail with reference to specific embodiments.

[0055] Figure 3 A flowchart illustrating the time synchronization method for heterogeneous sensors provided in this embodiment of the invention. Figure 1 .like Figure 3 As shown, a specific implementation of the time synchronization method for heterogeneous sensors may include the following steps:

[0056] S301, when the heterogeneous sensors are synchronized to the master clock of the vehicle domain controller, acquire multiple frames of images collected by multiple cameras in the heterogeneous sensors at the same time, and sensor data collected by non-camera sensors in the heterogeneous sensors. Each frame of the multiple images is marked with a first global timestamp, which is the midpoint of the effective imaging time of the image. The sensor data is marked with a second global timestamp.

[0057] For example, the master clock can be a VIU clock or a UTC clock. This embodiment of the invention does not limit this, and the specific clock can be determined according to the actual application requirements.

[0058] It is understandable that heterogeneous sensors each have their own internal hardware clock, and these internal hardware clocks have slight deviations, and clock drift may occur during operation. This embodiment of the invention synchronizes the time of the heterogeneous sensors to the master clock of the vehicle domain controller, that is, it calibrates the internal hardware clocks of the heterogeneous sensors with the master clock of the domain controller. This allows the heterogeneous sensors to use the master clock's timestamp when outputting data.

[0059] For example, heterogeneous sensors may include multiple camera sensors and non-camera sensors. Among them, non-camera sensors include, but are not limited to, LiDAR, millimeter-wave radar, and IMU.

[0060] For example, a multi-channel camera may include a wide-angle camera (which can also be described as a short-focus camera) and a non-wide-angle camera (such as a telephoto camera). This embodiment of the invention does not limit the specific number, type, or installation location of the cameras in the multi-channel camera system; the specific details can be determined according to actual application requirements.

[0061] For example, when multiple cameras acquire multiple frames of images at the same time, the corresponding "same time" can be the hardware trigger time of the vehicle domain controller, which is controlled by the master clock.

[0062] For example, multiple frames of images captured by multiple cameras in a heterogeneous sensor at the same time, and sensing data collected by non-camera sensors in a heterogeneous sensor, can be data collected by the heterogeneous sensor within the same control cycle.

[0063] For example, multiple frames may include one frame captured by each of the multiple cameras. The first global timestamp of each frame may be the same or different.

[0064] For example, the effective imaging time midpoint can be the midpoint between the start and end times of an image frame captured by the camera, used to characterize the true equivalent acquisition time of that image frame.

[0065] Understandably, when a camera uses progressive or global exposure, the entire frame image is not formed in a single instant, but rather over a continuous exposure time. In order to more accurately represent the actual physical moment corresponding to the frame image, the midpoint of the exposure time period, i.e. the midpoint of the effective imaging time, is usually used as the reference timestamp of the frame image for subsequent time synchronization and spatiotemporal alignment with heterogeneous sensors such as LiDAR, millimeter-wave radar, and IMU.

[0066] It is understandable that the first global timestamp marked on each frame of image and the second global timestamp marked on the sensor data are both timestamps corresponding to the master clock in the vehicle domain controller. That is, the local timestamps of heterogeneous sensors are synchronized to the global clock domain to ensure the comparability of cross-domain times of heterogeneous sensors.

[0067] For example, the sensor data may include, but is not limited to, point cloud frames acquired by lidar, target detection lists acquired by millimeter-wave radar, and vehicle attitude information (such as roll, pitch, yaw, acceleration, angular velocity, and attitude angles) acquired by IMU. The second global timestamps for different sensor data may be the same or different.

[0068] In this step, one possible implementation is as follows: when the time of the heterogeneous sensors is synchronized to the master clock of the vehicle domain controller through the time synchronization module, the main control module in the vehicle domain controller acquires multiple frames of images collected by multiple cameras in the heterogeneous sensors at the same time, and the sensing data collected by non-camera sensors in the heterogeneous sensors in real time.

[0069] S302, if the first global timestamp of each frame image is already timestamp aligned, then determine the target image timestamp based on the first global timestamp of each frame image, and update the timestamp of each frame image to the target image timestamp.

[0070] For example, the target image timestamp can be the first global timestamp of the image corresponding to the wide-angle camera (short-focus camera) among multiple cameras.

[0071] It is understandable that wide-angle cameras have a large field of view, a high degree of overlap with the sensing areas of sensors such as LiDAR, millimeter-wave radar, and IMU, and have less image motion blur and high timestamp stability. Therefore, they are more suitable as a benchmark for time synchronization of heterogeneous sensors.

[0072] In this step, one possible implementation is as follows: if the first global timestamp of each frame image is already timestamp aligned, then the first global timestamp of the image corresponding to the wide-angle camera in the multi-camera system is determined as the target image timestamp, and the timestamps of each frame image corresponding to the other cameras besides the wide-angle camera are updated from the first global timestamp to the target image timestamp.

[0073] Understandably, by updating the timestamp of each frame to the timestamp of the target image, the slight time discrepancies caused by trigger delays and hardware clock drift between multiple cameras can be effectively resolved, thereby achieving time synchronization of multiple cameras within the visual domain.

[0074] S303 uses the target image timestamp as a reference and synchronizes the sensing data according to the second global timestamp.

[0075] Understandably, after completing the time synchronization of multiple frames of images captured by multiple cameras at the same time, it is necessary to synchronize the multiple frames of images with the sensor data collected by non-camera sensors at the same time point, i.e., the timestamp of the target image, in order to improve the accuracy of subsequent data fusion using the time synchronization data of heterogeneous sensors.

[0076] It is understandable that the sampling frequency of non-camera sensors (such as 10Hz for LiDAR, 25Hz for millimeter-wave radar, and 100Hz for IMU) is different. The sampling frequency (1000Hz) of the non-camera sensor is inconsistent with that of the camera (e.g., 30Hz), causing non-camera sensors to be unable to directly acquire the sensor data of the target image timestamp. Therefore, an interpolation algorithm can be used to align the sensor data of the non-camera sensor with the timestamp of the target image, thus completing the time synchronization of the sensor data.

[0077] One possible implementation of this step is to use the target image timestamp as a reference and employ an interpolation algorithm to synchronize the sensing data in time based on the second global timestamp of the sensing data collected by the non-camera sensor.

[0078] In this embodiment of the invention, by employing a hierarchical time synchronization mechanism, firstly, for multiple cameras acquiring data at the same time, a unified target image timestamp is determined and the timestamp of each frame is updated, effectively solving the minor time deviations caused by trigger delays and hardware clock drift between multiple cameras, thereby achieving time synchronization of multiple cameras within the visual domain; furthermore, using the target image timestamp as a reference, the sensing data acquired by non-camera sensors is time-synchronized, achieving high-precision time synchronization of heterogeneous sensors on a visual time reference, thereby achieving spatiotemporal consistency of end-to-end sensing data.

[0079] The following is combined with Figure 4 The specific implementation method of step S303, which uses the target image timestamp as a reference and synchronizes the sensing data in time according to the second global timestamp, is described in detail.

[0080] Figure 4 A flowchart illustrating the time synchronization method for heterogeneous sensors provided in this embodiment of the invention. Figure 2 .like Figure 4 As shown, in this heterogeneous sensor time synchronization method, a specific implementation of time synchronization of sensor data based on the target image timestamp and the second global timestamp may include the following steps:

[0081] S401, using the target image timestamp as a reference, determines the differentiated time synchronization window corresponding to the non-camera sensor.

[0082] For example, the center of each window in the differential time synchronization window can be the target image timestamp.

[0083] For example, the differentiated time synchronization window for non-camera sensors can include time synchronization windows for different sensors, such as the time synchronization window for LiDAR, the time synchronization window for millimeter-wave radar, and the time synchronization window for IMU. The size of the time synchronization window differs for different sensors.

[0084] Optionally, one possible implementation of this step is to determine a differentiated time synchronization window based on the sampling frequency of the non-camera sensor, using the target image timestamp as a reference.

[0085] The sampling frequency of non-camera sensors is similar to that described above, and will not be repeated here.

[0086] For example, the time synchronization window corresponding to the lidar can be... The time synchronization window for millimeter-wave radar can be The time synchronization window corresponding to the IMU can be The embodiments of the present invention do not limit the size of the time synchronization window corresponding to different sensors; it can be determined according to the actual application requirements.

[0087] S402, based on the second global timestamp, determine the target data corresponding to the differentiated time synchronization window in the sensing data.

[0088] For example, the target data can be data within a differential time synchronization window.

[0089] Optionally, one possible implementation of this step is: in the sensing data, search for all data whose second global timestamp is within the differential time synchronization window; for each data in all data, calculate the time difference between the second global timestamp of the data and the timestamp of the target image, and determine the target data based on the time difference.

[0090] For example, the target data can be the two frames of data whose corresponding second global timestamp is closest to the target image timestamp, that is, the two frames of data before and after the target image timestamp that are closest to the target image timestamp.

[0091] The specific implementation methods of different sensors are explained below.

[0092] 1) For the sensing data collected by the lidar, for example, one possible implementation is: search for all data in the sensing data (i.e. point cloud data) collected by the lidar where the second global timestamp is within the time synchronization window corresponding to the lidar, calculate the time difference between the second global timestamp of the data and the timestamp of the target image for each data, and determine the two consecutive point cloud frames closest to the timestamp of the target image as the target data.

[0093] For example, when there is no data corresponding to the second global timestamp within the time synchronization window, the size of the time synchronization window corresponding to the LiDAR can be changed, such as increasing the size of the time synchronization window corresponding to the LiDAR (e.g., by...). Adjusted to ), and perform data searches based on the adjusted time synchronization window.

[0094] 2) For the sensing data acquired by millimeter-wave radar, an exemplary possible implementation is as follows: search all data in the sensing data (i.e., the target detection list) acquired by millimeter-wave radar where the second global timestamp is within the time synchronization window corresponding to the millimeter-wave radar. For each data, calculate the time difference between the second global timestamp of the data and the target image timestamp, and determine the two frames of the target detection list closest to the target image timestamp as the target data. The target detection list includes, but is not limited to, target position, velocity, and confidence level. Another possible implementation is as follows: if there is no sensing data in the sensing data acquired by millimeter-wave radar where the second global timestamp coincides with the target image timestamp, then search the sensing data acquired by millimeter-wave radar where the second global timestamp is closest to the target image timestamp within the time synchronization window corresponding to the millimeter-wave radar as the target data.

[0095] 3) For the sensor data collected by the IMU, for example, one possible implementation is: search for all data in the sensor data (i.e. vehicle attitude information) collected by the IMU where the second global timestamp is within the time synchronization window corresponding to the IMU, calculate the time difference between the second global timestamp of the data and the timestamp of the target image for each data, and determine the vehicle attitude information of the two frames closest to the timestamp of the target image as the target data.

[0096] S403, interpolate the target data to obtain time synchronization data of the non-camera sensor at the target image timestamp.

[0097] For example, for LiDAR, one possible implementation is to use a linear interpolation algorithm to interpolate the target data (such as point cloud pose) to obtain the LiDAR's time synchronization data at the target image timestamp, i.e., the point cloud pose at the target image timestamp. Here, the point cloud pose can be a pose transformation matrix.

[0098] For example, the interpolation of target data (such as point cloud pose) using a linear interpolation algorithm can be expressed by the following formula:

[0099]

[0100] in, Indicates the timestamp of the target image The pose transformation matrix at any given time is the time synchronization data. , They represent the first Time, Number The pose transformation matrix corresponding to each time step.

[0101] It is understandable that, through linear interpolation algorithms, for The pose transformation matrix at time t and the first Motion compensation can be performed on the pose transformation matrix at each time step to obtain alignment to... The pose transformation matrix at any given time, i.e., time synchronization data.

[0102] For example, for millimeter-wave radar, one possible implementation is to use a linear interpolation algorithm to interpolate the target data (such as a target detection list) to obtain the millimeter-wave radar's time synchronization data at the target image timestamp, i.e., the target detection list at the target image timestamp.

[0103] For example, the object detection list can be represented in the form of a state vector: such as ,in, This represents the state vector of target j. Indicates the position of target j. This represents the velocity of target j.

[0104] For example, interpolating target data (such as a target detection list) using a linear interpolation algorithm can be expressed by the following formula:

[0105]

[0106] in, Indicates the timestamp of the target image The state vector at any given moment is the time synchronization data. They represent the first Time and the The state vector at time t.

[0107] It is understandable that, through linear interpolation algorithms, for The state vector at time t and the first Motion compensation can be performed on the state vector at each time step to obtain an alignment to... The state vector at any given moment, i.e., time synchronization data.

[0108] For example, for millimeter-wave radar, another possible implementation is to determine the target data, which is the frame of data closest to the target image timestamp, as the time synchronization data corresponding to the target image timestamp.

[0109] For example, another possible implementation for millimeter-wave radar is to obtain time synchronization data of the target image timestamp by using historical trajectory prediction.

[0110] For example, for the IMU, a linear interpolation algorithm is used to interpolate the target data to obtain the IMU's time synchronization data at the target image timestamp. The specific implementation of the linear interpolation is similar to that described above and will not be repeated here.

[0111] Understandably, compared to related technologies, when a vehicle is traveling at high speed, the heterogeneous sensor data collected at different times exhibits spatial deviations due to changes in vehicle pose, affecting the accuracy of heterogeneous sensor data fusion and resulting in a lack of motion compensation. This invention employs linear interpolation to perform motion compensation on target data from non-camera sensors, ensuring consistency of heterogeneous sensor data in both time and space, thereby improving the fusion accuracy of heterogeneous sensor data.

[0112] In this embodiment of the invention, a differentiated time synchronization window corresponding to a non-camera sensor is determined based on the target image timestamp. Based on the second global timestamp, the target data corresponding to the differentiated time synchronization window is determined in the sensor data. The target data is then further interpolated to obtain the time synchronization data of the non-camera sensor at the target image timestamp. This achieves high-precision time synchronization of heterogeneous sensors on a visual time reference. Furthermore, when fusion is performed based on the time-synchronized heterogeneous sensor data, the fusion accuracy of the heterogeneous sensor data can be improved.

[0113] Figure 5 This is a schematic diagram illustrating the time synchronization of multiple frames of images acquired by multiple cameras and sensor data acquired by non-camera sensors, provided in an embodiment of the present invention. When the control period is 100ms, as... Figure 5 As shown, when the sampling period of the lidar is 100ms, it only acquires one frame of point cloud (L0) in one control cycle; when the sampling period of the multi-camera is 33ms, it can acquire 4 sets of frame images (M0 to M3) in one control cycle, where each camera sampling period contains one frame image acquired by each of the multi-cameras; when the sampling period of the millimeter-wave radar is 40ms, it can acquire 3 frames of target detection list (R0 to R2) in one control cycle; when the sampling period of the IMU is 10ms, it can acquire 10 frames of data (I0 to I9) in one control cycle.

[0114] like Figure 5 As shown, the time synchronization of heterogeneous sensors can be implemented as follows: For the four sets of frame images acquired by multiple cameras, if the first global timestamp of each frame in each set of frame images is already timestamped, then the time synchronization windows of the LiDAR, millimeter-wave radar, and IMU are determined based on the target image timestamp of that set of frame images. Furthermore, the sensing data acquired by the LiDAR is synchronized based on the LiDAR's time synchronization window, the sensing data acquired by the millimeter-wave radar is synchronized based on the millimeter-wave radar's time synchronization window, and the sensing data acquired by the IMU is synchronized based on the IMU's time synchronization window. The specific implementation method is similar to the above and will not be repeated here.

[0115] Optionally, one possible implementation of determining the target image timestamp based on the first global timestamp of each frame image in step S302 is to select a first global timestamp from the first global timestamps of each frame image as the target image timestamp.

[0116] For example, the first global timestamp selected can be the first global timestamp of the image corresponding to the wide-angle camera among multiple cameras.

[0117] The specific implementation method is similar to that described above, and will not be repeated here.

[0118] Optionally, in the time synchronization method for heterogeneous sensors provided in the embodiments of the present invention, one possible implementation of determining whether the first global timestamp of each frame image is timestamp aligned can be: determining whether the first global timestamp of each frame image is within the image time synchronization window; if the first global timestamp of each frame image is within the image time synchronization window, then determining that each frame image is timestamp aligned.

[0119] For example, the image time synchronization window is determined based on the sampling frequency of the multiple cameras. For instance, the image time synchronization window can be... .

[0120] For example, if the first global timestamp of at least one frame is not within the image time synchronization window, then it is determined that the timestamps of each frame are misaligned, and each frame is marked as an invalid image, and no subsequent time synchronization with non-camera sensors is performed.

[0121] Understandably, when at least one frame image from a camera is missing, multiple frames captured by multiple cameras at the same time are marked as invalid, and no subsequent time synchronization with non-camera sensors is performed.

[0122] Figure 6 This is a schematic diagram illustrating the frame image timestamp alignment process provided in an embodiment of the present invention. Figure 6 As shown, for multiple frames of images (image 0 to image n) acquired by multiple cameras at the same time and within the same sampling period, it is determined whether the first global timestamp of each frame of image 0 to image n is within the image time synchronization window. If the first global timestamp of each frame of image is within the image time synchronization window, it is determined that the timestamp of each frame of image is aligned, that is, the time synchronization of each frame of image is successful. If the first global timestamp of at least one frame of image is not within the image time synchronization window, it is determined that the timestamp of each frame of image is not aligned, and each frame of image is marked as invalid, and no subsequent time synchronization with non-camera sensors is performed.

[0123] Optionally, in the time synchronization method for heterogeneous sensors provided in the embodiments of the present invention, before acquiring multiple frames of images captured by multiple cameras in the heterogeneous sensor at the same time, the method further includes: sending a frame trigger pulse signal to the multiple cameras, wherein the frame trigger pulse signal carries a trigger time and is used to trigger the multiple cameras to acquire images at the trigger time.

[0124] For example, the triggering time is controlled by the master clock of the vehicle domain controller.

[0125] In this embodiment, one possible implementation is as follows: when the time of the heterogeneous sensors is synchronized to the master clock of the vehicle domain controller, the vehicle domain controller sends a frame trigger pulse signal carrying the trigger time to multiple cameras through a general-purpose input / output port (GPIO) or a dedicated synchronization signal line, so that the multiple cameras start acquiring images simultaneously based on the trigger time.

[0126] Understandably, compared to related technologies, traditional heterogeneous sensor synchronization methods rely on pure hardware triggering synchronization such as PPS pulses, which suffers from high cost, complex deployment, and poor flexibility. This invention, based on synchronizing the time of heterogeneous sensors to the vehicle domain controller's master clock, employs lightweight hardware triggering for frame-level coarse synchronization of multiple cameras, and further combines this with software timestamp alignment for fine synchronization. Through this dual mechanism of "hardware-triggered coarse synchronization + software timestamp fine alignment," the synchronization accuracy and system flexibility of multi-camera frame acquisition are significantly improved without increasing hardware costs excessively.

[0127] Optionally, in the multiple frames of images captured by multiple cameras at the same time, the first global timestamp of each frame is obtained in the following way: for each camera in the multiple cameras, the first global timestamp is calculated based on the camera's exposure time and trigger time.

[0128] For example, the exposure time of each camera can be defined as the duration between the global timestamp of the start of exposure and the global timestamp of the end of exposure, which are marked by the camera when it acquires an image.

[0129] For example, in one possible implementation, the mapping relationship between the exposure parameters (such as exposure time) of each camera and the environmental scene (such as light intensity and weather conditions) is pre-calibrated. Specifically, each camera can dynamically adjust its exposure time according to the current environmental scene; in the same scene, the exposure time of multiple cameras can be set to be the same to ensure image consistency, or it can be set differently according to the viewing angle and lighting conditions of each camera.

[0130] For example, the first global timestamp can be represented by the following formula:

[0131]

[0132] in, This represents the first global timestamp of the image captured by the i-th camera. Indicates the trigger time. Let represent the exposure time of the i-th camera, where i can take the values ​​1, 2, 3...m.

[0133] Optionally, one possible implementation of synchronizing the heterogeneous sensor time to the main clock of the vehicle domain controller in step S301 is as follows: obtain the reference time of the main clock, which is based on the time after the vehicle is powered on; broadcast a time synchronization signal to the heterogeneous sensor, which is used to control the heterogeneous sensor to generate a global timestamp of the heterogeneous sensor based on the local timestamp and time offset, and the global timestamp is synchronized with the main clock.

[0134] For example, the base time Using the time after the vehicle is powered on (i.e., starting from 0) as a reference, the accuracy can reach the nanosecond level.

[0135] The master clock is similar to the one described above, so it will not be repeated here.

[0136] For example, the time offset of a heterogeneous sensor can be the time deviation of the sensor relative to the master clock caused by factors such as network latency, and this time offset can be a dynamic time offset.

[0137] For example, the time offset of heterogeneous sensors can be an empirical value obtained through periodic time calibration. The time offsets of different sensors can be different.

[0138] For example, when the time synchronization module is a hardware clock and the master clock is a VIU clock (i.e., the reference time is the time of the VIU clock), after system startup, the time synchronization module with the integrated VIU clock broadcasts a Sync message to the vehicle Ethernet via Network Time Protocol (NTP) or gPTP. The LiDAR and vehicle domain controller access the vehicle network via Ethernet, receive the Sync message, and the LiDAR generates a global timestamp synchronized with the master clock based on its local timestamp and a dynamically calculated time offset using gPTP. Simultaneously, the vehicle domain controller completes time calibration with the master clock. Multiple cameras perform hardware time reference calibration based on PPS signals and, in conjunction with the master clock, read the system time from the time-calibrated vehicle domain controller via a driver program. Then, based on the local timestamp and the dynamic time offset, they generate a global timestamp synchronized with the master clock. The millimeter-wave radar receives a CAN message carrying the master clock timestamp from the time synchronization module via the CAN bus and generates a global timestamp synchronized with the master clock based on its local timestamp and the dynamic time offset. The IMU receives the CAN message carrying the master clock timestamp via SPI. The interface communicates with the vehicle domain controller to obtain time information that has been synchronized to the master clock, and generates a global timestamp that is synchronized with the master clock based on the local timestamp and dynamic time offset.

[0139] For example, when the time synchronization module is a GNSS timing module and the master clock is a UTC clock (i.e., the reference time is the UTC clock time), after the system starts, the main control module obtains the UTC time through GNSS and starts NTP or gPTP to synchronize time with the heterogeneous sensor. The specific implementation method is similar to that described above and will not be repeated here.

[0140] For example, when heterogeneous sensors are time-synchronized to the master clock of the vehicle domain controller, the clock deviation between the heterogeneous sensors and the vehicle domain controller is controlled within... Within.

[0141] For example, the global timestamp of a heterogeneous sensor can be represented by the following formula:

[0142]

[0143] in, This represents the global timestamp of sensor e. This represents the local timestamp of sensor e. This represents the time offset of sensor e.

[0144] For example, in the process of synchronizing the time of heterogeneous sensors to the master clock of the vehicle domain controller, for the data acquisition of heterogeneous sensors, a possible implementation is to perform global timestamp synchronization of the acquired data of heterogeneous sensors based on the master clock via NTP or gPTP. For multiple cameras (e.g., each camera generates images at a sampling frequency of 30Hz), local timestamps are marked at the start and end of image exposure, and these local timestamps are synchronized with the master clock to obtain the global timestamp of the image. For LiDAR, a point cloud frame is generated after each scan, and local timestamps are marked at the start and end of the scan, further synchronized with the master clock to obtain the global timestamp of the point cloud frame. The time deviation is controlled within a certain range. Within; for millimeter-wave radar, a local timestamp is marked after each target detection is completed, and the local timestamp is synchronized with the master clock to obtain the global timestamp of the target detection list; for IMU, when outputting vehicle attitude information at a high frequency (such as 100Hz), a local timestamp is marked for each sampling point (i.e., each data packet), and the local timestamp is synchronized with the master clock to obtain the global timestamp of the vehicle attitude information.

[0145] Understandably, by synchronizing the local timestamps of heterogeneous sensors to a unified global timestamp, the comparability and consistency of heterogeneous sensors in the time dimension can be ensured.

[0146] In this embodiment of the invention, by obtaining the reference time of the master clock and broadcasting it to the heterogeneous sensor to control the heterogeneous sensor to generate a global timestamp that keeps the heterogeneous sensor in time synchronize with the master clock based on the local timestamp and time offset, a unified time reference is provided for the heterogeneous sensor. Thus, when performing time synchronization of heterogeneous sensors under this condition, the time synchronization accuracy can be improved.

[0147] Optionally, the time synchronization method for heterogeneous sensors provided in this embodiment of the invention further includes, after synchronizing the sensing data, packaging and outputting the time synchronization data corresponding to the heterogeneous sensors.

[0148] For example, in one possible implementation, multiple frames of images synchronized by multiple cameras, point cloud frames of LiDAR, target detection list of millimeter-wave radar, vehicle attitude information of IMU, and time deviation records of the timestamps of each sensor and the target image are packaged into a single frame of synchronized data and output to the subsequent model inference engine perception fusion module for target detection, tracking, and environment modeling.

[0149] In summary, the time synchronization method for heterogeneous sensors provided in this invention achieves high-precision alignment of heterogeneous sensors in the time dimension by constructing a unified time reference and precise timestamp marking for the sensors, and employing delay compensation and data interpolation mechanisms. This improves the accuracy and stability of heterogeneous sensor data fusion. This method requires no complex hardware triggering, has strong compatibility, good engineering practicality and scalability, and can be widely applied to Level 2 and above autonomous vehicles, especially suitable for image-based autonomous driving perception systems.

[0150] The following are embodiments of the apparatus of the present invention, which can be used to execute embodiments of the method of the present invention. For details not disclosed in the embodiments of the apparatus of the present invention, please refer to the embodiments of the method of the present invention.

[0151] Figure 7 This is a schematic diagram of the time synchronization device for heterogeneous sensors provided in an embodiment of the present invention. Figure 7 As shown, the time synchronization device 70 for the heterogeneous sensor includes an acquisition module 710, a determination module 720, and a time synchronization module 730.

[0152] The acquisition module 710 is used to acquire multiple frames of images collected by multiple cameras in the heterogeneous sensors at the same time when the heterogeneous sensors are synchronized to the master clock of the vehicle domain controller, and sensor data collected by non-camera sensors in the heterogeneous sensors. Each frame of the multiple images is marked with a first global timestamp, which is the midpoint of the effective imaging time of the image. The sensor data is marked with a second global timestamp.

[0153] The determination module 720 is used to determine the target image timestamp based on the first global timestamp of each frame image when the first global timestamp of each frame image is aligned with the timestamp, and update the timestamp of each frame image to the target image timestamp; if the first global timestamp of at least one frame image is not aligned with the timestamp, then each frame image is marked as an invalid image and time synchronization with non-camera sensors is not performed.

[0154] The time synchronization module 730 is used to synchronize the sensor data in time based on the target image timestamp and the second global timestamp.

[0155] Furthermore, the time synchronization module 730 is specifically used to: determine the differentiated time synchronization window corresponding to the non-camera sensor based on the target image timestamp; determine the target data corresponding to the differentiated time synchronization window in the sensing data according to the second global timestamp; and perform interpolation processing on the target data to obtain the time synchronization data of the non-camera sensor at the target image timestamp.

[0156] Furthermore, the time synchronization module 730 is also used to: determine a differentiated time synchronization window based on the target image timestamp and the sampling frequency of the non-camera sensor.

[0157] Furthermore, the time synchronization module 730 is also used to: search for all data in the sensing data whose second global timestamp is within the differential time synchronization window; for each data in all data, calculate the time difference between the second global timestamp of the data and the timestamp of the target image, and determine the target data based on the time difference.

[0158] Furthermore, the determining module 720 is specifically used to: select a first global timestamp from the first global timestamp of each frame image as the target image timestamp.

[0159] Furthermore, the time stamp alignment of the first global timestamp of each frame is determined by the following method: determining whether the first global timestamp of each frame is within the image time synchronization window; if the first global timestamp of each frame is within the image time synchronization window, then the time stamp alignment of each frame is determined.

[0160] Furthermore, before acquiring multiple frames of images captured simultaneously by multiple cameras in the heterogeneous sensor, the time synchronization module of the heterogeneous sensor also includes a sending module (not shown). The sending module is used to send frame trigger pulse signals to the multiple cameras. The frame trigger pulse signals carry the trigger time and are used to trigger the multiple cameras to acquire images at the trigger time.

[0161] Furthermore, the first global timestamp is obtained in the following way: for each camera in the multi-camera setup, the first global timestamp is calculated based on the camera's exposure time and trigger time.

[0162] Furthermore, the time synchronization module 730 is also used to: acquire the reference time of the master clock, which is referenced to the time after the vehicle is powered on; broadcast a time synchronization signal to the heterogeneous sensor, which is used to control the heterogeneous sensor to generate a global timestamp of the heterogeneous sensor based on the local timestamp and time offset, and the global timestamp is synchronized with the master clock.

[0163] Furthermore, after synchronizing the sensing data, the time synchronization module of the heterogeneous sensor also includes a data processing module (not shown), which is used to package and output the time synchronization data corresponding to the heterogeneous sensor.

[0164] The time synchronization device for heterogeneous sensors provided in this embodiment of the invention can be used to execute the method steps of the above method embodiments. The specific implementation and technical effects are similar, and will not be repeated here.

[0165] This invention also provides a vehicle, including: a vehicle body and as described above. Figure 1 The time synchronization system of heterogeneous sensors shown.

[0166] For example, the vehicle could be an autonomous vehicle.

[0167] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0168] This invention also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0169] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0170] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0171] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0172] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0174] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0176] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0177] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0178] The above embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention.

Claims

1. A method for time synchronization of heterogeneous sensors, the method comprising: include: When the heterogeneous sensors are synchronized to the master clock of the vehicle domain controller, multiple frames of images captured by multiple cameras in the heterogeneous sensors at the same time are acquired, as well as sensing data collected by non-camera sensors in the heterogeneous sensors. Each frame of the multiple images is marked with a first global timestamp, which is the midpoint of the effective imaging time of the image. The sensing data is marked with a second global timestamp. If the first global timestamp of each frame image is already timestamp aligned, then the target image timestamp is determined based on the first global timestamp of each frame image, and the timestamp of each frame image is updated to the target image timestamp; If the first global timestamp of at least one frame is not timestamp aligned, then each frame is marked as an invalid image and time synchronization with non-camera sensors is not performed. Based on the target image timestamp, the sensor data is synchronized in time according to the second global timestamp; The step of synchronizing the sensing data based on the target image timestamp and the second global timestamp includes: Based on the timestamp of the target image, determine the differentiated time synchronization window corresponding to the non-camera sensor; Based on the second global timestamp, the target data corresponding to the differentiated time synchronization window is determined in the sensing data; The target data is interpolated to obtain the time synchronization data of the non-camera sensor at the timestamp of the target image.

2. The time synchronization method of a heterogeneous sensor according to claim 1, wherein, The step of determining the differentiated time synchronization window corresponding to the non-camera sensor based on the target image timestamp includes: Based on the timestamp of the target image, the differentiated time synchronization window is determined according to the sampling frequency of the non-camera sensor.

3. The time synchronization method for heterogeneous sensors according to claim 1, characterized in that, The step of determining the target data corresponding to the differentiated time synchronization window in the sensing data based on the second global timestamp includes: Search the sensor data for all data with the second global timestamp within the differential time synchronization window; For each piece of data, calculate the time difference between the second global timestamp of the data and the timestamp of the target image, and determine the target data based on the time difference.

4. The time synchronization method for heterogeneous sensors according to any one of claims 1 to 3, characterized in that, The step of determining the target image timestamp based on the first global timestamp of each frame includes: From the first global timestamp of each frame image, select one first global timestamp as the target image timestamp.

5. The time synchronization method for heterogeneous sensors according to any one of claims 1 to 3, characterized in that, The following method is used to determine whether the first global timestamp of each frame is timestamp aligned: Determine whether the first global timestamp of each frame is within the image time synchronization window; If the first global timestamp of each frame is within the image time synchronization window, then each frame is determined to be timestamp aligned.

6. The time synchronization method for heterogeneous sensors according to any one of claims 1 to 3, characterized in that, Before acquiring multiple frames of images captured simultaneously by multiple cameras in the heterogeneous sensor, the process further includes: A frame trigger pulse signal is sent to the multiple cameras. The frame trigger pulse signal carries a trigger time and is used to trigger the multiple cameras to acquire images at the trigger time.

7. The time synchronization method for heterogeneous sensors according to claim 6, characterized in that, The first global timestamp was obtained in the following way: For each of the multiple cameras, the first global timestamp is calculated based on the camera's exposure time and the trigger time.

8. The time synchronization method for heterogeneous sensors according to any one of claims 1 to 3, characterized in that, Synchronizing the time of heterogeneous sensors to the master clock of the vehicle domain controller includes: Obtain the reference time of the master clock, with the reference time being based on the time after the vehicle is powered on; A time synchronization signal is broadcast to the heterogeneous sensor. The time synchronization signal is used to control the heterogeneous sensor to generate a global timestamp of the heterogeneous sensor based on the local timestamp and time offset. The global timestamp is synchronized with the master clock.

9. The time synchronization method for heterogeneous sensors according to any one of claims 1 to 3, characterized in that, After synchronizing the sensing data in time, the method further includes: Package and output the time synchronization data corresponding to the heterogeneous sensors.

10. A time synchronization device for heterogeneous sensors, characterized in that, include: The acquisition module is used to acquire multiple frames of images collected by multiple cameras in the heterogeneous sensors at the same time, and sensing data collected by non-camera sensors in the heterogeneous sensors, when the heterogeneous sensors are synchronized to the master clock of the vehicle domain controller. Each frame of the multiple images is marked with a first global timestamp, which is the midpoint of the effective imaging time of the image. The sensing data is marked with a second global timestamp. The determination module is used to determine the target image timestamp based on the first global timestamp of each frame image when the first global timestamp of each frame image is aligned with the timestamp, and update the timestamp of each frame image to the target image timestamp; If the first global timestamp of at least one frame is not timestamp aligned, then each frame is marked as an invalid image and time synchronization with non-camera sensors is not performed. The time synchronization module is used to synchronize the sensing data in time based on the target image timestamp and the second global timestamp. The step of synchronizing the sensing data based on the target image timestamp and the second global timestamp includes: Based on the timestamp of the target image, determine the differentiated time synchronization window corresponding to the non-camera sensor; Based on the second global timestamp, the target data corresponding to the differentiated time synchronization window is determined in the sensing data; The target data is interpolated to obtain the time synchronization data of the non-camera sensor at the timestamp of the target image.

11. A time synchronization system for heterogeneous sensors, characterized in that, include: The vehicle domain controller and heterogeneous sensors are provided, wherein the vehicle domain controller is equipped with a time synchronization module and a main control module; The time synchronization module is used to synchronize the time of the heterogeneous sensor to the master clock; The main control module is used to execute the time synchronization method for heterogeneous sensors as described in any one of claims 1 to 9.

12. A vehicle, characterized in that, include: The vehicle body and the time synchronization system for the heterogeneous sensors as described in claim 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, include: A computer program that, when executed by a processor, implements the method of any one of claims 1 to 9.

Images