Data synchronization device and method thereof, computer program

Abstract

The present disclosure provides a data synchronization device and a method thereof, and a computer-readable storage medium. The data synchronization device includes a control unit and a camera, and an assembly integrated with the camera includes at least a synchronization relay unit and a sensing unit, and the sensing unit includes a first sensor and a second sensor. The control unit transmits a first control signal to the synchronization relay unit, and the synchronization relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal, and sets a timestamp for the first sensing data. The control unit receives the first sensing data and the second sensing data, and performs synchronous alignment of the first sensing data and the second sensing data based on the timestamp. The present disclosure solves the problem that synchronization of multiple sensing data cannot be realized in the prior art.

Description

Cross-reference to Related Applications

[0001] This application claims priority to a Chinese patent application with application number 202111350965.X and invention title "Data Synchronization Device and Its Method, Computer Readable Storage Medium", filed on November 15, 2021, and the entire content of the application is incorporated herein by reference.

Technical Field

[0002] The present disclosure relates to the field of data processing technology, specifically to a data synchronization device and its method, and a computer-readable storage medium.

Background Art

[0003] With the development of intelligent vehicles, autonomous driving technology has been widely applied to intelligent vehicles, and more and more vehicles are equipped with autonomous driving functions. Based on various vehicle-mounted sensors, various states of the vehicle itself and the surrounding environment are sensed, and driving policies are created accordingly. The realization of synchronization via sensors in data collection is very important for the autonomous driving function of intelligent vehicles.

[0004] In related technologies, a general data synchronization method only synchronizes the inertial sensor IMU and the camera, and triggers the synchronization between the IMU and the camera due to the interruption of the IMU itself. However, such a synchronization method has obvious drawbacks. Triggering synchronization by the IMU causes detachment from the control unit and cannot realize synchronization between the control unit and sensors other than the IMU and the camera. At the same time, such a synchronization method cannot synchronize multiple sensing data and image data on the camera side, and can only perform synchronization processing on the control unit side, which is likely to cause an increase in synchronization error and reduce the user experience.

[0005] No effective solution has been provided for the above problems.

Summary of the Invention

[0006] This disclosure provides a data synchronization device and method, and a computer-readable storage medium, which solve the problem of not being able to synchronize multiple sensing data in at least related technologies. [Means for solving the problem]

[0007] According to one aspect of the present disclosure, a data synchronization device is provided, comprising a control unit and a camera, wherein an assembly integrated with the camera comprises at least a synchronization relay unit and a sensing unit, the sensing unit comprising a first sensor and a second sensor, the control unit transmits a first control signal to the synchronization relay unit, the synchronization relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal, and sets a timestamp on the first sensing data, the control unit receives the first sensing data and the second sensing data, and performs synchronous alignment of the first sensing data and the second sensing data based on the timestamp.

[0008] Alternatively, the control unit may transmit a first control signal to the synchronous relay unit, and the camera may, after receiving an initialization signal transmitted from the control unit, start a self-check and initialization, and after the self-check is successful and initialization is completed, transmit an initialization success signal to the control unit; and the control unit may, after receiving the initialization success signal, transmit a first control signal to the synchronous relay unit.

[0009] Alternatively, the control unit includes an initial signal unit, a processing unit, and a deserializer, wherein the initial signal unit is configured to transmit an initial signal to the processing unit, the processing unit is configured to perform frequency division on the initial signal and generate a divided signal, and the deserializer is configured to generate the first control signal based on the divided signal.

[0010] Alternatively, the synchronous relay unit further includes a serializer configured to receive the first control signal transmitted from the deserializer in the control unit and to generate synchronous signals to control the first sensor and the second sensor, respectively.

[0011] Alternatively, the synchronous relay unit further includes a microcontroller configured to receive the synchronous signal transmitted from the serializer and generate a second control signal to synchronously control the first and second sensors.

[0012] Alternatively, if the microcontroller is not installed in the synchronous relay unit, the serializer and the first sensor are connected using a first bus, and if the microcontroller is installed in the synchronous relay unit, the first sensor and the microcontroller are connected using a second bus.

[0013] Alternatively, the system further includes a second sensing unit which includes at least one of a laser radar, millimeter-wave radar, and ultrasonic radar, and is configured to receive a frequency division signal generated by the processing unit within the control unit and to transmit third sensing data to the control unit under the control of the frequency division signal to synchronize the third sensing data with the data generated by the first sensing unit.

[0014] Alternatively, the first sensor may include at least one of an inertial sensor and a geomagnetometer, and the second sensor may be an image sensor.

[0015] According to one aspect of this disclosure, a data synchronization method is provided that is applied to a camera in an in-vehicle device, wherein the camera and a control unit are pre-connected, and the assembly integrated in the camera includes at least a synchronous relay unit and a sensing unit, wherein the sensing unit includes a first sensor and a second sensor, and the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on a first control signal transmitted from the control unit; sets a timestamp on the first sensing data; transmits the first sensing data and the second sensing data with the timestamp set to the control unit, and the control unit synchronously aligns the first sensing data and the second sensing data based on the timestamp.

[0016] Alternatively, the synchronous relay unit may further include the steps of receiving an initialization signal transmitted from the control unit before the step of synchronously controlling the first sensor to generate first sensing data and the second sensor to generate second sensing data based on a first control signal transmitted from the control unit; starting a self-check operation and an initialization operation based on the initialization signal; and transmitting an initialization success signal to the control unit if there are no failures in the self-check and initialization is completed.

[0017] Alternatively, the synchronous relay unit further includes a serializer, which further includes transmitting the first sensing data and the second sensing data to the control unit via the serializer.

[0018] Another aspect of this disclosure provides a computer-readable storage medium containing a stored computer program, wherein when the computer program is executed, the computer-readable storage medium controls the device on which the computer-readable storage medium resides to perform any one of the data synchronization methods described above.

[0019] In this disclosure, a data synchronization device includes a camera, which is an assembly comprising a control unit, at least a synchronous relay unit, and a sensing unit, wherein the sensing unit includes a first sensor and a second sensor; the control unit transmits a first control signal to the synchronous relay unit; the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal, and sets a timestamp on the first sensing data; the control unit receives the first sensing data and the second sensing data, and performs synchronous alignment of the first sensing data and the second sensing data based on the timestamp.

[0020] In this disclosure, a timestamp can be set on the first sensing data (e.g., inertial sensing data) via a synchronous relay unit in the camera, the first sensing data and the second sensing data (e.g., image data) can be synchronized, and the control unit can synchronize the two sensing data based on the timestamp, thereby reducing errors between multiple sensing data and improving the user experience, solving the problem of not being able to synchronize multiple sensing data in related technologies.

[0021] In the present disclosure, an inertial sensor IMU and an image sensor can be installed in the same camera. The vibration data of the camera can be collected in real time via the IMU. Further, through a synchronization relay unit on the camera, synchronous alignment of the image data and vibration data acquired by the image sensor is performed, and the synchronized image data and vibration data are transmitted to a remote control unit to achieve vibration prevention of the image. At the same time, a synchronization relay unit can be arranged on the camera, which has the advantage of high real-time performance and can reduce the synchronization error between the image data and the vibration data.

[0022] Furthermore, in the present disclosure, there are no requirements for the platform and computing power of the remote processing unit, and it is compatible with various platforms.

Brief Description of the Drawings

[0023] The drawings described herein are used to provide a further understanding of the present disclosure, constitute a part of this application, and the exemplary embodiments and their descriptions of the present disclosure are used to interpret the present disclosure and do not constitute an undue limitation to the present disclosure. [Figure 1] It is a schematic diagram of a data synchronization device having an alternative microcontroller according to the present disclosure. [Figure 2] It is a timing diagram of each signal in a data synchronization device having an alternative microcontroller according to the present disclosure. [Figure 3] It is a flowchart of an alternative data synchronization method according to the present disclosure. [Figure 4] It is a schematic diagram of another data synchronization device without an alternative microcontroller according to the present disclosure. [Figure 5] It is a timing diagram of each signal in a synchronization device without an alternative microcontroller according to the present disclosure. [Figure 6] It is a schematic diagram of another data synchronization device having an alternative dual camera according to the present disclosure. [Figure 7] It is a timing diagram of each signal in a synchronization device having an alternative dual camera according to the present disclosure. [Figure 8] It is a schematic diagram of a data synchronization device having another alternative radar according to the present disclosure. [Figure 9] It is a timing diagram of each signal in a synchronization device having an alternative radar according to the present disclosure.

Embodiments for Carrying Out the Invention

[0024] In order for those skilled in the art to more clearly understand the solution means of the present disclosure, the technical solution means in the embodiments of the present disclosure will be clearly and completely described below while referring to the drawings in the embodiments of the present disclosure. As an obvious point, the described embodiments are some embodiments of the present disclosure, not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative efforts all belong to the protection scope of the present disclosure.

[0025] Note that terms such as "first", "second", etc. in the specification, claims, and the above drawings of the present disclosure are set for distinguishing similar objects and are not set for explaining a specific order or the order of before and after. The data used is exchangeable in appropriate situations, so it should be understood that the embodiments of the present disclosure described in this specification can be implemented in an order other than that illustrated or described in this specification. Furthermore, the terms "including" and "having" and their synonyms are intended to be non-exclusive. For example, a process, method, system, product, or device including a series of steps or elements is not necessarily limited to the explicitly listed steps or elements, and may include steps or elements not explicitly listed or other steps or elements specific to the process, method, product, or device.

[0026] In order for those skilled in the art to more easily understand the present disclosure, some terms or nouns related to each embodiment of the present disclosure will be explained below.

[0027] An IMU (Inertial Measurement Unit) is a sensor mainly for detecting and measuring acceleration and rotational motion.

[0028] A Microcontroller Unit (MCU) is a chip-level computer that appropriately reduces the frequency and simplifies the specifications of a central processing unit (processing unit), while integrating peripheral interfaces such as memory, counters, USB, A / D converter, UART, PLC, DMA, and even an LCD driver circuit onto a single chip. It performs different combinations of control in various application situations, such as the electronic control of automobiles.

[0029] Each of the following embodiments of this disclosure can be applied to situations where multiple data need to be synchronized, for example, in an in-vehicle device having multiple sensors installed on various means of transportation, including but not limited to automobiles, buses, motorcycles, airplanes, trains, etc.

[0030] In this disclosure, an inertial sensor (IMU) and an image sensor can be installed in the same camera, camera vibration data can be collected in real time via the IMU, and further synchronization of image data and vibration data acquired by the image sensor can be performed via a synchronization relay unit on the camera, and the synchronized image data and vibration data can be transmitted to a remote control unit to prevent image vibration. At the same time, the synchronization relay unit can be placed on the camera, which has the advantage of high real-time performance, can reduce synchronization errors of image data and vibration data, and does not have requirements for the platform and computing power of the remote processing unit, and is compatible with various platforms.

[0031] (Example 1) Figure 1 is a schematic diagram of a data synchronization device having an alternative microcontroller according to the present disclosure, and as shown in Figure 1, the synchronization device is a data synchronization device including a control unit and a camera, wherein the assembly integrated into the camera includes at least a synchronization relay unit and a sensing unit, the synchronization relay unit includes a microcontroller and a serializer, the sensing unit includes a first sensor and a second sensor, and the control unit includes at least an initial signal unit, a processing unit and a deserializer.

[0032] In this disclosure, a control unit transmits a first control signal to a synchronous relay unit, which synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal, and sets a timestamp on the first sensing data. The control unit receives the first sensing data and the second sensing data and performs synchronous alignment of the first sensing data and the second sensing data based on the timestamp. The first sensor may include an inertial sensor, a geomagnetic sensor, etc., and this disclosure will describe in detail the case where the first sensor is an inertial sensor as an example.

[0033] In this disclosure, the synchronous relay unit receives data from the sensing unit (i.e., first sensing data and second sensing data), synchronizes the image data and IMU data (i.e., synchronously controls the first and second sensors based on the second control signal generated by the microcontroller), transmits it to the control unit using a low-voltage differential signal LVDS (i.e., transmits the synchronized data from the first and second sensors to the control unit), transmits the image to the processing unit of the control unit, and then performs image processing using an ISP module mounted on the processing unit. By adding an ISP module to the camera's sensing unit, the image can be processed. Image processing can be performed before transmitting the image data to the processing unit of the control unit. The control unit can generate a GPS reference clock signal (i.e., an initial signal) via an initial signal unit and provide it to the processing unit. The processing unit performs frequency division based on the GPS reference clock signal to obtain clock frequency division information (i.e., a frequency division signal). Subsequently, the deserializer generates a first control signal based on the frequency division signal and transmits the first control signal to the serializer. The serializer generates a synchronization signal based on the first control signal and transmits it to the microcontroller. The microcontroller can then generate a second control signal based on the synchronization signal.

[0034] In this disclosure, after receiving first and second sensing data with timestamps set, the control unit can synchronize the first and second sensing data based on the timestamps, and if the second sensor is an image sensor, the timestamps are set according to the exposure start time of each frame image in the second sensing data (i.e., image data).

[0035] The specific workflow is as follows:

[0036] 10. The control unit and camera are powered on, and the control unit begins a self-check and initializes the camera.

[0037] 20. After receiving the initialization signal from the processing unit, the camera starts a self-check and initialization. Upon completion, the microcontroller (MCU) sends an initialization success signal via the I2C bus and notifies the remote processing unit via the serializer and deserializer.

[0038] 30. The processing unit receives an initialization success signal transmitted from the MCU, provides a GPS reference clock signal (i.e., an initial signal) via the initial signal unit, transmits a first control signal via the deserializer after frequency division, and connects to the MCU via the serializer.

[0039] 40. After the MCU receives the first control signal via the serializer, it converts it into a second control signal and sends it to the first sensor in the sensing unit. At the same time, it starts adding a timestamp to the IMU data (i.e., the first sensing data) and returns it to the processing unit via the serializer and deserializer.

[0040] 50. After the second sensor in the sensing unit receives the second control signal, it starts collecting image data (i.e., second sensing data). This image data is transmitted to the processing unit via a serializer and a deserializer. The processing unit receives the image data of this frame, as well as IMU data with a timestamp, and then synchronizes the image data with the IMU data.

[0041] In this disclosure, Figure 2 is a timing diagram of each signal in a data synchronization device having an alternative microcontroller according to this disclosure, and includes an initial signal, a frequency divider signal, a first control signal, first sensing data (unsynchronized), first sensing data (synchronized and aligned), and second sensing data, which align the transmitted second sensing data with the first sensing data.

[0042] Alternatively, the process may include the steps of: before the control unit transmits a first control signal to the synchronous relay unit, the camera receives an initialization signal from the control unit, starts a self-check and initialization, and after the self-check is successful and initialization is complete, transmits an initialization success signal to the control unit; and after the control unit receives the initialization success signal, transmits a first control signal to the synchronous relay unit.

[0043] In this disclosure, first the power to the control unit and the camera must be turned on, then the control unit starts a self-check and initializes the camera (i.e., after the control unit performs a self-check, it sends an initialization signal to the camera), the camera starts a self-check and initialization after receiving the initialization signal, and after the camera's self-check is successful and initialization is complete, if the synchronous relay unit includes a microcontroller MCU, the MCU can send an initialization success signal (i.e., an indication that the camera's self-check was successful and initialization is complete) via various types of buses (e.g., an I2C bus), and notify a remote processing unit (i.e., the processing unit of the control unit) via a serializer and deserializer, and after the processing unit receives the initialization success signal sent from the MCU, it sends a first control signal to the synchronous relay unit.

[0044] Alternatively, the control unit includes an initial signal unit, a processing unit, and a deserializer, wherein the initial signal unit is configured to transmit an initial signal to the processing unit, the processing unit is configured to perform frequency division on the initial signal and generate a divided signal, and the deserializer is configured to generate a first control signal based on the divided signal.

[0045] In this disclosure, after the processing unit receives an initialization success signal transmitted from the MCU, the processing unit performs a frequency division operation using a GPS reference clock signal provided by the initial signal unit (i.e., the initial signal unit is configured to transmit an initial signal to the processing unit) (the processing unit is configured to perform a frequency division operation on the initial signal) to obtain clock frequency division information (i.e., a frequency division signal), and the deserializer is configured to generate a first control signal based on the frequency division signal.

[0046] Alternatively, the synchronous relay unit further includes a serializer configured to receive a first control signal transmitted from a deserializer in the control unit and to generate synchronous signals to control the first and second sensors, respectively.

[0047] In this disclosure, the control unit performs frequency division processing and then transmits a first control signal via a deserializer, and the serializer generates a synchronization signal based on the first control signal to control the first sensor and the second sensor, respectively.

[0048] Alternatively, the synchronous relay unit further includes a microcontroller configured to receive a synchronous signal transmitted from the serializer and generate a second control signal to synchronously control the first and second sensors.

[0049] In this disclosure, the serializer generates a synchronization signal based on a first control signal and transmits the synchronization signal to a microcontroller, and the microcontroller generates a second control signal based on the synchronization signal and synchronously controls the first sensor and the second sensor.

[0050] Alternatively, if a microcontroller is not installed in the synchronous relay unit, the serializer and the first sensor are connected using the first bus; and if a microcontroller is installed in the synchronous relay unit, the first sensor and the microcontroller are connected using the second bus.

[0051] In this disclosure, if a microcontroller is not installed in the synchronous relay unit, the inertial sensor (i.e., the first sensor) and the serializer are directly connected using a first bus (e.g., an I2C bus). If the synchronous relay unit assembly also includes a microcontroller (i.e., an MCU) in addition to the serializer, the inertial sensor and the microcontroller are directly connected using a second bus.

[0052] Alternatively, the data synchronization device is configured to receive a frequency division signal generated by a processing unit within the control unit and to transmit third sensing data to the control unit under the control of the frequency division signal, thereby aligning the third sensing data with the data generated by the first sensing unit, and further includes a second sensing unit which includes at least one of a laser radar, millimeter-wave radar, or ultrasonic radar.

[0053] In this disclosure, the data synchronization device is provided as an example to include a laser radar and / or millimeter-wave radar. A radar module (i.e., a second sensing unit) including at least a laser radar and / or millimeter-wave radar is connected to a control unit, which can transmit a laser radar synchronization signal and / or millimeter-wave synchronization signal after performing a frequency division operation. After the radar module receives the laser radar synchronization signal and / or millimeter-wave synchronization signal (i.e., after receiving the frequency division signal generated by a processing unit within the control unit), it can timestamp the laser radar data and / or millimeter-wave radar data and return it to the processing unit (i.e., transmit third sensing data to the control unit under the control of the frequency division signal). The processing unit receives the image data, and after also receiving timestamped IMU data (i.e., data generated by the first sensing unit including image data and sensing data) and timestamped radar data (i.e., third sensing data, which includes laser radar data and / or millimeter-wave radar data), it can synchronize the image data, IMU data and radar data.

[0054] Alternatively, the first sensor may include inertial sensors, geomagnetic sensors, etc., while the second sensor may be an image sensor such as a standard RGB image sensor, depth image sensor, or thermal image sensor.

[0055] In this disclosure, inertial sensors and image sensors can be installed in the same camera, synchronization is initiated by a synchronization relay unit within the camera, and the camera synchronizes sensing data and image data, enabling accurate synchronization time and high platform compatibility. Furthermore, this disclosure allows for the addition of an MCU to the camera to synchronize multiple sensors, resulting in high real-time performance of MCU control and reduced synchronization errors between sensing data and image data. Additionally, because the MCU is located in the camera, there are no platform and computing power requirements for the remote processing unit, resulting in high compatibility with various platforms.

[0056] (Example 2) The present disclosure provides a data synchronization method, and it should be noted that the steps shown in the flowchart of the drawings can be executed, for example, in a computer system of a set of computer executable commands, and although the flowchart shows a logical order, in some cases the steps shown or described can be executed in a different order.

[0057] Figure 3 is a flowchart of an alternative data synchronization method relating to this disclosure, and as shown in Figure 3, the method is The synchronous relay unit performs step S302, which synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on a first control signal transmitted from the control unit, Step S304 involves setting a timestamp for the first sensing data, The process includes step S306, which involves transmitting first sensing data and second sensing data with timestamps set to a control unit, and the control unit performing synchronization alignment of the first sensing data and second sensing data based on the timestamps.

[0058] In the steps described above, the synchronous relay unit synchronizes the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal transmitted from the control unit, sets a timestamp on the first sensing data, and transmits the first and second sensing data with the timestamp set to the control unit, which then synchronizes the first and second sensing data based on the timestamp. In this disclosure, sensing data and image data can be synchronized via a synchronous relay unit in the camera, a timestamp can be set on the sensing data, and the sensing data and image data with the timestamp set can be transmitted to the control unit. This enables alignment processing between sensing data and image data, reduces errors between multiple sensing data, improves the user experience, and solves the problem of not being able to synchronize multiple sensing data in related technologies.

[0059] The present disclosure is described in detail below, with reference to each of the steps described above.

[0060] In this disclosure, each of the following steps applies to a camera in an in-vehicle device installed in various modes of transport having multiple sensors, the camera and the control unit are pre-connected, the assembly integrated into the camera includes at least a synchronous relay unit and a sensing unit, the sensing unit includes a first sensor and a second sensor, the synchronous relay unit includes at least a serializer and further includes a microcontroller, the control unit includes a deserializer, an initial signal unit and a processing unit, the first sensor may include an inertial sensor, a geomagnetometer, etc., and this disclosure will describe in detail using the example that the first sensor is an inertial sensor.

[0061] In this disclosure, prior to the step in which the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on a first control signal transmitted from the control unit, the data synchronization method further includes the steps of receiving an initialization signal transmitted from the control unit, starting a self-check operation and an initialization operation based on the initialization signal, and transmitting an initialization success signal to the control unit if there are no failures in the self-check and initialization is completed.

[0062] In this disclosure, first the control unit and camera must be powered on, then the control unit starts a self-check and initializes the camera (i.e., after the control unit performs a self-check, it sends an initialization signal to the camera), the camera starts a self-check and initialization after receiving the initialization signal, and after the camera's self-check is successful and initialization is complete, if the synchronous relay unit includes an MCU, the MCU can send an initialization success signal (i.e., an indication that the camera's self-check was successful and initialization is complete) via various types of buses (e.g., an I2C bus) and notify a remote processing unit via a serializer and deserializer (i.e., it sends an initialization success signal to the control unit). If the synchronous relay unit does not include an MCU, the serializer sends an initialization success signal via a certain type of bus (e.g., an I2C bus) and notifies a remote processing unit (it sends an initialization success signal to the control unit).

[0063] In step S302, the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal transmitted from the control unit.

[0064] In this disclosure, the control unit performs frequency division and then transmits a first control signal via a deserializer. If the synchronous relay unit includes an MCU, it can be connected to the MCU via the serializer, and the MCU receives the first control signal and generates a second control signal to synchronize the first sensor to generate first sensing data and the second sensor to generate second sensing data. If the synchronous relay unit does not include an MCU, it can be connected to a sensing unit via the serializer, and the sensing unit receives the first control signal and synchronizes the first sensor to generate first sensing data and the second sensor to generate second sensing data.

[0065] In step S304, a timestamp is set for the first sensing data.

[0066] In this disclosure, if the synchronous relay unit includes an MCU, a second control signal generated by the MCU can be used to synchronously control the first sensor to generate first sensing data and the second sensor to generate second sensing data, and simultaneously, the application of a timestamp to the IMU data (i.e., the first sensing data) is initiated. If the synchronous relay unit does not include an MCU, a first control signal can be used to synchronously control the first sensor to generate first sensing data and the second sensor to generate second sensing data, and simultaneously, a timestamp is applied to the IMU data received at this point.

[0067] In step S306, the first sensing data and the second sensing data, each with a timestamp, are transmitted to the control unit, and the control unit synchronizes the first sensing data and the second sensing data based on the timestamp.

[0068] In this disclosure, after synchronizing the image data and sensing data (i.e., after setting a timestamp on the first sensing data), the first and second sensing data, with timestamps set by low-voltage differential signals (LVDS), can be transmitted to a control unit. After receiving the first and second sensing data, the control unit can synchronize the first and second sensing data based on the timestamps. If the second sensor is an image sensor, the timestamps are set according to the exposure start time of each frame image in the second sensing data (i.e., image data).

[0069] Alternatively, the synchronous relay unit further includes a serializer, and the data synchronization method further includes the step of transmitting first sensing data and second sensing data to a control unit via the serializer.

[0070] In this disclosure, aligned sensing data and image data can be transmitted to a deserializer of an in-vehicle device control unit via a serializer of a synchronous relay unit, and then transmitted to a processing unit by the deserializer.

[0071] In this disclosure, inertial sensors and image sensors can be installed in the same camera, synchronization can be initiated by a synchronization relay unit within the camera, the camera can synchronize sensing data and image data, camera vibration data can be collected in real time, and image vibration prevention can be set. Furthermore, this disclosure allows for the addition of an MCU to the camera to synchronize multiple sensors, the real-time capability of MCU control is high, the synchronization error between multiple sensors and image frames can be reduced, and because the MCU is located in the camera, there are no requirements for the platform and computing power of the remote processing unit, and it is compatible with various platforms.

[0072] (Example 3) Figure 4 is a schematic diagram of another alternative data synchronization device without a microcontroller as described herein, which, as shown in Figure 4, includes a control unit and a camera, the assembly integrated into the camera includes at least a synchronization relay unit and a sensing unit, the synchronization relay unit includes at least a serializer, the sensing unit includes a first sensor and a second sensor, the first sensor may include an inertial sensor, a geomagnetometer, etc., and this disclosure describes in detail, using the example that the first sensor is an inertial sensor, and the control unit includes at least an initial signal unit, a processing unit and a deserializer.

[0073] In this disclosure, the sensing unit receives the image data signal to be transmitted, the IMU data signal to be transmitted, the frame synchronization signal to turn on exposure, and the power supply, etc. These signals are connected to the synchronization relay unit via an FPC cable, the synchronization relay unit receives the image data and IMU data and transmits them to the control unit via low voltage differential signaling (LVDS), the image data is transmitted to the processing unit of the control unit, and the processing unit can synchronize the image data and IMU data, and the image processing of the image data can be performed by an ISP mounted on the processing unit, and by adding an ISP to the camera's sensing unit, image processing can be performed before the image data is transmitted to the processing unit of the control unit, and the control unit can generate a GPS reference clock signal via an initial signal unit and provide it to the processing unit, and the specific operation flow is as follows: 11. The control unit and camera are powered on, and the control unit begins a self-check and initializes the camera.

[0074] 21. After receiving the initialization signal from the processing unit, the camera starts a self-check and initialization. Upon completion, the serializer sends an initialization success signal via the I2C bus to notify the remote processing unit. Simultaneously, data from the microcontroller (MCU) is transmitted to the processing unit via the I2C bus, serializer, and deserializer.

[0075] 31. The processing unit provides a GPS reference clock signal (i.e., an initial signal) via the initial signal unit, and after frequency division, the processing unit transmits a first control signal via the deserializer, which is converted into a synchronization signal via the serializer and then connected to a first sensor in the sensing unit, while simultaneously adding a timestamp to the IMU data (i.e., first sensing data) received at this point.

[0076] 41. After receiving the synchronization signal transmitted from the serializer, the sensing unit begins collecting image data (i.e., second sensing data). This image data is transmitted to the processing unit via the serializer and deserializer. The processing unit receives the image data of this frame, stores the IMU data which already has a timestamp, and then performs synchronization alignment between the image data and the IMU data.

[0077] In this disclosure, Figure 5 is a timing diagram of the signals in a synchronization device that does not include an alternative microcontroller as described herein, and includes an initial signal, a frequency divider signal, second sensing data, and first sensing data (unsynchronized), which then align the transmitted second sensing data with the first sensing data.

[0078] In this disclosure, inertial sensors and image sensors can be placed together, and by having the processing unit initiate synchronization and synchronizing multiple sensing data on the camera side, the synchronization time can be made accurate, and platform compatibility is high.

[0079] (Example 4) Figure 6 is a schematic diagram of another alternative dual-camera data synchronization device according to the present disclosure, which, as shown in Figure 6, includes a control unit, camera A, and camera B, where camera A includes a sensing unit A and a synchronization relay unit A, camera B includes a sensing unit B and a synchronization relay unit B, the sensing unit includes a first sensor and a second sensor, the synchronization relay unit includes a microcontroller and a serializer, the control unit includes a deserializer, an initial signal unit and a processing unit, the first sensor may include an inertial sensor, a geomagnetometer, etc., and the present disclosure will describe in detail the case where the first sensor is an inertial sensor as an example.

[0080] In this disclosure, the sensing unit receives the image data signal to be transmitted, the IMU data signal to be transmitted, the frame synchronization signal to turn on exposure, and the power supply, and these signals are connected to the synchronization relay unit via an FPC cable. The synchronization relay unit receives the image data and IMU data and transmits them to the control unit via low voltage differential signaling (LVDS). After the image data is transmitted to the processing unit of the control unit, the processing unit can synchronize the image data with the IMU data GPS reference clock signal. Image processing of the image data can be performed by an ISP mounted on the processing unit. By adding an ISP to the camera's sensing unit, image processing can also be performed before the image data is transmitted to the processing unit of the control unit. The control unit can generate a GPS reference clock signal via an initial signal unit and provide it to the processing unit. The specific operation flow is as follows.

[0081] 12. The power to the control unit and cameras A and B is turned on, and the control unit starts a self-check and initializes cameras A and B.

[0082] 22. After receiving the initialization signal from the processing unit, cameras A and B begin self-checking and initialization. Upon completion, the microcontroller (MCU) sends an initialization success signal via the I2C bus and notifies the remote processing unit via the serializer and deserializer.

[0083] 32. The processing unit receives the initialization success signal transmitted from the MCU, and the processing unit provides the GPS reference clock signal (i.e., the initial signal) via the initial signal unit, which is then divided and converted into a first control signal via the deserializer, and received by the MCUs of cameras A and B via the serializer.

[0084] 42. The two MCUs simultaneously receive the first control signal via the serializer, then convert it into a second control signal and transmit it to the first sensor in each of the two sensing units. At the same time, they begin adding timestamps to their respective IMU data (i.e., the first sensing data) and return it to the processing unit via the serializer and deserializer.

[0085] 52. After the second sensor in the two sensing units receives the second control signal, it starts collecting image data (i.e., second sensing data). This image data is transmitted to the processing unit via a serializer and a deserializer. The processing unit receives the image data from the two sensing units, as well as the IMU data with a timestamp. The processing unit then synchronizes the image data from cameras A and B with the IMU data.

[0086] In this disclosure, Figure 7 is a timing diagram of the signals in an alternative dual-camera synchronization device relating to this disclosure, which includes an initial signal, a frequency divider signal, a first control signal (A), a first control signal (B), a second sensing data (A), a first sensing data (A) (unsynchronized), a second sensing data (B), and a first sensing data (B) (unsynchronized), and these signals align the transmitted second sensing data with the first sensing data.

[0087] In this disclosure, the synchronization device can have multiple cameras and can synchronize multiple sensing data within multiple cameras, requiring only one control unit. This disclosure uses two cameras as an example, in which an inertial sensor and an image sensor can be placed together in each camera, synchronization is initiated by the processing unit, and the cameras themselves synchronize multiple sensing data, ensuring accurate synchronization time and high platform compatibility. Furthermore, this disclosure allows for the addition of an MCU to the camera to synchronize multiple sensors, resulting in high real-time performance of MCU control and reduced synchronization errors between multiple sensing data. Additionally, because the MCU is located in the camera, there are no platform and computing power requirements for the remote processing unit, resulting in high compatibility with various platforms.

[0088] (Example 5) Figure 8 is a schematic diagram of a data synchronization device having another alternative radar according to the present disclosure, as shown in Figure 8, the synchronization device includes a control unit, a second sensing unit, and a camera, the camera includes a sensing unit and a synchronization relay unit, the sensing unit includes a first sensor and a second sensor, the first sensor may include an inertial sensor, a geomagnetometer, etc. (the present disclosure will describe in detail an example in which the first sensor is an inertial sensor), the synchronization relay unit includes a microcontroller and a serializer, the control unit includes a deserializer, an initial signal unit and a processing unit, and the second sensing unit includes at least a laser radar and a millimeter-wave radar (the second sensing unit may further include an ultrasonic radar, etc., the present disclosure will describe in detail an example in which the data synchronization device includes a laser radar and / or a millimeter-wave radar).

[0089] In this disclosure, the sensing unit receives the image data signal to be transmitted, the IMU data signal to be transmitted, the frame synchronization signal to turn on exposure, and the power supply, etc. These signals are connected to the synchronization relay unit via an FPC cable, the synchronization relay unit receives the image data and IMU data and transmits them to the control unit via low voltage differential signaling (LVDS), the image data is transmitted to the processing unit of the control unit, and the processing unit can synchronize the image data with the IMU data GPS reference clock signal, and the image processing of the image data can be performed by an ISP mounted on the processing unit, and by adding an ISP to the camera's sensing unit, image processing can be performed before the image data is transmitted to the processing unit of the control unit, the control unit can generate a GPS reference clock signal via an initial signal unit and provide it to the processing unit, the processing unit of the control unit outputs a laser radar synchronization signal and a millimeter wave synchronization signal to synchronize the laser radar and millimeter wave radar data, and the specific operation flow is as follows.

[0090] 13. Power is turned on to the control unit, camera, and radar (i.e., the second sensing unit). The control unit begins a self-check and initializes the camera and radar.

[0091] 23. After receiving the initialization signal from the processing unit, the camera starts a self-check and initialization. Upon completion, the microcontroller (MCU) sends an initialization success signal via the I2C bus and notifies the remote processing unit via the serializer and deserializer.

[0092] 33. The processing unit receives an initialization success signal transmitted from the MCU, and the processing unit provides a GPS reference clock signal (i.e., an initial signal) via an initial signal unit, which is then divided and converted into a first control signal via a deserializer, received by the camera's MCU via a serializer, and simultaneously transmits radar synchronization signals (including laser radar synchronization signals and millimeter-wave synchronization signals).

[0093] 43. After receiving the first control signal via the serializer, the MCU converts it into a second control signal and transmits it to the first sensor in the sensing unit. At the same time, it begins adding a timestamp to the IMU data (i.e., the first sensing data) and returns it to the processing unit via the serializer and deserializer.

[0094] 53. After the second sensor in the sensing unit receives the second control signal, the data is transmitted to the processing unit via a serializer and deserializer. The processing unit receives the image data (second sensing data) from the sensing unit, as well as time-stamped IMU data. Simultaneously, it also receives time-stamped laser radar and millimeter-wave radar data (third sensing data). Finally, it synchronizes the camera image data, IMU data, laser radar data, and millimeter-wave radar data.

[0095] In this disclosure, Figure 9 is a timing diagram of each signal in a synchronization device having an alternative radar according to this disclosure, and includes an initial signal, a frequency divider signal, a first control signal, second sensing data, first sensing data (unsynchronized), a laser radar synchronization signal, laser radar data, a millimeter-wave synchronization signal, and millimeter-wave radar data, which align multiple transmitted sensing data (including image data, inertial sensing data, laser radar data, and millimeter-wave radar data).

[0096] In this disclosure, inertial sensors can be placed together, synchronization is initiated by a processing unit, and multiple sensing data is synchronized on the camera side. Furthermore, it can support multiple types of sensors such as laser radar, millimeter-wave radar, and ultrasonic radar, ensuring accurate synchronization time and high platform compatibility. In addition, this disclosure allows for the addition of an MCU to the camera to synchronize multiple sensors, resulting in high real-time performance of MCU control and reduced synchronization errors for multiple sensing data. Moreover, because the MCU is placed in the camera, there are no platform and computing power requirements for the remote processing unit, resulting in high compatibility with various platforms.

[0097] In another aspect of this disclosure, we further provide a computer-readable storage medium that includes a stored computer program, and when the computer program is executed, controls a device on which the computer-readable storage medium resides to perform the data synchronization method for the in-vehicle device described in any one of the above paragraphs.

[0098] The example numbers in this disclosure are for illustrative purposes only and do not indicate any superiority or inferiority among the examples.

[0099] In the embodiments described herein, each description focuses on specific points, and information not detailed in one embodiment can be found in the descriptions of other embodiments.

[0100] In some embodiments provided herein, the disclosed technical content can, of course, be implemented in other ways. Herein, the embodiments of the apparatus described herein are merely illustrative, for example, the division of the units is merely a division of logical functions, and other division methods may be used in actual implementation, for example, multiple units or assemblies may be integrated or combined into another system, or certain features may be ignored or not implemented. In other words, the mutual coupling or direct coupling or communication connection illustrated or discussed herein may be an indirect coupling or communication connection via an interface, unit or module, and may be in an electrical or other form.

[0101] Furthermore, the units described as separations may or may not be physically separated, and the parts shown as units may be physical units, not physical units, located in the same place, or arranged in multiple units. Some or all of these units may be selected as required in order to achieve the objectives of the solutions of this disclosure.

[0102] Furthermore, each functional unit in the various embodiments of this disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The integrated units may be implemented in the form of either hardware or software functional units.

[0103] The integrated units described above may be implemented as software function units and, if sold or used as standalone products, may be stored on a computer-readable storage medium. Based on this understanding, a substantial portion of the technical solution of this disclosure that contributes to the prior art, or part or all of the technical solution, can be implemented in the form of a software product. The computer software product is stored on a single storage medium and includes several commands that execute all or part of the steps described in the method of each embodiment of this disclosure by a single computer device (which may be a personal computer, server, or network device, etc.). Examples of such storage mediums include USB memory, ROM (Read-Only Memory), RAM (Random Access Memory), portable hard disks, disks, and CD-ROMs, which are media capable of storing program code.

[0104] The above describes only preferred embodiments of the present disclosure, and those skilled in the art can make several further improvements and modifications without departing from the principles of the present invention, and such improvements and modifications should also be considered within the scope of the protection of the present invention.

Claims

1. A data synchronization device comprising a control unit and a camera, wherein an assembly integrated into the camera comprises at least a synchronization relay unit and a sensing unit, the sensing unit comprising a first sensor and a second sensor, the first sensor comprising at least one of an inertial sensor and a geomagnetometer, and the second sensor being an image sensor. The control unit transmits a first signal to the synchronous relay unit. The synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal, and sets a timestamp in the first sensing data. The control unit is a data synchronization device that receives the first sensing data and the second sensing data and synchronizes the first sensing data and the second sensing data based on the timestamp.

2. Before the control unit transmits the first signal to the synchronous relay unit, The camera, after receiving an initialization signal transmitted from the control unit, starts a self-check and initialization, and after the self-check is successful and initialization is complete, transmits an initialization success signal to the control unit. The data synchronization device according to claim 1, further comprising the step of the control unit transmitting a first control signal to the synchronization relay unit after receiving the initialization success signal.

3. The control unit includes an initial signal unit, a processing unit, and a deserializer. The initial signal unit is configured to transmit an initial signal to the processing unit. The processing unit is configured to perform frequency division on the initial signal and generate a divided signal. The data synchronization device according to claim 1, wherein the deserializer is configured to generate the first control signal based on the frequency division signal.

4. The data synchronization device according to claim 3, wherein the synchronization relay unit further includes a serializer configured to receive the first control signal transmitted from the deserializer in the control unit and to generate synchronization signals for controlling the first sensor and the second sensor, respectively.

5. The data synchronization device according to claim 4, further comprising a microcontroller configured to receive the synchronization signal transmitted from the serializer and generate a second control signal to synchronize the first sensor and the second sensor.

6. The data synchronization device according to claim 5, wherein if the microcontroller is not installed in the synchronous relay unit, the serializer and the first sensor are connected using a first bus, and if the microcontroller is installed in the synchronous relay unit, the first sensor and the microcontroller are connected using a second bus.

7. The data synchronization device according to claim 3, further comprising a second sensing unit configured to receive a frequency division signal generated by the processing unit within the control unit and to transmit third sensing data to the control unit under the control of the frequency division signal, wherein the second sensing unit includes at least one of a laser radar, a millimeter-wave radar, and an ultrasonic radar.

8. A data synchronization method applied to a camera in an in-vehicle device, wherein the camera and the control unit are pre-connected, and the assembly integrated into the camera includes at least a synchronization relay unit and a sensing unit, the sensing unit includes a first sensor and a second sensor, the first sensor includes at least one of an inertial sensor and a geomagnetometer, and the second sensor is an image sensor. The synchronous relay unit performs the steps of synchronously controlling the first sensor to generate first sensing data and the second sensor to generate second sensing data based on a first control signal transmitted from the control unit, The steps include setting a timestamp on the first sensing data, The first sensing data and the second sensing data, each with a timestamp set, are transmitted to the control unit, and the control unit performs synchronization alignment of the first sensing data and the second sensing data based on the timestamp. A data synchronization method that includes this.

9. Before the step in which the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal transmitted from the control unit, The steps include receiving an initialization signal transmitted from the control unit, The steps include: starting a self-check operation and an initialization operation based on the initialization signal; If no malfunctions are found during the self-check and initialization is completed, the process includes sending an initialization success signal to the control unit. The data synchronization method according to claim 8, further comprising:

10. The aforementioned synchronous relay unit further includes a serializer, The data synchronization method according to claim 8, further comprising the step of transmitting the first sensing data and the second sensing data to the control unit via the serializer.

11. A computer program that causes a data synchronization method according to any one of claims 8 to 10 to be executed.

Images