Data synchronization device and method, and computer program
The data synchronization device synchronizes multiple sensing data in intelligent vehicles by using a control unit and camera with integrated relay units, addressing synchronization errors and enhancing user experience through accurate real-time alignment.
Patent Information
- Application Number
- JP2024527861
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-15
- Filing Date
- 2022-11-10
- Publication Date
- 2024-12-05
- Estimated Expiration
- 2042-11-10
AI Technical Summary
Existing data synchronization methods for intelligent vehicles fail to synchronize multiple sensing data, particularly between inertial sensors and cameras, leading to synchronization errors and degraded user experience.
A data synchronization device and method that includes a control unit and a camera with integrated synchronization relay units and sensing units, allowing for synchronized alignment of multiple sensing data using time stamps, and optionally incorporating microcontrollers for enhanced control.
The solution enables accurate synchronization of multiple sensing data, reducing errors and improving user experience by synchronizing inertial and image data in real-time without requiring additional platform or computing power, and is compatible with various platforms.
Smart Images

Figure 2024544974000001_ABST
Abstract
Description
CROSS-REFERENCE TO RELATED APPLICATIONS
[0001] This application claims priority to a Chinese patent application filed on November 15, 2021, with application number 202111350965.X, entitled "Data synchronization device and method, and computer-readable storage medium," the entire contents of which are incorporated herein by reference. [Technical field]
[0002] The present disclosure relates to the field of data processing technology, and in particular to a data synchronization device and method, and a computer-readable storage medium. [Background technology]
[0003] With the development of intelligent automobiles, autonomous driving technology has been widely applied to intelligent automobiles, and more and more automobiles are equipped with autonomous driving functions. Based on various on-board sensors, the vehicle itself and the various conditions of the surrounding environment are sensed, and driving policies are created accordingly. The realization of synchronization through sensors in data collection is very important for the autonomous driving function of intelligent automobiles.
[0004] In the related art, a common data synchronization method is to synchronize the sensor only with the inertial sensor IMU and the camera, and to trigger the synchronization between the IMU and the camera by interrupting the IMU itself, but this synchronization method has obvious drawbacks. Triggering the synchronization by the IMU will cause a disconnection from the control unit, and the control unit cannot achieve synchronization with sensors other than the IMU and the camera. At the same time, this synchronization method cannot synchronize multiple sensing data and image data on the camera side, and can only perform synchronization processing on the control unit side, which is prone to increasing synchronization errors and degrading the user experience.
[0005] No effective solution to the above problem has been provided. Summary of the Invention [Problem to be solved by the invention]
[0006] The present disclosure provides a data synchronization device, a method thereof, and a computer-readable storage medium that solves the problem that synchronization of multiple sensing data cannot be realized in at least the related art. [Means for solving the problem]
[0007] According to one aspect of the present disclosure, there is provided a data synchronization device including a control unit and a camera, wherein an assembly integrated in the camera includes at least a synchronous relay unit and a sensing unit, the sensing unit including a first sensor and a second sensor, the control unit transmits a first control signal to the synchronous relay unit, the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal, and sets a timestamp to the first sensing data, the control unit receives the first sensing data and the second sensing data, and synchronously aligns the first sensing data and the second sensing data based on the timestamp.
[0008] Alternatively, before the control unit sends a first control signal to the synchronization relay unit, the camera starts a self-check and initialization after receiving an initialization signal sent from the control unit, and after the self-check finds no faults and the initialization is completed, sends an initialization success signal to the control unit; and after the control unit receives the initialization success signal, sends a first control signal to the synchronization relay unit.
[0009] Alternatively, the control unit includes an initial signal unit, a processing unit, and a deserializer, the initial signal unit configured to send an initial signal to the processing unit, the processing unit configured to perform a division process on the initial signal and generate a divided signal, and the deserializer configured to generate the first control signal based on the divided signal.
[0010] Alternatively, the synchronization relay unit further includes a serializer configured to receive the first control signal transmitted from a deserializer in the control unit and to generate synchronization signals for controlling the first sensor and the second sensor, respectively.
[0011] Alternatively, the synchronization relay unit further includes a microcontroller configured to receive the synchronization signal transmitted from the serializer and generate a second control signal to synchronously control the first sensor and the second sensor.
[0012] Alternatively, when the microcontroller is not installed in the synchronous relay unit, the serializer and the first sensor are connected using a first bus, and when the microcontroller is installed in the synchronous relay unit, the first sensor and the microcontroller are connected using a second bus.
[0013] Alternatively, the system further includes a second sensing unit configured to receive a frequency-divided signal generated by the processing unit in the control unit and transmit third sensing data to the control unit under control of the frequency-divided signal to synchronously align the third sensing data with the data generated by the first sensing unit, the second sensing unit including at least one of a laser radar, a millimeter wave radar, and an ultrasonic radar.
[0014] Alternatively, the first sensor includes at least one of an inertial sensor, a geomagnetometer, and the second sensor is an image sensor.
[0015] According to one aspect of the present disclosure, there is provided a data synchronization method that is applied to a camera of an in-vehicle device, in which a connection is established between the camera and a control unit in advance, an assembly integrated in the camera includes at least a synchronization relay unit and a sensing unit, and the sensing unit includes a first sensor and a second sensor, the data synchronization method including: a step of the synchronization relay unit performing synchronization control so that the first sensor generates first sensing data and the second sensor generates second sensing data based on a first control signal transmitted from the control unit; a step of setting a timestamp to the first sensing data; and a step of transmitting the first sensing data and the second sensing data with the timestamp set to the control unit, and the control unit performing synchronization alignment of the first sensing data and the second sensing data based on the timestamp.
[0016] Alternatively, before the step of the synchronous relay unit synchronously controlling the first sensor to generate first sensing data and the second sensor to generate second sensing data based on a first control signal transmitted from the control unit, the method further includes the steps of receiving an initialization signal transmitted from the control unit, starting a self-check operation and an initialization operation based on the initialization signal, and sending an initialization success signal to the control unit if no failure is found in the self-check and the initialization is completed.
[0017] Alternatively, the synchronous relay unit further includes a serializer, and the method further includes transmitting the first sensing data and the second sensing data to the control unit via the serializer.
[0018] According to another aspect of the present disclosure, there is further provided a computer readable storage medium including a computer program stored thereon, the computer program being configured to, when executed, control a device in which the computer readable storage medium is located to perform any one of the data synchronization methods described above.
[0019] In the present disclosure, there is provided a data synchronization device including a camera in which an assembly including a control unit, at least a synchronous relay unit, and a sensing unit is integrated, wherein the sensing unit includes a first sensor and a second sensor, the control unit transmits a first control signal to the synchronous relay unit, the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal, and sets a timestamp for the first sensing data, the control unit receives the first sensing data and the second sensing data, and synchronously aligns the first sensing data and the second sensing data based on the timestamp.
[0020] In the present disclosure, a timestamp can be set on the first sensing data (e.g., inertial sensing data) through a synchronization relay unit in the camera, the first sensing data and the second sensing data (e.g., image data) are synchronized, and the control unit can synchronously align the two sensing data based on the timestamp, thereby reducing errors between multiple sensing data and improving the user experience, thereby solving the problem that synchronization of multiple sensing data cannot be realized in related technologies.
[0021] In the present disclosure, an inertial sensor IMU and an image sensor can be installed in the same camera, and vibration data of the camera can be collected in real time via the IMU. Furthermore, the image data and vibration data acquired by the image sensor can be synchronously aligned via a synchronous relay unit on the camera, and the synchronized image data and vibration data can be transmitted to a remote control unit to prevent image vibration. At the same time, a synchronous relay unit can be placed on the camera, which has the advantage of high real-time performance and can reduce synchronization errors between image data and vibration data.
[0022] Furthermore, in the present disclosure, there is no requirement on the platform and computing power of the remote processing unit, and it is compatible with various platforms. [Brief description of the drawings]
[0023] The drawings described herein are used to provide a further understanding of the present disclosure, constitute a part of this application, and the exemplary embodiments of the present disclosure and the description thereof are used to interpret the present disclosure and are not intended to constitute undue limitations on the present disclosure. [Figure 1] 1 is a schematic diagram of a data synchronizer having an alternative microcontroller in accordance with the present disclosure; [Diagram 2] FIG. 4 is a timing diagram of signals in a data synchronizer having an alternative microcontroller according to the present disclosure. [Diagram 3] 4 is a flow chart of an alternative data synchronization method according to the present disclosure. [Figure 4] 1 is a schematic diagram of another alternative non-microcontroller data synchronizer in accordance with the present disclosure. [Diagram 5] FIG. 4 is a timing diagram of signals in an alternative non-microcontroller synchronizer according to the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of another alternative dual camera data synchronizer in accordance with the present disclosure. [Figure 7] FIG. 11 is a timing diagram of signals in an alternative dual camera synchronizer according to the present disclosure. [Figure 8] FIG. 1 is a schematic diagram of a data synchronizer with another alternative radar in accordance with the present disclosure. [Figure 9] FIG. 11 is a timing diagram of signals in a synchronizer having an alternative radar according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0024] In order to allow those skilled in the art to understand the solutions of the present disclosure more clearly, the technical solutions of the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure, but it is obvious that the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without any creative efforts belong to the protection scope of the present disclosure.
[0025] In addition, the terms "first", "second", and the like in the specification and claims of the present disclosure and the drawings are set to distinguish similar objects, and are not set to describe a particular order or sequence. It is understood that the data used can be interchanged under appropriate circumstances, and therefore the embodiments of the present disclosure described herein may be performed in an order other than that shown or described herein. Furthermore, the terms "comprise" and "have" and their variants are intended to be non-exclusive, for example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to the steps or elements expressly recited, and may include other steps or elements that are not expressly recited or that are inherent to the process, method, product, or apparatus.
[0026] In order to facilitate understanding of the present disclosure for those skilled in the art, some terms or nouns related to each embodiment of the present disclosure are explained below.
[0027] An IMU (Inertial Measurement Unit) is a sensor primarily used to detect and measure acceleration and rotational motion.
[0028] An MCU (Microcontroller Unit) is a chip-level computer that is constructed by appropriately lowering the frequency of a central processing unit (processing unit), simplifying its specifications, and integrating peripheral interfaces such as memory, counters, USB, A / D conversion, UART, PLC, DMA, and even LCD drive circuits onto a single chip, and performs different combination controls in different application situations, such as electronic control of automobiles.
[0029] The following embodiments of the present disclosure may be applied in situations where multiple data need to be synchronized, such as in-vehicle devices having multiple sensors installed in various means of transportation, including but not limited to cars, buses, motorcycles, airplanes, trains, etc.
[0030] In the present disclosure, the inertial sensor IMU and the image sensor can be installed in the same camera, and the vibration data of the camera is collected in real time through the IMU. Furthermore, the image data and the vibration data acquired by the image sensor are synchronously aligned through the synchronous relay unit on the camera, and the synchronized image data and vibration data are transmitted to a remote control unit to realize prevention of image vibration. At the same time, the synchronous relay unit can be arranged on the camera, which has the advantages of high real-time performance, can reduce the synchronization error of the image data and the vibration data, and has no requirements on the platform and computing power of the remote processing unit, and is compatible with various platforms.
[0031] Example 1 FIG. 1 is a schematic diagram of a data synchronizer having an alternative microcontroller according to the present disclosure. As shown in FIG. 1, the synchronizer is a data synchronizer including a control unit and a camera, and an assembly integrated in the camera includes at least a synchronization relay unit and a sensing unit, the synchronization relay unit includes a microcontroller and a serializer, the sensing unit includes a first sensor and a second sensor, and the control unit includes at least an initial signal unit, a processing unit, and a deserializer.
[0032] In the present disclosure, the control unit transmits a first control signal to the synchronous relay unit, the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal, and sets a timestamp on the first sensing data, the control unit receives the first sensing data and the second sensing data, and synchronously aligns the first sensing data and the second sensing data based on the timestamp, the first sensor may include an inertial sensor, a geomagnetometer, etc., and the present disclosure will be described in detail using an example in which the first sensor is an inertial sensor.
[0033] In the present disclosure, the synchronous relay unit receives data (i.e., the first sensing data and the second sensing data) of the sensing unit, and after synchronizing the image data and the IMU data (i.e., synchronously controlling the first sensor and the second sensor according to the second control signal generated by the microcontroller), transmits the data to the control unit by the low-voltage differential signal LVDS (i.e., transmits the synchronized data of the first sensor and the second sensor to the control unit), transmits the image to the processing unit of the control unit, and then performs image processing by the ISP module mounted on the processing unit. By adding an ISP module to the sensing unit of the camera, it is possible to improve the image quality. Image processing can also be performed before the image data is transmitted to the processing unit of the control unit, and a GPS reference clock signal (i.e., an initial signal) can be generated in the control unit via an initial signal unit and provided to the processing unit, and the processing unit performs division processing based on the GPS reference clock signal to obtain clock division information (i.e., a divided signal), and then the deserializer generates a first control signal based on the divided signal and transmits the first control signal to the serializer, the serializer generates a synchronization signal based on the first control signal and transmits it to the microcontroller, and the microcontroller generates a second control signal based on the synchronization signal.
[0034] In the present disclosure, after receiving the first sensing data and the second sensing data with timestamps set, the control unit can synchronously align the first sensing data and the second sensing data based on the timestamps, and when the second sensor is an image sensor, the timestamps are set according to the exposure start time of each frame image in the second sensing data (i.e., image data).
[0035] The specific workflow is as follows:
[0036] 10. Power on the control unit and camera, the control unit starts self-check and initializes the camera.
[0037] 20. After receiving the initialization signal from the processing unit, the camera starts self-check and initialization, and after completion, the microcontroller (MCU) sends an initialization success signal via the I2C bus and notifies the remote processing unit via the serializer and deserializer.
[0038] 30. The processing unit receives an initialization success signal sent from the MCU, and the processing unit provides a GPS reference clock signal (i.e., an initial signal) through the initial signal unit, sends a first control signal through the deserializer after frequency division, and connects to the MCU through the serializer.
[0039] 40. After receiving the first control signal via the serializer, the MCU further converts it into a second control signal and sends it to the first sensor in the sensing unit, and at the same time starts adding a time stamp to the IMU data (i.e., the first sensing data) and returns it to the processing unit via the serializer and deserializer.
[0040] 50. After receiving the second control signal, the second sensor in the sensing unit starts collecting image data (i.e., the second sensing data), and the image data is transmitted to the processing unit via the serializer and deserializer, and the processing unit receives the image data of this frame and also receives the IMU data with the time stamp, and then synchronously aligns the image data and the IMU data.
[0041] In the present disclosure, FIG. 2 is a timing diagram of each signal in a data synchronizer having an alternative microcontroller according to the present disclosure, including an initial signal, a frequency division signal, a first control signal, a first sensing data (not synchronized), a first sensing data (synchronized alignment), and a second sensing data, and these signals align the transmitted second sensing data with the first sensing data.
[0042] Alternatively, before the control unit sends the first control signal to the synchronous relay unit, the camera starts a self-check and initialization after receiving an initialization signal sent from the control unit, and after the self-check finds no faults and the initialization is completed, sends an initialization success signal to the control unit; and after the control unit receives the initialization success signal, sends the first control signal to the synchronous relay unit.
[0043] In the present disclosure, the control unit and the camera need to be powered on first, and then the control unit will start a self-check and initialize the camera (i.e., the control unit will perform a self-check and then send an initialization signal to the camera), and the camera will start a self-check and initialization after receiving the initialization signal, and after the camera self-check has no faults and the initialization is completed, if the synchronous relay unit includes a microcontroller MCU, the MCU can send an initialization success signal (i.e., indicating that the camera self-check has no faults and the initialization is completed) via various types of buses (e.g., I2C bus), and notify a remote processing unit (i.e., the processing unit of the control unit) via a serializer and a deserializer, and after the processing unit receives the initialization success signal sent from the MCU, it will send a first control signal to the synchronous relay unit.
[0044] Alternatively, the control unit includes an initial signal unit, a processing unit, and a deserializer, where the initial signal unit is configured to send an initial signal to the processing unit, the processing unit is configured to perform a division process on the initial signal and generate a divided signal, and the deserializer is configured to generate the first control signal based on the divided signal.
[0045] In the present disclosure, after the processing unit receives an initialization success signal sent from the MCU, the processing unit performs division processing (the processing unit is configured to perform division processing on the initial signal) based on the GPS reference clock signal provided by the initial signal unit (i.e., the initial signal unit is configured to send an initial signal to the processing unit) to obtain clock division information (i.e., the divided signal), and the deserializer is configured to generate a first control signal based on the divided signal.
[0046] Alternatively, the synchronous relay unit further includes a serializer configured to receive a first control signal transmitted from the deserializer in the control unit and to generate a synchronization signal for controlling the first sensor and the second sensor, respectively.
[0047] In the present disclosure, the control unit performs frequency division processing and then transmits a first control signal through the deserializer, and the serializer generates a synchronization signal based on the first control signal to control the first sensor and the second sensor respectively.
[0048] Alternatively, the synchronous relay unit further includes a microcontroller configured to receive the synchronization signal transmitted from the serializer and generate a second control signal to synchronously control the first sensor and the second sensor.
[0049] In the present disclosure, the serializer generates a synchronization signal based on a first control signal and transmits the synchronization signal to a microcontroller, and the microcontroller generates a second control signal based on the synchronization signal to synchronously control the first sensor and the second sensor.
[0050] Alternatively, when a microcontroller is not installed in the synchronous relay unit, the serializer and the first sensor are connected using a first bus, and when a microcontroller is installed in the synchronous relay unit, the first sensor and the microcontroller are connected using a second bus.
[0051] In the present disclosure, when a microcontroller is not installed in the synchronous relay unit, the inertial sensor (i.e., the first sensor) and the serializer are directly connected using a first bus (e.g., an I2C bus), and when the assembly of the synchronous relay unit further includes a microcontroller unit (i.e., an MCU) in addition to the serializer, the inertial sensor and the microcontroller unit are directly connected using a second bus.
[0052] Alternatively, the data synchronizer further includes a second sensing unit configured to receive a frequency-divided signal generated by a processing unit in the control unit and transmit third sensing data to the control unit under the control of the frequency-divided signal to synchronously align the third sensing data with the data generated by the first sensing unit, the second sensing unit including at least one of a laser radar, a millimeter wave radar, and an ultrasonic radar.
[0053] In the present disclosure, the data synchronizer is exemplified as including a laser radar and / or a millimeter wave radar. A radar module (i.e., a second sensing unit) including at least a laser radar and / or a millimeter wave radar is connected to a control unit, and the control unit can transmit a laser radar synchronization signal and / or a millimeter wave synchronization signal after frequency division processing, and the radar module can time stamp the laser radar data and / or the millimeter wave radar data after receiving the laser radar synchronization signal and / or the millimeter wave synchronization signal (i.e., after receiving a frequency division signal generated by a processing unit in the control unit), and return it to the processing unit (i.e., send third sensing data to the control unit under the control of the frequency division signal), and the processing unit can receive image data, and after receiving time stamped IMU data (i.e., data generated by the first sensing unit including image data and sensing data) and time stamped radar data (i.e., third sensing data, which is the laser radar data and / or millimeter wave radar data), perform synchronization alignment of the image data, the IMU data, and the radar data.
[0054] Alternatively, the first sensor includes an inertial sensor, a geomagnetic sensor, etc., and the second sensor is an image sensor, such as a regular RGB image sensor, a depth image sensor, a thermal image sensor, etc.
[0055] In the present disclosure, an inertial sensor and an image sensor can be installed in the same camera, synchronization is initiated by a synchronization relay unit in the camera, and synchronization of sensing data and image data is performed on the camera side, making the synchronization time accurate and providing high platform compatibility. In addition, the present disclosure can add an MCU to the camera to perform synchronization of multiple sensors, providing high real-time control of the MCU and reducing errors in synchronization between sensing data and image data. Furthermore, since the MCU is placed in the camera, there are no requirements for the platform and computing power of the remote processing unit, and providing high compatibility with various platforms.
[0056] Example 2 In accordance with the present disclosure, a data synchronization method is provided, and it should be noted that the steps illustrated in the flowcharts of the drawings may be performed, for example, in a computer system as a set of computer-executable commands, and that although a logical order is shown in the flowcharts, in some cases the steps may be performed in an order different from that shown or described.
[0057] FIG. 3 is a flow chart of an alternative data synchronization method according to the present disclosure. As shown in FIG. 3, the method includes: Step S302: the synchronous relay unit synchronously controls the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal transmitted from the control unit; A step S304 of setting a time stamp to the first sensing data; The method includes a step S306 of transmitting the first sensing data and the second sensing data with the time stamp set to a control unit, and the control unit synchronously aligning the first sensing data and the second sensing data based on the time stamp.
[0058] According to the above steps, the synchronous relay unit synchronously controls the first sensor to generate the first sensing data and the second sensor to generate the second sensing data based on the first control signal transmitted from the control unit, sets a time stamp on the first sensing data, and transmits the first sensing data and the second sensing data with the time stamp set to the control unit, and the control unit synchronously aligns the first sensing data and the second sensing data based on the time stamp. In the present disclosure, the synchronous relay unit in the camera synchronously controls the sensing data and the image data, and sets a time stamp on the sensing data, and transmits the time stamped sensing data and the image data to the control unit, thereby realizing the alignment process between the sensing data and the image data, reducing the error between multiple sensing data and improving the user experience, and solving the problem that the synchronization of multiple sensing data cannot be realized in the related art.
[0059] The present disclosure will now be described in detail with reference to each of the above steps.
[0060] In the present disclosure, each of the steps below is applied to a camera of an on-board device installed in various transportation facilities having multiple sensors, a connection is established in advance between the camera and the control unit, an assembly integrated in the camera includes at least a synchronous relay unit and a sensing unit, the sensing unit includes a first sensor and a second sensor, the synchronous relay unit includes at least a serializer and further includes a microcontroller, the control unit includes a deserializer, an initial signal unit and a processing unit, the first sensor may include an inertial sensor, a geomagnetometer, etc., and the present disclosure will be described in detail as an example in which the first sensor is an inertial sensor.
[0061] In the present disclosure, before the step of the synchronization relay unit synchronously controlling the first sensor to generate first sensing data and the second sensor to generate second sensing data based on the first control signal transmitted from the control unit, the data synchronization method further includes the steps of receiving an initialization signal transmitted from the control unit, starting a self-check operation and an initialization operation based on the initialization signal, and sending an initialization success signal to the control unit if no failure is found in the self-check and the initialization is completed.
[0062] In this disclosure, the control unit and the camera need to be powered on first, then the control unit starts a self-check and initializes the camera (i.e., the control unit performs a self-check and then sends an initialization signal to the camera), the camera starts a self-check and initialization after receiving the initialization signal, after the camera self-check has no faults and the initialization is completed, if the synchronous relay unit includes an MCU, the MCU can send an initialization success signal (i.e., indicates that the camera self-check has no faults and the initialization is completed) via various types of buses (e.g., I2C bus), and notify the remote processing unit via the serializer and deserializer (i.e., sends an initialization success signal to the control unit). If the synchronous relay unit does not include an MCU, the serializer sends an initialization success signal via a certain type of bus (e.g., I2C bus), and notifies the remote processing unit (sends an initialization success signal to the control unit).
[0063] In step S302, the synchronous relay unit performs synchronous control such that the first sensor generates first sensing data and the second sensor generates second sensing data according to the first control signal transmitted from the control unit.
[0064] In the present disclosure, the control unit transmits the first control signal through the deserializer after frequency division, and when the synchronous relay unit includes an MCU, it can be connected to the MCU through the serializer, and the MCU receives the first control signal and generates a second control signal to synchronously control the first sensor to generate the first sensing data and the second sensor to generate the second sensing data.When the synchronous relay unit does not include an MCU, it can be connected to the sensing unit through the serializer, and the sensing unit receives the first control signal and synchronously controls the first sensor to generate the first sensing data and the second sensor to generate the second sensing data.
[0065] In step S304, a time stamp is set in the first sensing data.
[0066] In the present disclosure, when the synchronous relay unit includes an MCU, the second control signal generated by the MCU can be used to synchronously control the first sensor to generate the first sensing data and the second sensor to generate the second sensing data, and at the same time, start adding a time stamp to the IMU data (i.e., the first sensing data).When the synchronous relay unit does not include an MCU, the first control signal can be used to synchronously control the first sensor to generate the first sensing data and the second sensor to generate the second sensing data, and at the same time, add a time stamp to the IMU data received at this point.
[0067] In step S306, the first sensing data and the second sensing data with the time stamps are transmitted to the control unit, and the control unit synchronously aligns the first sensing data and the second sensing data based on the time stamps.
[0068] In the present disclosure, after synchronizing the image data and the sensing data (i.e., after setting a timestamp in the first sensing data), the first sensing data and the second sensing data with the timestamp set can be transmitted to a control unit by a low-voltage differential signal LVDS, and after receiving the first sensing data and the second sensing data, the control unit can synchronously align the first sensing data and the second sensing data based on the timestamp, and when the second sensor is an image sensor, the timestamp is set according to the exposure start time of each frame image in the second sensing data (i.e., image data).
[0069] Alternatively, the synchronization relay unit further includes a serializer, and the data synchronization method further includes transmitting the first sensing data and the second sensing data to the control unit via the serializer.
[0070] In the present disclosure, the aligned sensing data and image data can be transmitted to a deserializer of a control unit of an in-vehicle device via a serializer of a synchronous relay unit, and then transmitted to a processing unit by the deserializer.
[0071] In the present disclosure, an inertial sensor and an image sensor can be installed in the same camera, synchronization can be initiated by a synchronization relay unit in the camera, synchronization of sensing data and image data can be performed on the camera side, camera vibration data can be collected in real time, and image vibration prevention can be set. In addition, the present disclosure can add an MCU to the camera to synchronize multiple sensors, the real-time nature of the MCU control can be high, and the synchronization error between multiple sensors and image frames can be reduced. In addition, since the MCU is placed in the camera, there are no requirements for the platform and computing power of the remote processing unit, and it is compatible with various platforms.
[0072] Example 3 FIG. 4 is a schematic diagram of another alternative data synchronizer not including a microcontroller according to the present disclosure. As shown in FIG. 4, the synchronizer includes a control unit and a camera, and an assembly integrated in the camera includes at least a synchronization relay unit and a sensing unit, and the synchronization relay unit includes at least a serializer, and the sensing unit includes a first sensor and a second sensor, and the first sensor can include an inertial sensor, a geomagnetometer, etc., and the present disclosure describes in detail the first sensor being an inertial sensor as an example, and the control unit includes at least an initial signal unit, a processing unit, and a deserializer.
[0073] In the present disclosure, the sensing unit introduces the image data signal to be transmitted, the IMU data signal to be transmitted, the frame synchronization signal for turning on the exposure, and the power supply, etc., and these signals are connected to the synchronous relay unit through the FPC cable, and the synchronous relay unit receives the image data and the IMU data and transmits them to the control unit through the low-voltage differential signal LVDS, and after the image data is transmitted to the processing unit of the control unit, the image data and the IMU data can be synchronized in the processing unit, and the image processing of the image data can be performed by the ISP mounted on the processing unit. By adding an ISP to the sensing unit of the camera, image processing can also be performed before the image data is transmitted to the processing unit of the control unit, and a GPS reference clock signal can be generated in the control unit through the initial signal unit and provided to the processing unit. The specific operation flow is as follows: 11. Power on the control unit and camera, the control unit will start self-check and initialize the camera.
[0074] 21. After receiving the initialization signal from the processing unit, the camera starts self-check and initialization. After completion, the serializer sends an initialization success signal via the I2C bus to notify the remote processing unit. At the same time, the data of the microcontroller (MCU) is transmitted to the processing unit via the I2C bus, the serializer, and the deserializer.
[0075] 31. The processing unit provides a GPS reference clock signal (i.e., an initial signal) through the initial signal unit, and after frequency division, the processing unit sends a first control signal through the deserializer, which is converted into a synchronization signal through the serializer and then connected to the first sensor in the sensing unit, and at the same time, gives a timestamp to the IMU data (i.e., the first sensing data) received at this time.
[0076] 41. After receiving the synchronization signal sent from the serializer, the sensing unit starts collecting image data (i.e., the second sensing data), and the image data is transmitted to the processing unit via the serializer and deserializer. The processing unit receives the image data of this frame, and also stores the IMU data that has already been time-stamped, and then synchronously aligns the image data and the IMU data.
[0077] In the present disclosure, FIG. 5 is a timing diagram of each signal in an alternative microcontroller-free synchronizer according to the present disclosure, including an initial signal, a divided signal, a second sensing data, and a first sensing data (not synchronized), which then align the transmitted second sensing data with the first sensing data.
[0078] In the present disclosure, the inertial sensor and the image sensor can be arranged together, and the processing unit initiates synchronization, and multiple sensing data are synchronized on the camera side, making the synchronization time accurate and achieving high platform compatibility.
[0079] Example 4 FIG. 6 is a schematic diagram of another alternative data synchronizer having dual cameras according to the present disclosure. As shown in FIG. 6, the synchronizer includes a control unit, a camera A, and a camera B, where the camera A includes a sensing unit A and a synchronous relay unit A, the camera B includes a sensing unit B and a synchronous relay unit B, the sensing unit includes a first sensor and a second sensor, the synchronous relay unit includes a microcontroller and a serializer, the control unit includes a deserializer, an initial signal unit, and a processing unit, and the first sensor may include an inertial sensor, a geomagnetometer, etc., and the present disclosure will be described in detail as an example in which the first sensor is an inertial sensor.
[0080] In the present disclosure, the sensing unit introduces the image data signal to be transmitted, the IMU data signal to be transmitted, the frame synchronization signal for turning on the exposure, and the power supply, and these signals are connected to the synchronous relay unit via the FPC cable. The synchronous relay unit receives the image data and the IMU data, and transmits them to the control unit via the low-voltage differential signal LVDS. After the image data is transmitted to the processing unit of the control unit, the image data can be synchronized with the IMU data GPS reference clock signal in the processing unit, and the image processing of the image data can be performed by the ISP mounted on the processing unit. By adding an ISP to the sensing unit of the camera, the image data can also be processed before being transmitted to the processing unit of the control unit. The control unit can generate a GPS reference clock signal through the initial signal unit and provide it to the processing unit. The specific operation flow is as follows:
[0081] 12. Power on the control unit and camera A and camera B, the control unit starts self-checking and initializes camera A and camera B.
[0082] 22. After receiving the initialization signal from the processing unit, camera A and camera B start self-check and initialization. After completion, the microcontroller (MCU) sends an initialization success signal via the I2C bus and notifies the remote processing unit via the serializer and deserializer.
[0083] 32. The processing unit receives the initialization success signal sent from the MCU, and the processing unit provides a GPS reference clock signal (i.e., an initial signal) through the initial signal unit, which is converted into a first control signal through the deserializer after frequency division, and received by the MCUs of camera A and camera B through the serializer.
[0084] 42. The two MCUs simultaneously receive the first control signal through the serializer, then convert it into a second control signal and send it to the first sensors in the two sensing units respectively, and at the same time start adding a time stamp to each IMU data (i.e., the first sensing data) and return it to the processing unit via the serializer and deserializer.
[0085] 52. After receiving the second control signal, the second sensor in the two sensing units starts collecting image data (i.e., the second sensing data), and the image data is transmitted to the processing unit via the serializer and deserializer. The processing unit receives the image data of the two sensing units and also receives the respective IMU data with time stamps, and then synchronously aligns the image data of Camera A and Camera B with the IMU data.
[0086] In the present disclosure, FIG. 7 is a timing diagram of each signal in an alternative dual camera synchronizer of the present disclosure, including an initial signal, a divided signal, a first control signal (A), a first control signal (B), a second sensing data (A), a first sensing data (A) (not synchronized), a second sensing data (B), and a first sensing data (B) (not synchronized), and these signals align the transmitted second sensing data with the first sensing data.
[0087] In the present disclosure, the synchronization device can have multiple cameras, and can realize the synchronization of multiple sensing data in multiple cameras, and only one control unit is required. This disclosure takes two cameras as an example, and in each camera, an inertial sensor and an image sensor can be arranged together, and synchronization is initiated by a processing unit, and synchronization between multiple sensing data is performed on the camera side respectively, so that the synchronization time is accurate and the platform compatibility is high; and this disclosure adds an MCU to the camera to perform synchronization between multiple sensors, so that the real-time control of the MCU is high, and the synchronization error between multiple sensing data can be reduced; and since the MCU is arranged in the camera, there is no requirement for the platform and computing power of the remote processing unit, and the compatibility with various platforms is high.
[0088] Example 5 FIG. 8 is a schematic diagram of another alternative data synchronizer having a radar according to the present disclosure. As shown in FIG. 8, the synchronizer includes a control unit, a second sensing unit, and a camera, the camera includes a sensing unit and a synchronization relay unit, the sensing unit includes a first sensor and a second sensor, the first sensor may include an inertial sensor, a geomagnetometer, etc. (the present disclosure will be described in detail as an example in which the first sensor is an inertial sensor), the synchronization relay unit includes a microcontroller and a serializer, the control unit includes a deserializer, an initial signal unit, and a processing unit, and the second sensing unit includes at least a laser radar and a millimeter-wave radar (the second sensing unit may further include an ultrasonic radar, etc., and the present disclosure will be described in detail as an example in which the data synchronizer includes a laser radar and / or a millimeter-wave radar).
[0089] In the present disclosure, the sensing unit introduces an image data signal to be transmitted, an IMU data signal to be transmitted, a frame synchronization signal for turning on exposure, and a power supply, etc., and these signals are connected to the synchronous relay unit via an FPC cable, and the synchronous relay unit receives the image data and IMU data and transmits them to the control unit via a low-voltage differential signal LVDS. After the image data is transmitted to the processing unit of the control unit, the image data and the IMU data can be synchronized with a GPS reference clock signal in the processing unit, and image processing of the image data can be performed by an ISP mounted on the processing unit. By adding an ISP to the sensing unit of the camera, image processing can also be performed before the image data is transmitted to the processing unit of the control unit. A GPS reference clock signal can be generated and provided to the processing unit via an initial signal unit in the control unit, and a laser radar synchronization signal and a millimeter wave synchronization signal are output from the processing unit of the control unit to synchronize the laser radar and millimeter wave radar data. The specific operation flow is as follows:
[0090] 13. Power on the control unit, camera and radar (i.e. the second sensing unit), the control unit starts self-check and initializes the camera and radar.
[0091] 23. After receiving the initialization signal from the processing unit, the camera starts self-check and initialization, and after completion, the microcontroller (MCU) sends an initialization success signal via the I2C bus and notifies the remote processing unit via the serializer and deserializer.
[0092] 33. The processing unit receives the initialization success signal sent from the MCU, and the processing unit provides a GPS reference clock signal (i.e., an initial signal) through the initial signal unit, which is converted into a first control signal through the deserializer after frequency division, which is received by the camera's MCU through the serializer, and at the same time sends a radar synchronization signal (including a laser radar synchronization signal and a millimeter wave synchronization signal).
[0093] 43. After receiving the first control signal through the serializer, the MCU further converts it into a second control signal and sends it to the first sensor in the sensing unit, and at the same time starts adding a time stamp to the IMU data (i.e., the first sensing data) and returns it to the processing unit via the serializer and deserializer.
[0094] 53. After the second sensor in the sensing unit receives the second control signal, the data is transmitted to the processing unit via the serializer and deserializer, and the processing unit receives the image data (second sensing data) of the sensing unit, also receives the IMU data with time stamp, and simultaneously receives the laser radar and millimeter wave radar data (third sensing data) with time stamp, and finally synchronously aligns the camera image data, IMU data, laser radar data, and millimeter wave radar data.
[0095] In the present disclosure, Figure 9 is a timing diagram of each signal in a synchronization device having an alternative radar according to the present disclosure, including an initial signal, a frequency-divided signal, a first control signal, a second sensing data, a first sensing data (not synchronized), a laser radar synchronization signal, a laser radar data, a millimeter-wave synchronization signal, and millimeter-wave radar data, and these signals align the multiple transmitted sensing data (including image data, inertial sensing data, laser radar data, and millimeter-wave radar data).
[0096] In the present disclosure, the inertial sensor and the inertial sensor can be arranged together, synchronization is initiated by the processing unit, synchronization of multiple sensing data is performed on the camera side, and multiple types of sensors such as laser radar, millimeter wave radar, ultrasonic radar, etc. can be supported, the synchronization time can be accurate, and platform compatibility can be high; and the present disclosure can add an MCU to the camera to synchronize multiple sensors, the real-time performance of the MCU control can be high, and the synchronization error of multiple sensing data can be reduced; and since the MCU is arranged in the camera, there is no requirement for the platform and computing power of the remote processing unit, and the compatibility with various platforms is high.
[0097] According to another aspect of the present disclosure, there is further provided a computer-readable storage medium including a stored computer program, which, when executed, controls a device in which the computer-readable storage medium is located so as to perform the data synchronization method for an in-vehicle device described in any one of the above claims.
[0098] The example numbers in the above disclosure are for illustrative purposes only and do not indicate superiority or inferiority of the examples.
[0099] In the above embodiments of the present disclosure, the description of each embodiment focuses on specific points, and for content that is not detailed in one embodiment, reference can be made to the descriptions in other embodiments.
[0100] In some embodiments provided by the present application, it is to be understood that the disclosed technical contents can be realized in other ways. Here, the embodiments of the device described above are merely illustrative, for example, the division of the units is merely a division of logical functions, and may have other division methods when actually realized, for example, multiple units or assemblies may be integrated or integrated into another system, or some features may be omitted or not implemented. In other respects, the illustrated or discussed mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections via interfaces, units or modules, and may be electrical or other forms.
[0101] In addition, the units described as separate parts may or may not be physically separate, and the parts shown as units may be physical units, or may be co-located, or may be arranged in multiple units, although not physical units. Some or all of these units may be selected according to actual needs to achieve the objectives of the solutions of the present disclosure.
[0102] In addition, each functional unit in various embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The integrated units may be implemented in the form of either hardware or software functional units.
[0103] The integrated units may be realized as software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, a substantial part of the technical solution of the present disclosure that contributes to the prior art, or a part or all of the technical solution, may be realized in the form of a software product. The computer software product is stored in one storage medium and includes several commands for executing all or a part of the steps included in the method according to each embodiment of the present disclosure by one computer device (which may be a personal computer, a server, or a network device, etc.). The storage medium may be a medium capable of storing program code, such as a USB memory, a ROM (Read-Only Memory), a RAM (Random Access Memory), a portable hard disk, a disk, or a CD-ROM.
[0104] The above is only a preferred embodiment of the present disclosure, and those skilled in the art may further make some improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be regarded as within the protection scope of the present invention.
Claims
1. A data synchronization device including a control unit and a camera, wherein an assembly integrated with the camera includes at least a synchronization relay unit and a sensing unit, the sensing unit including a first sensor and a second sensor; The control unit transmits a first control signal to the synchronous relay unit; The synchronous relay unit performs synchronous control so that the first sensor generates first sensing data and the second sensor generates second sensing data based on the first control signal, and sets a time stamp to the first sensing data; The control unit receives the first sensing data and the second sensing data, and synchronously aligns the first sensing data and the second sensing data based on the time stamp.
2. Before the control unit transmits a first control signal to the synchronous relay unit, the camera starts a self-check and initialization after receiving the initialization signal sent from the control unit, and sends an initialization success signal to the control unit after the self-check has no fault and the initialization has been completed; The data synchronizer according to claim 1 , further comprising: a step of: the control unit transmitting a first control signal to the synchronization relay unit after receiving the initialization success signal.
3. The control unit includes an initial signal unit, a processing unit, and a deserializer; the initial signal unit is configured to send an initial signal to the processing unit; The processing unit is configured to perform a frequency division process on the initial signal and generate a frequency-divided signal; The data synchronizer of claim 1 , wherein the deserializer is configured to generate the first control signal based on the divided signal.
4. 4. The data synchronizer of claim 3, wherein the synchronization relay unit further includes a serializer configured to receive the first control signal transmitted from a deserializer in the control unit and to generate synchronization signals for controlling the first sensor and the second sensor, respectively.
5. The data synchronization device of claim 4 , wherein the synchronization relay unit further includes a microcontroller configured to receive the synchronization signal transmitted from the serializer and generate a second control signal to synchronously control the first sensor and the second sensor.
6. 6. The data synchronization device of claim 5, wherein when the microcontroller is not installed in the synchronization relay unit, the serializer and the first sensor are connected using a first bus, and when the microcontroller is installed in the synchronization relay unit, the first sensor and the microcontroller are connected using a second bus.
7. 4. The data synchronization device of claim 3, further comprising a second sensing unit configured to receive a frequency-divided signal generated by the processing unit in the control unit and transmit third sensing data to the control unit under control of the frequency-divided signal to synchronously align the third sensing data with the data generated by the first sensing unit, wherein the second sensing unit comprises at least one of a laser radar, a millimeter wave radar, and an ultrasonic radar.
8. The data synchronizer of claim 1 , wherein the first sensor includes at least one of an inertial sensor and a geomagnetic sensor, and the second sensor is an image sensor.
9. A data synchronization method applied to a camera of an in-vehicle device, the camera and a control unit are connected in advance, and an assembly integrated in the camera includes at least a synchronization relay unit and a sensing unit, the sensing unit includes a first sensor and a second sensor, The synchronous relay unit performs synchronous control so that the first sensor generates first sensing data and the second sensor generates second sensing data based on a first control signal transmitted from a control unit; setting a timestamp on the first sensing data; transmitting the first sensing data and the second sensing data, to which a time stamp is set, to the control unit, and the control unit synchronously aligns the first sensing data and the second sensing data based on the time stamp; A data synchronization method including:
10. Before the step of synchronously controlling the first sensor to generate first sensing data and the second sensor to generate second sensing data based on a first control signal transmitted from a control unit, the synchronous relay unit receiving an initialization signal transmitted from the control unit; starting a self-check operation and an initialization operation based on the initialization signal; sending an initialization success signal to the control unit when the self-check shows no failure and the initialization is completed; The data synchronization method of claim 9 , further comprising:
11. The synchronous relay unit further includes a serializer; The data synchronization method according to claim 9 , further comprising the step of transmitting the first sensing data and the second sensing data to the control unit via the serializer.
12. A computer readable storage medium containing a computer program stored thereon, A computer readable storage medium which, when executed, controls a device in which the computer readable storage medium resides to perform the data synchronization method according to any one of claims 9 to 11.
Citation Information
Patent Citations
On-vehicle device
JP2015023510A
External hybrid photomapping
JP2016540187A
Imaging device and imaging system
WO2013031514A1
Data synchronization device, data synchronization method and data synchronization program
WO2019215923A1
