Multi-sensor time synchronization system and self-moving device

Abstract

A multi-sensor time synchronization system and a self-moving device. The system comprises a main controller, a microcontroller, an inertial measurement unit, and cameras. The main controller is used for sending first system time to the microcontroller. The microcontroller is used for receiving the first system time and performing time synchronization with the main controller, and is further used for: controlling the inertial measurement unit to start and collect inertial measurement data, receiving the inertial measurement data and an interrupt signal of the inertial measurement unit, and recording a first timestamp; and generating a synchronization trigger signal on the basis of the interrupt signal and sending the synchronization trigger signal to the cameras. The cameras are used for receiving the synchronization trigger signal and then performing exposure to collect image data. The microcontroller is further used for receiving exposure starting time of the cameras and recording a second timestamp. The main controller is further used for matching the first timestamp, the inertial measurement data of the inertial measurement unit, the second timestamp, and the image data. The method provided by the present application solves the problem of low time synchronization accuracy of conventional inertial measurement units and cameras.

Description

Multi-sensor time synchronization system and self-moving equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 1, 2023, with application number 202310965138.4 and invention name “System and self-mobile device for multi-sensor time synchronization”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of automation technology, and in particular relates to a multi-sensor time synchronization system and a self-propelled device. Background Art

[0003] Autonomous vehicles have become a hot topic in the field of machine vision, encompassing emerging areas such as robotic lawn mowers, sweepers, companion robots, autonomous vehicles, and drones. Autonomous vehicles can incorporate inertial motion units (IMUs) and cameras to enable visual localization (VSLAM) and mapping (MAPPING). While IMUs can improve robustness and accuracy, precise time synchronization is crucial for multi-sensor fusion.

[0004] In the prior art, a self-moving device can achieve time synchronization between an inertial measurement unit and a camera through software scheduling. However, since the self-moving device includes multiple clock source scheduling and the software scheduling of the operating system also causes a large delay, the accuracy of the time synchronization between the inertial measurement unit and the camera is very low. Technical issues

[0005] The purpose of this application is to provide a multi-sensor time synchronization system and a self-moving device, aiming to solve the problem of low accuracy of time synchronization between traditional inertial measurement units and cameras. Technical Solutions

[0006] A first aspect of an embodiment of the present application proposes a multi-sensor time synchronization system, which includes a main controller, a microcontroller, an inertial measurement unit and a camera: the main controller is used to send a first system time to the microcontroller; the microcontroller is used to receive the first system time and synchronize time with the main controller based on the first system time; the microcontroller is also used to control the inertial measurement unit to start to collect inertial measurement data, receive the inertial measurement data and an interrupt signal from the inertial measurement unit and record a first timestamp; generate a synchronization trigger signal according to the interrupt signal and send it to the camera; the camera is used to perform an exposure operation after receiving the synchronization trigger signal to collect image data; the microcontroller is also used to receive the exposure start time of the camera and record a second timestamp; the main controller is also used to match the first timestamp, the inertial measurement data of the inertial measurement unit, the second timestamp and the image data to associate the correspondence between the inertial measurement data of the inertial measurement unit and the image data.

[0007] In some embodiments, the microcontroller is specifically configured to determine a synchronization trigger signal based on a first frame rate and a second frame rate, wherein the first frame rate is a frame rate of inertial measurement data of an inertial measurement unit, and the second frame rate is a frame rate of image data captured by a camera.

[0008] In some embodiments, the camera is specifically configured to start acquiring image data upon receiving a synchronization trigger signal.

[0009] In some embodiments, the camera is specifically configured to, when receiving a synchronization trigger signal, determine an exposure start time based on the synchronization trigger signal and the exposure duration of the camera, start collecting image data at the exposure start time, and send the exposure start time to the microcontroller.

[0010] In some embodiments, the camera includes a master camera and a slave camera.

[0011] In some embodiments, the microcontroller is specifically configured to obtain a second timestamp corresponding to the master camera and a second timestamp corresponding to the slave camera, and send the second timestamp corresponding to the master camera and the second timestamp corresponding to the slave camera to the master controller; the master controller is specifically configured to correspond image data captured by the master camera to the second timestamp of the master camera, and correspond image data captured by the slave camera to the second timestamp of the slave camera.

[0012] In some embodiments, the main controller is specifically used to obtain the first system time and send the first system time and a timing signal to the microcontroller; the microcontroller is specifically used to start timing when receiving the timing signal, stop timing when receiving the first system time, and determine the second system time of the microcontroller based on the timing duration and the first system time.

[0013] A second aspect of the embodiments of the present application provides a self-moving device, comprising any of the above-mentioned systems.

[0014] In some embodiments, the self-moving device comprises a lawn mowing robot.

[0015] In some embodiments, the camera includes an RGB camera, a fisheye camera, a panoramic camera, or a depth camera. Beneficial effects

[0016] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0017] In an embodiment of the present application, the main controller can be used to send a first system time to the microcontroller and perform hardware real-time synchronization with the microcontroller based on the first system time. The microcontroller can be used to control the inertial measurement unit to start to collect inertial measurement data, receive the inertial measurement data and interrupt signal of the inertial measurement unit and record the first timestamp, generate a synchronization trigger signal based on the interrupt signal and send it to the camera, and the camera is used to perform an exposure operation to collect image data after receiving the synchronization trigger signal, that is, to achieve hardware real-time synchronization between the inertial measurement unit and the camera. The microcontroller is used to receive the exposure start time of the camera and record the second timestamp. The main controller is also used to match the first timestamp, the inertial measurement data of the inertial measurement unit, the second timestamp and the image data to associate the correspondence between the inertial measurement data of the inertial measurement unit and the image data. The microcontroller controls the inertial measurement unit to start collecting inertial measurement data and generates a synchronization trigger signal, so that the camera can be controlled to collect image data based on the synchronization trigger signal. That is, hardware real-time synchronization is first performed through the main controller and the microcontroller, and then hardware real-time synchronization is achieved between the inertial measurement unit and the camera through the microcontroller, which greatly improves the accuracy of time synchronization between the inertial measurement unit and the camera, and also improves the accuracy of establishing the association between the inertial measurement data of the inertial measurement unit and the image data of the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG1 is a schematic diagram of the structure of a multi-sensor time synchronization system provided by an embodiment of the present application;

[0019] FIG2 is a schematic diagram of a flow chart of a multi-sensor time synchronization method provided in an embodiment of the present application;

[0020] FIG3 is a schematic diagram of a flow chart of a time synchronization method provided in an embodiment of the present application;

[0021] FIG4 is a schematic diagram of a flow chart of another time synchronization method provided in an embodiment of the present application;

[0022] FIG5 is a schematic diagram of an exposure start point synchronization timing provided by an embodiment of the present application;

[0023] FIG6 is a schematic diagram of an exposure center point alignment synchronization timing according to an embodiment of the present application;

[0024] FIG7 is a schematic diagram of another exposure center point alignment synchronization timing provided by an embodiment of the present application;

[0025] FIG8 is a schematic diagram of a flow chart of another time synchronization method provided in an embodiment of the present application;

[0026] FIG9 is a schematic structural diagram of a self-moving device provided in an embodiment of the present application. Modes for Carrying Out the Invention

[0027] In order to make the technical problems, technical solutions and beneficial effects to be solved by this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0028] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0029] Figure 1 shows a schematic diagram of the structure of a multi-sensor time synchronization system provided by an embodiment of the present application. The system includes an inertial measurement unit 110, one or more cameras 120 (only two cameras are shown in Figure 1), a microcontroller unit (MCU) 130, and a main controller 140.

[0030] The inertial measurement unit 110 may include an accelerometer and a gyroscope. The inertial measurement unit 110 can be used to detect acceleration and angular velocity, thereby indicating motion and motion intensity. In some embodiments, the inertial measurement unit 110 can collect inertial measurement data (IMU Data) at a preset first frame rate and send an interrupt signal (Data Ready INT) to the microcontroller 130. The interrupt signal can be used to notify the microcontroller 130 that inertial measurement data collection is complete. In some embodiments, the first frame rate can be 1000 Hz (Hertz) or 200 Hz. Of course, in actual applications, the first frame rate can be other values.

[0031] The camera 120, also known as a sensor, can be used to capture image data of the surrounding environment. In some embodiments, the camera 120 can be a color (Red, Green, Blue, RGB) camera, a fisheye camera, a panoramic camera, or a depth camera. It should be noted that the multiple cameras 120 can be of the same type, such as RGB cameras; alternatively, the multiple cameras 120 can be of different types. In some embodiments, as shown in FIG1 , the multiple cameras 120 can include a master camera and slave cameras. In some embodiments, the camera 120 can capture image data at a preset second frame rate. In some embodiments, the second frame rate can be 40 Hz. Of course, in actual applications, the second frame rate can be other values. It is understood that different cameras 120 can correspond to different second frame rates. In some embodiments, the camera 120 can trigger image data capture by a synchronous trigger signal (also known as a trigger pulse) sent by the microcontroller 130. In some embodiments, the camera 120 can transmit the captured image data to the main controller 140.

[0032] The microcontroller 130 can control the inertial measurement unit 110 to start collecting inertial measurement data, receive the inertial measurement data and interrupt signal from the inertial measurement unit 110, and record a first timestamp. The first timestamp can be used to indicate the time when the inertial measurement data was collected. Based on the interrupt signal, the microcontroller 130 can generate a synchronization trigger signal and send it to the camera 120 to trigger the camera 120 to collect image data. The microcontroller 130 receives the exposure start time from the camera 120 and records a second timestamp. The second timestamp can be used to indicate the time when the camera 120 begins the exposure operation. In other words, the microcontroller 130 can achieve hardware real-time synchronization between the inertial measurement unit 110 and the camera 120. In some embodiments, the microcontroller 130 can send the inertial measurement data from the inertial measurement unit 110, the first timestamp, and the second timestamp to the main controller 140. In some embodiments, the microcontroller 130 can receive the first system time from the main controller 140 and perform hardware real-time synchronization with the main controller 140 based on the first system time.

[0033] The algorithms configured in the main controller 140 may include a visual-inertial odometry (VIO) algorithm, a simultaneous localization and mapping (SLAM) algorithm, and the like. In some embodiments, the main controller 140 may match the first timestamp, the inertial measurement data from the inertial measurement unit (IMU), the second timestamp, and the image data to correlate the inertial measurement data from the IMU with the image data. The main controller 140 may also be used to implement functions such as positioning and navigation based on the inertial measurement data and image data. The main controller 140 may issue instructions in the form of pulses or potentials according to preset logical operation requirements to control the coordinated operation of the computer's input, operation, and output components.

[0034] It is understood that the embodiment of the present application only uses Figure 1 as an example to illustrate the structure of the multi-sensor time synchronization system, but is not limiting. In some embodiments, the multi-sensor time synchronization system may include more or fewer structures. In some embodiments, the multiple structures in Figure 1 may also be integrated into one structure, or one structure may also be split into multiple structures. For example, the microcontroller 130 and the main controller 140 in Figure 1 may be integrated into one controller. For another example, the microcontroller 130 in Figure 1 may also be referred to as a first processor, and the main controller 140 may also be referred to as a second processor.

[0035] FIG2 shows a multi-sensor time synchronization method provided in an embodiment of the present application, which can be applied to the system in FIG1 .

[0036] S201, the microcontroller and the main controller perform hardware real-time synchronization.

[0037] The microcontroller and the main controller perform hardware real-time synchronization, which can synchronize the system time of the microcontroller and the main controller.

[0038] In some implementations, the microcontroller may calibrate the system time of the microcontroller based on the system time of the main controller.

[0039] The method for hardware real-time synchronization between the microcontroller and the main controller can also refer to the method shown in Figure 3 below. Figure 3 shows a flow chart of a time synchronization method provided by an embodiment of the present application. The method can be a detailed description of S201. The method includes:

[0040] S301: The main controller obtains a first system time of the main controller and sends a timing signal to the microcontroller.

[0041] When the main controller and the microcontroller are synchronized, the first system time (Thost1) of the main controller needs to be sent to the microcontroller. Since the sending process takes time and the time taken is not fixed, the time difference (dt) of this process needs to be calculated.

[0042] In some embodiments, in order to prevent the main controller from being preempted by a high-priority task during the gap in obtaining the first system time, resulting in errors in the obtained first system time, the main controller can start the system interrupt disabling process and enable the system interrupt after obtaining the first system time.

[0043] In some embodiments, the timing signal can be used to instruct the microcontroller to start timing, thereby enabling the microcontroller to determine the time difference between obtaining the first system time from the main controller and receiving the first system time from the microcontroller. In some embodiments, the timing signal can be a high-level input / output (I / O) signal. It is understood that in actual applications, the timing signal can also be a signal in other forms, and the embodiments of this application do not limit the specific form of the timing signal.

[0044] S302, the microcontroller receives a timing signal and starts timing.

[0045] In some implementations, the microcontroller may start timing via a real-time clock (RTC) built into the microcontroller when receiving a high-level I / O signal.

[0046] S303: The main controller sends the first system time to the microcontroller.

[0047] It should be noted that the main controller can send the first system time to the microcontroller at any time after sending the timing signal.

[0048] S304 , when the microcontroller receives the first system time, it stops timing and determines the second system time of the microcontroller.

[0049] After the microcontroller obtains the first system time of the microcontroller, the first system time is added to the time difference counted by S302-S304 (that is, the timing duration between the time the microcontroller receives the timing signal and the time the microcontroller receives the first system time) to obtain the second system time, that is, (Thost2=Thost1+dt), which can ensure that the time accuracy between the microcontroller and the main controller is at the microsecond level.

[0050] In the embodiment of the present application, the main controller can achieve real-time hardware synchronization with the microcontroller. After the system time is synchronized, the second system time of the microcontroller can be made more accurate.

[0051] In some embodiments, the above steps S301-S304 may be executed at intervals of the first duration, thereby periodically performing hardware real-time synchronization on the microcontroller and the main controller to further improve the accuracy of the second system time of the microcontroller. It should be noted that the specific length of the first duration is not limited in this embodiment of the present application.

[0052] S202 , the microcontroller controls the inertial measurement unit and the camera to perform hardware real-time synchronization.

[0053] The microcontroller controls the inertial measurement unit (IMU) and the camera for real-time hardware synchronization, synchronizing the time the IMU collects inertial measurement data with the time the camera collects image data. This facilitates matching the inertial measurement data with the image data based on the temporal relationship. For example, if the IMU collects inertial measurement data 1 at time 1 and the camera also collects image data 1 at time 1, then inertial measurement data 1 and image data 1 are data collected at the same time, and inertial measurement data 1 corresponds to image data 1.

[0054] The microcontroller controls the inertial measurement unit and the camera to perform hardware real-time synchronization, which can also refer to the method shown in Figure 4 below. Figure 4 shows a flow chart of a time synchronization method provided in an embodiment of the present application. The method can be a detailed description of the aforementioned S202. The method includes:

[0055] S401, the microcontroller controls the inertial measurement unit to start up to collect inertial measurement data, receives the inertial measurement data and an interrupt signal from the inertial measurement unit and records a first timestamp, and generates a synchronization trigger signal according to the interrupt signal and sends it to the camera.

[0056] In some embodiments, the inertial measurement unit may generate an interrupt signal when the inertial measurement data is collected. The microcontroller may record a first timestamp upon receiving the inertial measurement data and the interrupt signal. The first timestamp may indicate the time when the inertial measurement data is collected.

[0057] In some embodiments, the microcontroller determines a synchronization trigger signal based on a first frame rate and a second frame rate, where the first frame rate is a frame rate at which the inertial measurement unit collects inertial measurement data and the second frame rate is a frame rate at which the camera collects image data.

[0058] Due to delays such as triggering and transmission delays, the timing between the camera capturing images and the inertial measurement unit collecting inertial measurement data does not match, resulting in a time difference between the camera and the inertial measurement unit. To reduce this time difference, the inertial measurement unit and camera can be synchronized using a synchronous trigger signal. This means that real-time hardware synchronization between the inertial measurement unit and the camera is achieved through the microcontroller, improving the accuracy of time synchronization.

[0059] In some implementations, the first frame rate may be obtained from an inertial measurement unit. For example, the first frame rate may be determined based on an interrupt signal sent by the inertial measurement unit.

[0060] In some implementations, a second frame rate corresponding to the camera may be acquired from the camera, where the second frame rate may be lower than the first frame rate.

[0061] It is understandable that, in practical applications, the first frame rate and the second frame rate may also be determined in other ways.

[0062] In some embodiments, the frequency of the synchronization trigger signal may be the lowest common divisor of the first frame rate and the second frame rate. In some embodiments, the frequency of the synchronization trigger signal is the same as the second frame rate corresponding to the camera.

[0063] For example, as shown in Figures 5 and 6, the first frame rate is the frame rate of the interrupt signal of the inertial measurement unit, and the first frame rate is 4 times the frequency of the synchronization trigger signal; the second frame rate is the shutter frequency of the first camera or the shutter frequency of the second camera, and the second frame rate is the same as the frequency of the synchronization trigger signal.

[0064] S402: After receiving the synchronous trigger signal, the camera performs an exposure operation to collect image data.

[0065] In some embodiments, a camera can begin exposure to capture image data upon receiving a synchronization trigger signal. Specifically, the synchronization trigger signal is used directly as an exposure start signal for each camera, and each camera begins opening its shutter to expose and capture image data upon receiving the synchronization trigger signal. In some embodiments, the camera can send the exposure start time to a microcontroller, which receives the exposure start time and records a second timestamp, which can be used to indicate the exposure start time. Alternatively, the camera can directly send the second timestamp to the microcontroller. Alternatively, the camera may not send the exposure start time or the second timestamp to the microcontroller, but instead the microcontroller determines the exposure start time based on the synchronization trigger signal and records the second timestamp. Because in this synchronization method, the exposure start time of one or more cameras coincides with the time when the inertial measurement unit (IMU) collects inertial measurement data, this method of synchronizing the IMU with the cameras can be referred to as exposure start point synchronization.

[0066] In some embodiments, as shown in FIG5 , at least one camera includes a first camera and a second camera. The microcontroller may send a synchronization trigger signal to the first camera and the second camera. Upon receiving the synchronization trigger signal, the first camera and the second camera begin exposure operations and capture image data.

[0067] In some embodiments, upon receiving the synchronization trigger signal, the camera determines an exposure start time based on the synchronization trigger signal and the camera's exposure duration, begins collecting image data at the exposure start time, and sends the exposure start time to the microcontroller, which receives the exposure start time and records a second timestamp. Alternatively, the camera may directly send the second timestamp to the microcontroller. In some embodiments, the exposure start time may be determined based on the time the camera receives the synchronization trigger signal, the period of the synchronization trigger signal, and the camera's exposure duration, where the exposure start time = the time the synchronization trigger signal is received + the period of the synchronization trigger signal - the camera's exposure duration / 2. In some embodiments, the period of the synchronization trigger signal may be equal to the period at which the camera collects image data. Since, in this synchronization method, the exposure center time of one or more cameras is the same as the time when the inertial measurement unit collects inertial measurement data, this synchronization method between the inertial measurement unit and the camera may be referred to as exposure center point synchronization.

[0068] The exposure time refers to the length of time that light is received. In practical applications, the exposure time can be adjusted based on changes in the brightness of the shooting environment. In dark environments, the exposure time can be increased to increase the brightness of the image. In bright environments such as strong sunlight, the exposure time can be reduced to reduce the brightness of the image and prevent overexposure. In Figure 6, the block diagram corresponding to the first camera and the block diagram corresponding to the second camera are of different sizes. The longer the block diagram, the longer the exposure time. In other words, the exposure time of different cameras can also be different. From the perspective of visual applications, there will be a large phase difference between the inertial measurement unit and the camera, and due to the influence of the exposure time, the difference in this phase difference is not fixed. Therefore, the exposure center point alignment method can be adopted, that is, the interrupt signal of the inertial measurement unit is always at the center point of the camera's exposure time.

[0069] In some embodiments, the first and second cameras utilize global exposure. As shown in FIG6 , after the rising edge of the synchronization trigger signal, the first camera begins exposure after a time period t2. Here, t1 represents the period during which the first camera captures one frame of image data, which is also the period of the synchronization trigger signal. If the second frame rate of the first camera is 40, then time t1 = 25 ms (milliseconds), and the exposure duration of the first camera is Texp. The center point of the exposure duration is Texp_a / 2, and the starting exposure time of the next frame of the first camera is t2 = t1 - Texp_a / 2 = 25 - Texp_a / 2. Similarly, the starting exposure time of the next frame of the second camera is t3 = t1 - Texp_b / 2 = 25 - Texp_b / 2.

[0070] Global exposure means that when the camera's aperture is open, all pixels on the image sensor are exposed simultaneously. When the aperture is closed, all pixels are exposed simultaneously and then output as pixel data. In other words, a global exposure camera can capture the entire image of an object in one shot and then output it.

[0071] In some embodiments, the exposure mode of the first and second cameras is rolling shutter exposure. As shown in Figure 7, the horizontal axis t is the time axis, the vertical axis ra is the row address axis, AB is the single-row exposure time t4, B is the time t5 when the first row of the photosensitive chip is first read, and BC is the time interval t6 between reading all rows of pixels in the photosensitive chip. For example, for a resolution of 1280×720, this is the time required to read all 720 rows of data. The exposure center point of the first camera, Tn_a, = B + (CB - BA) / 2 = t5 + (t6 - t4) / 2. If t1 is the period for the first camera to capture one frame of image data, and the second frame rate of the first camera is 40, then time t1 = 25 ms, and the starting exposure time of the next frame of the first camera, t2, = t1 - Tn_a = 25 - Tn_a. Similarly, the starting exposure time of the next frame of the second camera, t2, = t1 - Tn_b = 25 - Tn_b.

[0072] Rolling shutter exposure uses a row-by-row scanning method. After all pixels in the previous row are exposed simultaneously, all pixels in the next row are exposed simultaneously until all rows are exposed. Because each row of pixels in rolling shutter exposure takes a different amount of time to capture, the image may shift when capturing moving objects. When the camera uses rolling shutter exposure, the output image is a parallelogram, with the exposure center at the center of the parallelogram.

[0073] S403 , the microcontroller receives the exposure start time of the camera and records a second timestamp.

[0074] The second timestamp may be used to indicate the exposure start time of the camera.

[0075] In an embodiment of the present application, the microcontroller controls the inertial measurement unit to start up to collect inertial measurement data, receives the inertial measurement data and an interrupt signal of the inertial measurement unit and records a first timestamp, generates a synchronization trigger signal according to the interrupt signal and sends it to the camera, so that the camera performs an exposure operation to collect image data after receiving the synchronization trigger signal, and the microcontroller receives the exposure start time of the camera and records a second timestamp. That is, the microcontroller can realize real-time hardware synchronization between the inertial measurement unit and the camera based on the synchronization trigger signal, thereby improving the accuracy of the time synchronization between the inertial measurement unit and the camera, and thereby enabling the inertial measurement data collected by the inertial measurement unit to accurately correspond to the image data collected by the camera.

[0076] S203: The main controller matches the inertial measurement data of the inertial measurement unit with the image data of the camera.

[0077] The main controller can match the inertial measurement data of the inertial measurement unit with the image data of the camera, and then perform other services such as visual positioning and mapping based on the matched inertial measurement data and image data.

[0078] The main controller matches the inertial measurement data of the inertial measurement unit with the image data of the camera, which can refer to the method shown in Figure 8 below. Figure 8 shows a flow chart of a time synchronization method provided by an embodiment of the present application. This method can be described in detail in S203 above. The method includes:

[0079] S801 , the microcontroller sends inertial measurement data, a first timestamp, and a second timestamp to the main controller, and the camera sends image data to the main controller.

[0080] In some embodiments, when the microcontroller obtains the first timestamp and inertial measurement data, the microcontroller may send the first timestamp and the inertial measurement unit inertial measurement data to the main controller; and when the microcontroller obtains the second timestamp, the microcontroller may send the second timestamp to the main controller. It is also understood that the microcontroller may also send the first timestamp, the second timestamp, and the inertial measurement data to the main controller upon obtaining the first timestamp, the second timestamp, and the inertial measurement data.

[0081] In some embodiments, the microcontroller may send a second timestamp corresponding to each camera to the main controller based on a camera interface corresponding to each camera. In some embodiments, the camera includes a first camera and a second camera, and the microcontroller may send a second timestamp corresponding to the first camera to the main controller based on the camera interface corresponding to the first camera, and send a second timestamp corresponding to the second camera to the main controller based on the camera interface corresponding to the second camera.

[0082] Different cameras have different camera interfaces, and each camera interface can be used to identify the camera and its corresponding data. In some embodiments, the device interface can be a Universal Serial Bus (USB) interface, a Bluetooth transmission interface, an Ethernet interface, an HDMI interface, a UART interface, a ModBus interface, etc. It should be noted that the embodiments of this application do not limit the specific type of camera interface.

[0083] In some implementations, the camera may send the collected image data to the main controller based on a camera interface corresponding to the camera.

[0084] S802: The main controller matches the first timestamp, the inertial measurement data of the inertial measurement unit, the second timestamp, and the image data to associate the correspondence between the inertial measurement data of the inertial measurement unit and the image data.

[0085] Since the first timestamp can indicate the moment when the inertial measurement unit collects inertial measurement data, and the second timestamp can indicate the moment when the camera collects image data, the image data collected by each camera can be matched with the inertial measurement data based on the first timestamp and the second timestamp corresponding to each camera, thereby determining the timing relationship between the inertial measurement data and the image data.

[0086] In some embodiments, the main controller may store the first timestamp corresponding to the inertial measurement data, and store the second timestamp corresponding to each camera corresponding to the image data acquired by each camera. Since the first timestamp and the second timestamp are both recorded by the microcontroller, and the microcontroller has been synchronized with the main controller, the inertial measurement unit and the camera in real time by hardware, the timing relationship between the inertial measurement data and the image data acquired by each camera is also accurate.

[0087] In some embodiments, the multi-sensor time synchronization system includes a camera, such as the first camera, and the main controller can directly match the second timestamp obtained from the microcontroller with the image data obtained from the camera.

[0088] In some embodiments, the multi-sensor time synchronization system includes multiple cameras, such as a first camera and a second camera, and the main controller can match the second timestamp corresponding to the first camera obtained from the microcontroller with the image data obtained from the first camera, and match the second timestamp corresponding to the second camera obtained from the microcontroller with the image data obtained from the second camera.

[0089] In an embodiment of the present application, hardware real-time synchronization is first performed through the main controller and the microcontroller, and then hardware real-time synchronization between the inertial measurement unit and the camera is achieved through the microcontroller, which greatly improves the accuracy of time synchronization between the inertial measurement unit and the camera, and also improves the accuracy of establishing the association between the inertial measurement data of the inertial measurement unit and the image data of the camera.

[0090] In an embodiment of the present application, since the microcontroller and the main controller can achieve real-time hardware synchronization, and the microcontroller can also achieve real-time hardware synchronization between the inertial measurement unit and the camera through a synchronization trigger signal, the microcontroller can send a first timestamp and inertial measurement data to the main controller. The first timestamp can indicate the moment when the inertial measurement unit collects inertial measurement data. The microcontroller can also send a second timestamp corresponding to the camera to the main controller, and use the second timestamp to indicate the moment when each camera collects image data. Therefore, the main controller can accurately match the image data collected by the camera with the inertial measurement data collected by the inertial measurement unit based on the first timestamp and the second timestamp.

[0091] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0092] Figure 9 shows a schematic structural diagram of a self-moving device provided in an embodiment of the present application. The self-moving device 900 includes at least part of the structure of the system shown in Figure 1. In some embodiments, the self-moving device 900 may include a lawn mowing robot, a sweeping robot, a companion robot, or a guide robot, etc., which can move on their own. Of course, in actual applications, the self-moving device 900 may also include other devices that can move on their own, or the self-moving device 900 may include more structures than Figure 1. In some embodiments, the self-moving device 900 may include a main controller 140, a microcontroller 130, an inertial measurement unit 110, and a camera 120.

[0093] The main controller 140 is configured to send the first system time to the microcontroller 130 .

[0094] The microcontroller 130 is configured to receive the first system time and perform time synchronization with the main controller 140 based on the first system time.

[0095] The microcontroller 130 is further configured to control the inertial measurement unit 110 to start up to collect inertial measurement data, receive the inertial measurement data and interrupt signal from the inertial measurement unit 110 and record a first timestamp; and generate a synchronization trigger signal according to the interrupt signal and send it to the camera 120 .

[0096] The camera 120 is configured to perform an exposure operation to collect image data after receiving a synchronous trigger signal.

[0097] The microcontroller 130 is further configured to receive the exposure start time from the camera 120 and record a second timestamp.

[0098] The main controller 140 is further configured to match the first timestamp, the inertial measurement data of the inertial measurement unit 110 , the second timestamp, and the image data to associate the correspondence between the inertial measurement data of the inertial measurement unit 110 and the image data.

[0099] It can be understood that the modules and functions of the mobile device 900 can also be referred to the relevant descriptions in the aforementioned embodiments, and will not be repeated here.

[0100] The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the same. Although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions described in the above embodiments may be modified or some of the technical features thereof may be replaced by equivalents. However, such modifications or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application.

Claims

1. A multi-sensor time synchronization system, characterized in that: The system includes a main controller, a microcontroller, an inertial measurement unit and a camera: The main controller is used to send a first system time to the microcontroller; The microcontroller is used to receive the first system time and perform time synchronization with the main controller based on the first system time; The microcontroller is further used to control the inertial measurement unit to start up to collect inertial measurement data, receive the inertial measurement data and an interrupt signal from the inertial measurement unit and record a first timestamp; and generate a synchronous trigger signal according to the interrupt signal and send it to the camera; The camera is used to perform an exposure operation to collect image data after receiving the synchronization trigger signal; The microcontroller is further used to receive the exposure start time of the camera and record a second timestamp; The main controller is further used to match the first timestamp, the inertial measurement data of the inertial measurement unit, the second timestamp and the image data to associate the corresponding relationship between the inertial measurement data of the inertial measurement unit and the image data.

2. The system according to claim 1, characterized in that The microcontroller is specifically used to determine the synchronization trigger signal based on a first frame rate and a second frame rate, wherein the first frame rate is a frame rate at which the inertial measurement unit collects the inertial measurement data, and the second frame rate is a frame rate at which the camera collects the image data.

3. The system according to claim 2, characterized in that The camera is specifically configured to start collecting the image data when receiving the synchronization trigger signal.

4. The system according to claim 2, characterized in that The camera is specifically used to determine the exposure start time based on the synchronization trigger signal and the exposure duration of the camera when receiving the synchronization trigger signal, start collecting the image data at the exposure start time, and send the exposure start time to the microcontroller.

5. The system according to claim 1, wherein: The camera includes a master camera and a slave camera.

6. The system according to claim 5, characterized in that The microcontroller is specifically used to obtain the second timestamp corresponding to the master camera and the second timestamp corresponding to the slave camera, and send the second timestamp corresponding to the master camera and the second timestamp corresponding to the slave camera to the master controller; The main controller is specifically configured to make the image data collected by the main camera correspond to the second timestamp of the main camera, and make the image data collected by the slave camera correspond to the second timestamp of the slave camera.

7. The system according to claim 1, characterized in that The main controller is specifically used to obtain the first system time and send the first system time and timing signal to the microcontroller; The microcontroller is specifically configured to start timing when the timing signal is received, stop timing when the first system time is received, and determine the second system time of the microcontroller based on the timing duration and the first system time.

8. A self-propelled device, characterized in that: Comprising a system as described in any one of claims 1-7.

9. The self-moving device according to claim 8, characterized in that: The self-moving device includes a lawn mowing robot.

10. The self-moving device according to claim 8 or 9, characterized in that: The camera includes an RGB camera, a fisheye camera, a panoramic camera or a depth camera.

Images