Sensor adapter

Abstract

The present specification provides a sensor adapter (100) configured for synchronizing data of sensors. The device comprises a light sensor (115) configured for detecting ambient light intensity, a visual sensor (110) configured for capturing images, and an inertial measurement unit (IMU) (120) configured for determining inertial related parameters. The device also includes a processor (105) in communication with the light sensor, the visual sensor, and the IMU. The processor is then configured for determining (205), using data received from the light sensor, ambient light intensity; setting (210) an initial exposure time of the visual sensor based on the determined ambient light intensity; determining (215) a mid-point exposure time of the initial exposure time; determining (220) a sampling time of the IMU based on the determined mid-point exposure time; and adjusting (225) the initial exposure time so that a mid-point exposure time of the adjusted exposure time aligns with the sampling time of the IMU.

Description

SENSOR ADAPTERTechnical Field

[0001] The present invention relates generally to synchronizing data of different devices and, in particular, to a sensor adapter configured for synchronizing data from different sensors.Background

[0002] Devices using multiple sensors, such as cameras, inertial measurement units (IM Us), and light detection and ranging (LiDAR) sensors, are widely used in diverse fields, such as mobile robotics, autonomous driving, and virtual reality. The data from these sensors need to be fused for the data to be useful. Most data fusion methods require accurately aligning the timestamps of all the measurement data to a single clock. Consequently, high-precision time synchronization is crucial for effective data fusion in multi-sensor systems.

[0003] Various approaches have been proposed for time synchronization and can be generally categorized into two groups. The first group assumes that the time offsets between sensors’ clock systems are stable and constant. Based on this assumption, it is possible to develop a framework of temporal calibration algorithms to reduce or eliminate data synchronization errors among multiple sensors. However, this approach results in errors when the time offsets are dynamically shifting, which often occurs in real-world situations.

[0004] The second group assumes dynamic shifts of systems’ clocks, which requires addressing time synchronization at the hardware level or applications requiring high-precision data fusion. It is crucial in this approach to compensate for the temporal offset between the respective clock references before finer alignments are applied. A common approach to synchronizing multiple clocks is to run an NTP (Network Time Protocol) server on one device and adjust other devices to the reference device. However, the method may suffer from significant latency. Improved techniques such as TICSync have been developed to increase efficiency, but fluctuations in calculating the offset values between time references still exist, compromising the accuracy and reliability of the synchronization. There are developments to this approach. However, while these developments have shown significant improvement in accuracy, their design is often sensor-specific and limited to synchronization between sensor pairs. When new sensors are introduced, the systems still struggle to synchronize the time between different clock references. A more adaptive architecture that can synchronize multiple sensors is needed.

[0005] Figs. 7A and 7B show conventional arrangements for synchronizing data of a visual sensor (e.g., a camera) and an inertial measurement unit (IMU). Fig. 7A shows one conventional approach of simultaneously triggering both the visual sensor and the IMU periodically by sending a synchronization signal (not shown as the synchronization signals correspond to the IMU Start Polling markers). Each synchronization signal commences the sampling of the IMU (indicated by IMU Start Polling markers, which are the arrows pointing up) and the exposure of the visual sensor (indicated by the rising edge of the pulse marked as camera exposure duration). The shading around the IMU Start Polling markers indicates the uncertainty as to the exact sampling time of the IMU, as the time between an IMU receiving a synchronization signal and commencing sampling is unknown. Further, although synchronization signals may be sent according to the sampling frequency of the IMU (e.g., at 100Hz), the visual sensor typically has a lower frequency sampling rate of 25Hz. The visual sensor therefore cannot be triggered after every synchronization signal.

[0006] As shown in Fig. 7A, the visual sensor is sampled at the mid-exposure point of the camera exposure time, while the IMU is sampled periodically in accordance with the synchronization signal. One of the IMU sampling time is marked as ti and the mid-exposure sampling of the camera relevant to this IMU sampling time ti is marked as tc. There is a total time discrepancy of da between the IMU sampling time ti and the mid-exposure sampling of the camera tc. Accordingly, this conventional approach does not provide good synchronization between the visual sensor data and the IMU data. This synchronization problem is exacerbated when the camera exposure duration varies due to varying ambient light intensity, when the camera is in automatic mode.

[0007] Fig. 7B shows another conventional approach that attempts to provide better synchronization between the visual sensor and the IMU. Fig. 7A shows that there is a time delay of de between the commencement of the camera exposure and the camera sampling time tc. This other conventional approach aims to better synchronize the visual sensor and the IMU by removing the time delay de. To do so, this conventional approach triggers the camera exposure and predicts the mid-point exposure of the camera. This conventional approach then attempts to align the IMU sampling time to the mid-point exposure of the camera by triggering the IMU sampling accordingly. This conventional approach provides better synchronization, but there is typically a time discrepancy of di between the camera sampling and the IMU sampling. Further, this other conventional approach does not take into account the varying camera exposure time due to varying ambient light intensity, resulting in varying time delay de. This then results in worse time discrepancy di when the mid-point exposure prediction is based on a previous mid-point exposure (i.e. , a previous time delay dc).Summary

[0008] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0009] Disclosed are arrangements which seek to address the above problems by using a sensor adapter configured for controlling the operation of different sensors so that the sampling times of different sensors are synchronized.

[0010] According to a first aspect of the present disclosure, there is provided a sensor adapter configured for synchronizing data of sensors, the device comprising: a light sensor configured for detecting ambient light intensity; a visual sensor configured for capturing images; an inertial measurement unit (I MU) configured for determining inertial related parameters; a processor in communication with the light sensor, the visual sensor, and the IMU, the processor configured for: determining, using data received from the light sensor, ambient light intensity; setting an initial exposure time of the visual sensor based on the determined ambient light intensity; determining a mid-point exposure time of the initial exposure time; determining a sampling time of the IMU based on the determined mid-point exposure time; and adjusting the initial exposure time so that a mid-point exposure time of the adjusted exposure time aligns with the sampling time of the IMU.

[0011] According to another aspect of the present disclosure, there is provided a method of synchronizing data of sensors, the sensors include a visual sensor configured for capturing images and an IMU configured for determining inertial related parameters, the method comprising: determining, using data received from a light sensor, ambient light intensity; setting an initial exposure time of the visual sensor based on the determined ambient light intensity; determining a mid-point exposure time of the initial exposure time; determining a sampling time of the IMU based on the determined mid-point exposure time; and adjusting the initial exposure time so that a mid-point exposure time of the adjusted exposure time aligns with the sampling time of the IMU.

[0012] According to another aspect of the present disclosure, there is provided a computer program product including a computer readable medium having recorded thereon a computer program for implementing any one of the methods described above.

[0013] Other aspects are also disclosed.Brief Description of the Drawings

[0014] Some aspects of the prior art and at least one embodiment of the present invention will now be described with reference to the drawings, in which:

[0015] Figs. 7A and 7B show a prior art arrangement for sampling different sensors;

[0016] Figs. 1 A and 1 B form a schematic block diagram of a sensor adapter upon which arrangements described can be practiced;

[0017] Fig. 2 is a flow diagram of a method of synchronizing different sensors as executed by the sensor adapter shown in Figs. 1A and 1B;

[0018] Fig. 3 shows a diagram of an ideal situation when the sampling times of an IMU align with the sampling times of a visual sensor;

[0019] Fig. 4 shows a comparison of time alignment discrepancies between a method according to the present disclosure and the conventional methods;

[0020] Fig. 5 shows various signals for synchronizing the sampling time of different sensors; and

[0021] Fig. 6 shows a diagram of signals generated to synchronize a LiDAR.Detailed Description

[0022] Where reference is made in any one or more of the accompanying drawings to steps and / or features, which have the same reference numerals, those steps and / or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

[0023] It is to be noted that the discussions contained in the "Background" section and that above relating to prior art arrangements relate to discussions of documents or devices which form public knowledge through their respective publication and / or use. Such should not be interpreted as a representation by the present inventor(s) or the patent applicant that such documents or devices in any way form part of the common general knowledge in the art.

[0024] Figs. 1A and 1 B collectively form a schematic block diagram of a sensor adapter 100 in communication with a visual sensor 110, an IMU 120, a LiDAR 130, and a light sensor 115.The visual sensor 110 (e.g., a camera) is a sensor configured for capturing visual images of a scene. The IMU 120 comprises accelerometers, gyroscopes, magnetometers, or a combination of any of these components for determining force, angular rate, and orientation of a body in which the IMU is disposed. The LiDAR 130 is a device for determining ranges of an object from the LiDAR. The light sensor 115 is configured for determining ambient light intensity. The devices 110, 115, 120, and 130 are well-known in the art and will not be described in detail in the present disclosure.

[0025] As seen in Fig. 1A, the sensor adapter 100 comprises an embedded controller 102. Accordingly, the sensor adapter 100 may be referred to as an “embedded device.” In the present example, the controller 102 has a processing unit (or processor) 105 which is bidirectionally coupled to an internal storage module 109. The storage module 109 may be formed from non-volatile semiconductor read only memory (ROM) 160 and semiconductor random access memory (RAM) 170, as seen in Fig. 1 B. The RAM 170 may be volatile, nonvolatile or a combination of volatile and non-volatile memory.

[0026] The sensor adapter 100 includes a display controller 107, which is connected to a video display 114, such as a liquid crystal display (LCD) panel or the like. The display controller 107 is configured for displaying graphical images on the video display 114 in accordance with instructions received from the embedded controller 102, to which the display controller 107 is connected. The video display 114 and the display controller 107 are optional components that may be used to display the sampling time of different sensors 110, 120, and 130.

[0027] The sensor adapter 100 also includes user input devices 113 which are typically formed by keys, a keypad or like controls. In some implementations, the user input devices 113 may include a touch sensitive panel physically associated with the display 114 to collectively form a touch-screen. Such a touch-screen may thus operate as one form of graphical user interface (GUI) as opposed to a prompt or menu driven GUI typically used with keypad-display combinations. Other forms of user input devices may also be used, such as a microphone (not illustrated) for voice commands or a joystick / thumb wheel (not illustrated) for ease of navigation about menus. The user input devices 113 are optional components that may be used by a user to adjust certain parameters (e.g., the sampling start time) of different sensors 110, 120, and 130, to reset the sensor adapter 100, and the like.

[0028] As seen in Fig. 1A, the sensor adapter 100 also comprises a portable memory interface 106, which is coupled to the processor 105 via a connection 119. The portable memory interface 106 allows a complementary portable memory device 125 to be coupled to the sensor adapter 100 to act as a source or destination of data or to supplement the internal storage module 109. Examples of such interfaces permit coupling with portable memory devices such as Universal Serial Bus (USB) memory devices, Secure Digital (SD) cards, Personal Computer Memory Card International Association (PCM I A) cards, optical disks and magnetic disks.

[0029] The sensor adapter 100 also has a communications interface 108 to permit coupling of the device 100 to a computer or communications network 120 via a connection 121. The connection 121 may be wired or wireless. For example, the connection 121 may be radio frequency or optical. An example of a wired connection includes Ethernet. Further, an example of wireless connection includes Bluetooth™ type local interconnection, Wi-Fi (including protocols based on the standards of the IEEE 802.11 family), Infrared Data Association (IrDa) and the like. The communications interface 108 also couples the sensor adapter 100 with the visual sensor 110, the IMU 120, and the LiDAR 130 to enable bi-directional communication between the sensor adapter 100 and the sensors 110, 120, and 130. The light sensor 115 is also coupled to the sensor adapter 100 via the communications interface 108 to enable the sensor adapter 100 to receive data (e.g., ambient light intensity) from the light sensor 115.

[0030] The sensor adapter 100 can control the visual sensor 110 by sending a control signal with exposure commencement time and duration to the visual sensor 110. In response to receiving such a control signal, the visual sensor 110 commences exposure of the visual sensor 110 for a duration of time to capture a scene in accordance with the received control signal.

[0031] The sensor adapter 100 can control the IMU by sending a control signal

[0032] The method described hereinafter may be implemented using the embedded controller 102, where the processes of Fig. 2 may be implemented as one or more software application programs 133 executable within the embedded controller 102. The sensor adapter 100 of Fig. 1A implements the described methods. In particular, with reference to Fig. 1B, the steps of the described methods are effected by instructions in the software 133 that are carried out within the controller 102. The software instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules perform thedescribed methods and a second part and the corresponding code modules manage a user interface between the first part and the user.

[0033] The software 133 of the embedded controller 102 is typically stored in the non-volatile ROM 160 of the internal storage module 109. The software 133 stored in the ROM 160 can be updated when required from a computer readable medium. The software 133 can be loaded into and executed by the processor 105. In some instances, the processor 105 may execute software instructions that are located in RAM 170. Software instructions may be loaded into the RAM 170 by the processor 105 initiating a copy of one or more code modules from ROM 160 into RAM 170. Alternatively, the software instructions of one or more code modules may be preinstalled in a non-volatile region of RAM 170 by a manufacturer. After one or more code modules have been located in RAM 170, the processor 105 may execute software instructions of the one or more code modules.

[0034] The application program 133 is typically pre-installed and stored in the ROM 160 by a manufacturer, prior to distribution of the sensor adapter 100. However, in some instances, the application program 133 may be supplied to the user encoded on one or more CD-ROM (not shown) and read via the portable memory interface 106 of Fig. 1A prior to storage in the internal storage module 109 or in the portable memory 125. In another alternative, the software application program 133 may be read by the processor 105 from the network 120, or loaded into the controller 102 or the portable storage medium 125 from other computer readable media. Computer readable storage media refers to any non-transitory tangible storage medium that participates in providing instructions and / or data to the controller 102 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, USB memory, a magneto-optical disk, flash memory, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the sensor adapter 100. Examples of transitory or nontangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the sensor adapter 100 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like. A computer readable medium having such software or computer program recorded on it is a computer program product.

[0035] The second part of the application programs 133 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 114 of Fig. 1A. Through manipulationof the user input device 113 (e.g., the keypad), a user of the sensor adapter 100 and the application programs 133 may manipulate the interface in a functionally adaptable manner to provide controlling commands and / or input to the applications associated with the Glll(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via loudspeakers (not illustrated) and user voice commands input via the microphone (not illustrated).

[0036] Fig. 1 B illustrates in detail the embedded controller 102 having the processor 105 for executing the application programs 133 and the internal storage 109. The internal storage 109 comprises read only memory (ROM) 160 and random access memory (RAM) 170. The processor 105 is able to execute the application program 133 stored in one or both of the connected memories 160 and 170. When the sensor adapter 100 is initially powered up, a system program resident in the ROM 160 is executed. The application program 133 permanently stored in the ROM 160 is sometimes referred to as “firmware”. Execution of the firmware by the processor 105 may fulfil various functions, including processor management, memory management, device management, storage management and user interface.

[0037] The processor 105 typically includes a number of functional modules including a control unit (Oil) 151, an arithmetic logic unit (ALU) 152, a digital signal processor (DSP) 153 and a local or internal memory comprising a set of registers 154 which typically contain atomic data elements 156, 157, along with internal buffer or cache memory 155. One or more internal buses 159 interconnect these functional modules. The processor 105 typically also has one or more interfaces 158 for communicating with external devices via system bus 181 , using a connection 161.

[0038] The application program 133 includes a sequence of instructions 162 though 163 that may include conditional branch and loop instructions. The program 133 may also include data, which is used in execution of the program 133. This data may be stored as part of the instruction or in a separate location 164 within the ROM 160 or RAM 170.

[0039] In general, the processor 105 is given a set of instructions, which are executed therein. This set of instructions may be organised into blocks, which perform specific tasks or handle specific events that occur in the electronic sensor adapter 100. Typically, the application program 133 waits for events and subsequently executes the block of code associated with that event. Events may be triggered in response to input from a user, via the user input devices 113 of Fig. 1A, as detected by the processor 105. Events may also be triggered in response to other sensors and interfaces in the electronic sensor adapter 100.

[0040] The execution of a set of the instructions may require numeric variables to be read and modified. Such numeric variables are stored in the RAM 170. The disclosed method uses input variables 171 that are stored in known locations 172, 173 in the memory 170. The input variables 171 are processed to produce output variables 177 that are stored in known locations 178, 179 in the memory 170. Intermediate variables 174 may be stored in additional memory locations in locations 175, 176 of the memory 170. Alternatively, some intermediate variables may only exist in the registers 154 of the processor 105.

[0041] The execution of a sequence of instructions is achieved in the processor 105 by repeated application of a fetch-execute cycle. The control unit 151 of the processor 105 maintains a register called the program counter, which contains the address in ROM 160 or RAM 170 of the next instruction to be executed. At the start of the fetch execute cycle, the contents of the memory address indexed by the program counter is loaded into the control unit 151. The instruction thus loaded controls the subsequent operation of the processor 105, causing for example, data to be loaded from ROM memory 160 into processor registers 154, the contents of a register to be arithmetically combined with the contents of another register, the contents of a register to be written to the location stored in another register and so on. At the end of the fetch execute cycle the program counter is updated to point to the next instruction in the system program code. Depending on the instruction just executed this may involve incrementing the address contained in the program counter or loading the program counter with a new address in order to achieve a branch operation.

[0042] Each step or sub-process in the processes of the methods described below is associated with one or more segments of the application program 133, and is performed by repeated execution of a fetch-execute cycle in the processor 105 or similar programmatic operation of other independent processor blocks in the electronic sensor adapter 100.

[0043] The sensor adapter 100 also includes a reference clock (not shown) governing the execution of the sequence of instructions in the processor 105. An example of the reference clock signals 510 is shown as pulses in Fig. 5A, where the reference clock signals 510 are generated periodically.

[0044] Fig. 2 shows a flowchart of a method 200 for synchronizing the visual sensor 110 and the IMU 120. The method 200 may be implemented as one or more software application programs 133 executable within the embedded controller 102. The method 200 synchronizes an image captured by the visual sensor 110 with the capturing of inertial measurement data of the IMU 120. As the visual sensor 110 captures a video (i.e. , a set of images), the method 200is performed for each synchronization loop (i.e. , the synchronization of one image of the video with the data of the I MU 120).

[0045] The method 200 commences at step 205 by determining ambient light intensity. The sensor adapter 100 determines the ambient light intensity by receiving the ambient light intensity detected by the light sensor 115. The method 200 then proceeds from step 205 to step 210.

[0046] In step 210, the method 200 sets an initial exposure duration of the visual sensor 110 based on the determined ambient light intensity. The initial exposure time of the visual sensor 110 is determined using the equation:

[0047] Where Tauto is the initial exposure duration when the visual sensor 110 is in automatic mode and Lvis the determined ambient light intensity, f is a function operator on the determined ambient light intensity. In one arrangement, f is a linear regression model. For example, the lower the ambient light intensity, the initial exposure duration is longer. Conversely, when the ambient light intensity is high, then the initial exposure duration is shorter. The method 200 then proceeds from step 210 to step 215.

[0048] In step 215, the method 200 determines a mid-point exposure time of the initial exposure duration. The mid-point exposure tc of the initial exposure duration can be determined by the equation:

[0049] Where to is the initial time of a synchronization loop.

[0050] Steps 210 and 215 are to eliminate the time delay cfc(see Fig. 7A). The method 200 then proceeds from step 215 to step 220,

[0051] In step 220, the method 200 determines a sampling time of the IMU 120 based on the determined mid-point exposure time. Fig. 3 shows an ideal situation where one of the IMU sampling times (i.e., the arrows pointing up) aligns with the camera sampling time (i.e., the arrows pointing down). The ideal situation is achieved when there is no time discrepancy of di(see Fig. 7B) between the camera sampling and the IMU sampling. Steps 220 and 225 of the method 200 aim to remove the time discrepancy of di.

[0052] The sampling of the IMU can be triggered by the sensor adapter 100 by periodically sending IMU synchronization signals 520 (see Fig. 5) to the IMU 120. As discussed in relation to the sensor adapter 100, computer program execution is governed by the reference clock signals 510. The IMU synchronization signals 520 can only occur according to the reference clock signals 510. Fig. 5A shows that each instance (i.e. , pulse) of the IMU synchronization signals 520 aligns with an instance (i.e., pulse) of the reference clock signals 510. Upon receiving an IMU synchronization signal 520, the IMU 120 in turn samples its components to provide the inertial measurement data after a period of time Tstdr (as shown by the triggering of the IMU sampling 530 (i.e., arrow pointing up)).

[0053] The IMU synchronization signals 520 occur periodically where the IMU synchronization signals 520 are separated by a predetermined IMU time interval Tspas shown in the IMU synchronization signals 520. The predetermined IMU time interval 7}spis adjustable by a user through, for example, the user input device 113. The predetermined IMU time interval Tspis equal to m x Tt, where m is a number of reference clock signals 510, and Ttis the duration of each reference clock signal 510. Ttis a fixed value dependent on the clock used in the sensor adapter 100. Accordingly, a user may adjust the value of the predetermined IMU time interval TiSpbased on the number m of reference clock signals 510.

[0054] Based on the predetermined IMU time interval Tspand the IMU sampling time Tstdr after receiving an IMU synchronization signals 520, the IMU sampling times t * is obtained using the following equation:

[0055] Where to is the initial time of a synchronization loop, and n is a pulse instance of the IMU synchronization signals 520. Accordingly, equation (3) provides the exact times of the IMU 120 capturing inertial measurement data in a synchronization loop.

[0056] Once the sampling times of the IMU 120 are determined, the most suitable sampling time of the IMU 120 that aligns with the mid-point exposure tc(determined by equation (2)) is determined. The following equations determine the most suitable sampling time of the IMU 120:

[0057] Where n* indicates the most suitable sampling time of the IMU 120, in particular the instance of the IMU sampling 530. n* can be found using the following equation:

[0058] Entering the value of n* found in equation (5) into equation (3) provides the most suitable IMU sampling 530.

[0059] The method 200 proceeds from step 220 to step 225.

[0060] In step 225, the method 200 adjusts the initial exposure time of the visual sensor 110 so that a mid-point exposure time of the adjusted exposure time aligns with a sampling time tf* of the IMU 120. Step 220 determines the exact time of inertial measurement data to be used when sampling the IMU 120. Step 225 adjusts the exposure time of the visual sensor 110 so that the mid-point exposure time of the visual sensor 110 aligns with the IMU sampling time.The initial exposure time Tauto is adjusted to an optimized exposure duration TstrObe, where Tstrobe can be found using the following equation:

[0061] Where [n*] is the round of n* (which is determined in equation (5)) andspand Tstdr are the parameters discussed in relation to step 220 above. Diagram 530 of Fig. 5 shows the commencement (arrow up at time to and ceasing (arrow down indicated by VfUu) of exposure of the visual sensor 110. Diagram 540 of Fig. 5 shows the mid-point exposure of the optimized exposure duration Tstrobe as the camera sampling time tc, which aligns with an IMU sampling time (shown in diagram 43).

[0062] The method 200 concludes at the conclusion of step 220.

[0063] Once the IMU sampling time of the IMU 120 and the optimized exposure duration Tstrobe of the visual sensor 110 are determined, the sensor adapter 100 synchronizes (for each synchronization loop) the data captured by the visual sensor 110 and the IMU 120 by transmitting the IMU synchronization signals 520 in accordance with the predetermined IMU time interval 7}sPand transmitting control signals to the visual sensor 110 to commence exposure of the visual sensor 110 at time to with duration Tstrobe- As described in step 220, thesampling time of the IMU is determined based on the predetermined time interval TiSp. This operation enables the sensor adapter 100 to receive measured data from both the visual sensor 110 and the IMU 120 at the same time (as the IMU sampling time aligns with the camera sampling time tc).

[0064] In one alternative arrangement, the sensor adapter 100 transmits control signals to the visual sensor 110 to commence exposure of the visual sensor 110 at time to and to cease exposure of the visual sensor 110 after a duration of TstrObe expires.

[0065] Fig. 4 shows time discrepancies between the conventional approaches (the two bars on the left hand side indicated as Traditional Method and Vl-Sensor method) and the present disclosure (the right-most bar indicated as Sensor Adapter Method). The bars show 3 instances of different camera exposure durations of 1.2 ms, 3.2 ms, and 5.2 ms respectively. At each camera exposure duration, the method 200 (i.e. , the Sensor Adapter Method shown in the rightmost bar at each camera exposure duration) shows significantly fewer time alignment discrepancies in comparison to the conventional approaches.

[0066] The sensor adapter 100 also synchronizes the LiDAR 130 by transmitting a pulse-per- second (PPS) signal 610 (shown in Fig. 6) and a global positioning recommended minimum (GPRMC) signal 620. The PPS signal 610 and GPRMC signal 620 align the internal clock of the LiDAR 130 with the reference clock of the sensor adapter 100. Consequently, the pointcloud data generated by the LiDAR 130 includes timestamp messages from the LiDAR internal clock. As the LiDAR internal clock is aligned with the reference clock of the sensor adapter 100, the point-cloud data of the LiDAR 130 can be aligned with the data from the visual sensor 110 and IMU 120 (as both sensors 110 and 120 capture data based on the reference clock of the sensor adapter 100).

[0067] The PPS signals 610 are pulses that are generated every second where each pulse has a high level duration of TH. Following each pulse of the PPS signal 610, a GPRMC signal 620 is transmitted. The GPRMC signal 620 includes a GPRMC message with a string identifier, a sensor adapter reference clock, and a status. In one example, the string identifier may be $GPRMC, the sensor adapter reference clock may be 210719.00, and the status can be either active (A) or void (V). When the LiDAR 130 receives both the PPS signal 610 and the GPRMC signal 620, the LiDAR 130 updates its internal clock at the time (i.e., the rising edge) of receiving a pulse of the PPS signal 610 with corresponding sensor adapter reference clock ofthe GPRMC signal 620. Accordingly, when the point-cloud data of the LiDAR 130 is generated, the data is timestamped with the internal clock of the LiDAR 130 that has been aligned with the reference clock of the sensor adapter 100.

[0068] When the method 200 is performed together with the LiDAR synchronization method, the sensor adapter 100 can synchronize data from the visual sensor 110, the I MU 120, and the LiDAR 130 based on the timestamps of the data of the visual sensor 110, the IMU 120, and the LiDAR 130.Industrial Applicability

[0069] The arrangements described are applicable to the computer and data processing industries and particularly for the synchronization of data of different sensors.

[0070] The foregoing describes only some embodiments of the present invention, and modifications and / or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

[0071] In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of”. Variations of the word "comprising", such as “comprise” and “comprises” have correspondingly varied meanings.

Claims

CLAIMS:

1. A sensor adapter configured for synchronizing data of sensors, the device comprising: a light sensor configured for detecting ambient light intensity; a visual sensor configured for capturing images; an inertial measurement unit (I MU) configured for determining inertial related parameters; a processor in communication with the light sensor, the visual sensor, and the IMU, the processor configured for: determining, using data received from the light sensor, ambient light intensity; setting an initial exposure time of the visual sensor based on the determined ambient light intensity; determining a mid-point exposure time of the initial exposure time; determining a sampling time of the IMU based on the determined mid-point exposure time; and adjusting the initial exposure time so that a mid-point exposure time of the adjusted exposure time aligns with the sampling time of the IMU.

2. The sensor adapter of claim 1 , further comprising: transmitting synchronization signals to the IMU periodically at a predetermined time interval, wherein the sampling time of the IMU is determined based on the predetermined time interval; and controlling a start time and a stop time of the visual sensor based on the adjusted exposure time.

3. The sensor adapter of claim 1 or 2, further comprising: transmitting signals configured for aligning an internal clock of a light detection and ranging (LiDAR) with a reference clock of the sensor adapter.

4. The sensor adapter of claim 3, wherein the signals transmitted to the LiDAR include a pulse-per-second signal and a global positioning recommended minimum (GPRMC) signal, wherein the GPRMC signal includes the reference clock of the sensor adapter and the internal clock of the LiDAR at the time of receiving the pulse-per-second signal is updated based on the received reference clock.

5. The sensor adapter of claim 3 or 4, wherein data from the visual sensor, the IMU, and the LiDAR are synchronized based on timestamps of the data of the visual sensor, the IMU, and the LiDAR.

6. A method of synchronizing data of sensors, the sensors include a visual sensor configured for capturing images and an inertial measurement unit (IMU) configured for determining inertial related parameters, the method comprising: determining, using data received from a light sensor, ambient light intensity; setting an initial exposure time of the visual sensor based on the determined ambient light intensity; determining a mid-point exposure time of the initial exposure time; determining a sampling time of the IMU based on the determined mid-point exposure time; and adjusting the initial exposure time so that a mid-point exposure time of the adjusted exposure time aligns with the sampling time of the IMU.

7. The method of claim 6, further comprising: transmitting synchronization signals to the IMU periodically at a predetermined time interval, wherein the sampling time of the IMU is determined based on the predetermined time interval; and controlling a start time and a stop time of the visual sensor based on the adjusted exposure time.

8. The method of claim 6 or 7, further comprising: transmitting signals configured for aligning an internal clock of a light detection and ranging (LiDAR) with a reference clock of the sensor adapter.

9. The sensor adapter of claim 8, wherein the signals transmitted to the LiDAR include a pulse-per-second signal and a global positioning recommended minimum (GPRMC) signal, wherein the GPRMC signal includes the reference clock of the sensor adapter and the internal clock of the LiDAR at the time of receiving the pulse-per-second signal is updated based on the received reference clock.

10. The method of claim 8 or 9, further comprising synchronizing data from the visual sensor, the IMU, and the LiDAR based on timestamps of the data of the visual sensor, the IMU, and the LiDAR.

11. A computer readable storage medium having a computer program recorded therein, the program being executable by a computer apparatus to make the computer apparatus perform a method of synchronizing data from sensors, the sensors include a visual sensor configured for capturing images and an inertial measurement unit (IMU) configured for determining inertial related parameters, said program comprising: code for determining, using data received from a light sensor, ambient light intensity; code for setting an initial exposure time of the visual sensor based on the determined ambient light intensity; code for determining a mid-point exposure time of the initial exposure time; code for determining a sampling time of the IMU based on the determined mid-point exposure time; and code for adjusting the initial exposure time so that a mid-point exposure time of the adjusted exposure time aligns with the sampling time of the IMU.

12. The computer readable storage medium of claim 11, the program further comprising: code for transmitting synchronization signals to the IMU periodically at a predetermined time interval, wherein the sampling time of the IMU is determined based on the predetermined time interval; and code for controlling a start time and a stop time of the visual sensor based on the adjusted exposure time.

13. The computer readable storage medium of claim 11 or 12, the program further comprising: code for transmitting signals configured for aligning an internal clock of a light detection and ranging (LiDAR) with a reference clock of the sensor adapter.

14. The computer readable storage medium of claim 13, wherein the signals transmitted to the LiDAR include a pulse-per-second signal and a global positioning recommended minimum (GPRMC) signal, wherein the GPRMC signal includes the reference clock of the sensor adapter and the internal clock of the LiDAR at the time of receiving the pulse-per-second signal is updated based on the received reference clock.

15. The computer readable storage medium of claim 13 or 14, the program further comprising code for synchronizing data from the visual sensor, the IMU, and the LiDAR based on timestamps of the data of the visual sensor, the IMU, and the LiDAR.

Images