Method and device for synchronous triggering of multiple cameras

The microcontroller uses the reference clock source to synchronously trigger the camera to acquire images and record timestamps, solving the problem of asynchronous time of multiple cameras and radars, improving the performance and safety of the autonomous driving system, and achieving the correct fusion of multi-sensor data.

CN118921432APending Publication Date: 2024-11-08ROBERT BOSCH GMBH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310508242.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-06
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

In the prior art, multiple cameras and radars have time asynchronous problems when acquiring images, resulting in decision-making errors in the autonomous driving control system and may cause accidents. Existing solutions such as software control and hardware control have problems with delay and non-real-time.

Method used

Using a microcontroller (such as Arduino), the precise time is obtained through the reference clock source, the clock is calibrated, and multiple cameras are triggered to collect images in real time, record the trigger time, and associate the timestamp with the image to ensure the time synchronization between the camera and radar.

Benefits of technology

It realizes precise time synchronization between cameras and radar, improves the operating performance and user experience of the autonomous driving system, and reduces costs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921432A_ABST
    Figure CN118921432A_ABST
Patent Text Reader

Abstract

The invention relates to a device for time synchronization, and the device comprises a single-chip microcomputer which is configured to obtain the precise time of a reference clock source to calibrate the clock of the single-chip microcomputer, synchronously trigger a plurality of cameras to collect images, record the trigger time at the same time, and send the trigger time; and the processor is configured to receive the trigger time from the single chip microcomputer, receive images acquired by the plurality of cameras, and associate the trigger time with the images acquired by the plurality of cameras at the trigger time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application generally relates to assisted driving and autonomous driving control systems, and more specifically, to time synchronization of multiple sensors (e.g., cameras, radars) in assisted driving and autonomous driving control systems. Background Art

[0002] Advanced driver assistance and autonomous driving technologies involve various functions such as environmental perception, planning and decision-making, and assisted driving. Advanced driver assistance and autonomous driving technologies require the use of multiple sensors (e.g., multiple cameras, millimeter-wave radars, ultrasonic radars, lidars, and other devices) to sense information about the surrounding environment, and transmit the sensed data to the processor for data processing for image recognition, visual synchronous positioning and three-dimensional environment reconstruction, identification of obstacles, feasible paths, etc., and based on the recognition results, route planning, driving speed setting, automatic driving control, etc.

[0003] In order to perceive the environment, it is expected that multiple cameras will synchronously capture images, perform image recognition, and analyze images in real time. The results of the image analysis will be integrated with radar sensing data, which will help improve environmental perception capabilities for positioning, path planning, wireless communication, pre- and post-collision warnings, near-real-time road conditions, blind spot warnings, and improved visibility, thereby achieving automation of the driving process and safe decision-making, etc.

[0004] If multiple cameras collect images asynchronously, or the camera and radar are not synchronized in time, correct fusion of multiple sensors may not be achieved, and in particular, it may cause incorrect decisions by the autonomous driving control system and lead to accidents.

[0005] In the prior art, there are mainly two methods to help trigger multiple cameras to synchronously capture images. The first is a software control method, which controls multiple cameras to capture images through software, calls the camera and obtains the image through the camera API, and obtains the API call time from the operating system. The disadvantage of this soft trigger based on the camera API is that since the camera trigger action is controlled by software, the software only knows the camera API call time, and cannot estimate the delay between the camera API call time and the actual triggering of the camera shutter, so it is impossible to obtain the actual time of the camera shutter being triggered to capture the image. The second method is a hardware control method, for example, using Raspberry pi to control multiple cameras to capture images, Raspberry pi controls the camera trigger cable by flipping the level, and records the time when the camera shutter is triggered to shoot. Its disadvantage is that Raspberry pi itself is not a real-time operating system, and the picture shooting time recorded by it is always delayed.

[0006] Therefore, it is expected to further improve the synchronization between multiple camera shutters and the time synchronization between cameras and radars, so as to better realize the fusion of multiple sensors, improve the operational performance of the autonomous driving system, and improve the user experience. Summary of the invention

[0007] The following summary is provided to introduce selected concepts in a simplified form that are further described in the detailed description that follows. This summary is not intended to highlight key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0008] According to one aspect of the present application, a device for time synchronization is provided, comprising: a single-chip microcomputer, configured to: obtain precise time from a reference clock source, calibrate the clock of the single-chip microcomputer, synchronously trigger multiple cameras to capture images, simultaneously record the trigger time, and send the trigger time; a processor, configured to: receive the trigger time from the single-chip microcomputer, receive the images captured by the multiple cameras, and associate the trigger time with the images captured by the multiple cameras at the trigger time.

[0009] According to one aspect of the present application, a system for time synchronization is provided, comprising: a reference clock source configured to provide precise time; multiple sensors, including multiple cameras for capturing images; and an apparatus for time synchronization described in the above aspects of the present application.

[0010] According to one aspect of the present application, a time synchronization method is provided, comprising: receiving, by a single-chip microcomputer, precise time from a reference clock source to calibrate the clock of the single-chip microcomputer; using the single-chip microcomputer to synchronously trigger multiple cameras to capture images and simultaneously record the trigger time; and associating the trigger time with the images captured by the multiple cameras at the trigger time.

[0011] According to one aspect of the present application, there is provided an apparatus for synchronously acquiring images with multiple cameras, comprising: a timer module, configured to set a timer for a time synchronization period; a precise time acquisition module, configured to obtain precise time from a reference clock source so as to calibrate the clock of the apparatus; a camera hard trigger module, configured to control the pin level of a single-chip microcomputer to flip periodically, triggering the shutters of multiple cameras to capture images at the flipping edge of the pin level; a timestamp recording module, configured to record the timestamp of the flipping of the pin level according to the calibrated clock; and a sending module, configured to send the timestamp of the flipping of the pin level.

[0012] According to one aspect of the present application, a device for time synchronization is provided, comprising: one or more processors; and a memory storing processor-executable instructions, which, when executed by the one or more processors, enable the one or more processors to perform the time synchronization method described in various embodiments of the present application.

[0013] According to one aspect of the present application, a machine-readable storage medium is provided, which stores processor-executable instructions. When the processor-executable instructions are executed, one or more processors execute the time synchronization method described in various embodiments of the present application.

[0014] According to one aspect of the present application, a computer program product is provided, which includes processor-executable instructions. When the processor-executable instructions are executed by one or more processors, the one or more processors execute the time synchronization method described in various embodiments of the present application.

[0015] Compared with the soft trigger based on the camera API, the solution of the present application directly controls the shutter of the camera and obtains the time of triggering the shutter to shoot in real time. Compared with the hard trigger based on non-real-time operating systems such as Raspberry pi, the single-chip microcomputer (preferably Arduino) used in the present application is a real-time operating system with fast timing interrupt response, which can always trigger the camera synchronously in real time to ensure the accuracy of the trigger timestamp. Therefore, the present application can realize the synchronous triggering of multiple cameras to capture images with a precise clock, and the time synchronization of the camera and the radar, so as to better realize the correct fusion of the sensing results of multiple sensors, improve the operating performance of the autonomous driving system, improve the user experience, and the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] A further understanding of the nature and advantages of the present application may be achieved by referring to the following drawings. In the accompanying drawings, similar components or features may have the same reference numerals.

[0017] Figure 1 A block diagram of a time synchronization system 100 for a sensing system is shown according to one embodiment.

[0018] Figure 2 A flow chart 200 is shown for time synchronization of a sensing system according to one embodiment.

[0019] Figure 3 A flow chart 300 is shown for fusing sensed data of multiple sensors of a sensing system according to one embodiment.

[0020] Figure 4 A schematic diagram of an apparatus 400 for performing time synchronization in a sensing system according to an embodiment is shown.

[0021] Figure 5 A block diagram of a computer system 500 for time synchronization is shown according to one embodiment. DETAILED DESCRIPTION

[0022] The subject matter described herein will now be discussed with reference to example implementations.

[0023] Figure 1 A block diagram of a time synchronization system 100 for a sensing system is shown according to one embodiment.

[0024] Figure 1 The time synchronization system 100 shown includes a reference clock source 102 , a switch 104 , a single-chip microcomputer 106 , a processor 108 , a plurality of radars 110 , a plurality of cameras 112 , and a USB hub 114 .

[0025] The reference clock source 102 provides accurate time (e.g., UTC) for the entire time synchronization system 100. The reference clock source 102 includes satellite clocks, network time protocol (NTP) servers, atomic clocks, observatory clocks, and Internet clocks, etc. UTC is Coordinated Universal Time, also known as Universal Time, World Standard Time, and International Coordinated Time. For example, the reference clock source 102 can be connected to the switch 104 via an Ethernet cable 103-1, and connected to the single-chip computer 106, the processor 108, and the plurality of radars 110 via Ethernet cables 103-2, 103-3, and 103-4 through the switch 104.

[0026] After the single-chip microcomputer 106 performs clock calibration according to the precise time from the reference clock source 102, the level of the pins (PIN 106-1, GND 106-2) of the single-chip microcomputer 106 is controlled to flip periodically (for example, every 100ms) (from 0V to 3.3V, or from 3.3V to 0V), and the shutters of multiple cameras 112 are synchronously triggered to capture images via the camera hard trigger cable (TC) 111. The single-chip microcomputer 106 sends the pin level flip timestamp to the processor 108 via a user datagram protocol (UDP) data packet. For example, the single-chip microcomputer 106 can be a micro control unit (MCU), or a system on chip (SOC), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Preferably, the single-chip microcomputer 106 can also be, for example, Arduino.

[0027] Once the processor 108 receives the user datagram protocol (UDP) data packet from the microcontroller 106, it acquires images from the memory of the plurality of cameras 112 through the camera API via the USB hub 114. The processor 108 also interacts and processes data with the plurality of radars 110 through the Ethernet cable 103-5. In addition, the processor 108 also provides power (e.g., 5V power) to the microcontroller 106 through the USB 114-1.

[0028] Multiple radars 110 (eg, millimeter wave radar, ultrasonic radar, lidar, etc.) utilize detection and ranging to obtain parameters such as the distance, direction, altitude, speed, attitude, and even shape of a target, thereby detecting, identifying, and tracking the target.

[0029] The USB hub 114 (eg, a USB 3.0 HUB) is used to provide power to the plurality of cameras 112 via the USB 114 - 3 , and to transmit image data acquired by the plurality of cameras 112 to the processor 108 via the USB 114 - 3 and the USB 114 - 2 .

[0030] Figure 2 A flowchart 200 for time synchronization of a sensing system is shown according to one embodiment. The flowchart 200 begins at block 202 .

[0031] At block 204 , the microcontroller 106 is initialized and a timer for a time synchronization period is set inside the microcontroller 106 .

[0032] At block 206 , the microcontroller 106 sends a synchronization request to the reference clock source 102 .

[0033] At block 208 , the microcontroller 106 waits for the reference clock source 102 to reply.

[0034] At block 210, the microcontroller 106 determines whether the reference clock source 102 has responded. If the microcontroller 106 determines that the reference clock source 102 has responded, then at block 212 the microcontroller 106 receives the precise time from the reference clock source 102, thereby calibrating the clock of the microcontroller 106. If the microcontroller 106 determines that the reference clock source 102 has not responded, then the microcontroller 106 returns to block 206 and continues to repeat the execution.

[0035] At block 214, is it determined whether the shutters of the multiple cameras 112 are ready to be triggered for shooting? If ready, the camera hard trigger is performed at block 216, for example, the microcontroller 106 controls the level of its pin to periodically (for example, every 100ms) flip (for example, from 0V to 3.3V, or from 3.3V to 0V), and the camera shutter is triggered to capture an image via the camera hard trigger cable (TC) 111 at the rising edge and / or falling edge of the pin level. If not ready, the execution is returned to block 212 to continue.

[0036] At block 218 , the microcontroller 106 uses the clock calibrated by the reference clock source 102 to record a pin level transition timestamp of the microcontroller 106 .

[0037] At block 220, the microcontroller 106 sends the pin level flip timestamp to the processor 108 via a user datagram protocol (UDP) data packet. At the same time, the processor 108 receives images from the plurality of cameras 112 and records the images. The processor 108 parses the UDP data packet and associates the pin level flip timestamp received from the microcontroller 106 with the images captured by the plurality of cameras 112 at the pin level flip timestamp (e.g., using each pin level flip timestamp to name the image captured at the corresponding pin level flip time; or, (e.g., using a table) establishes a one-to-one correspondence between each pin level flip timestamp and the image captured at the corresponding pin level flip time).

[0038] At block 222, the microcontroller 106 determines whether the timer for the time synchronization period has expired. If it is determined that the timer has expired, the process returns to block 206 to continue execution, thereby periodically calibrating the clock of the microcontroller 106 with the reference clock source 102, and then the microcontroller 106 continues to synchronously trigger multiple cameras to capture images using the recalibrated clock. If it is determined that the timer has not expired, the process returns to block 214 to continue to repeat execution.

[0039] Figure 3 A flow chart 300 is shown for fusing sensed data of multiple sensors of a sensing system according to one embodiment.

[0040] At block 302, the microcontroller 106 receives the precise time from the reference clock source 102 through the switch 104 to calibrate the clock of the microcontroller 106, and uses the calibrated clock to synchronously trigger the shutters of the multiple cameras 112 to capture images, thereby achieving camera synchronous triggering, as shown above. Figure 2 shown.

[0041] At block 304 , the plurality of radars 110 obtain accurate time from the reference clock source 102 via the switch 104 , thereby calibrating the clocks of the plurality of radars 110 .

[0042] At block 306 , the processor 108 obtains accurate time from the reference clock source 102 via the switch 104 , thereby calibrating the clock of the processor 108 .

[0043] In this embodiment, the reference clock source 102 is first used to perform clock calibration, and then the calibrated clock is used to perform synchronous sensing and fusion of multiple sensors (for example, multiple cameras, multiple radars, etc.) (for example, the results of camera image analysis are fused with radar sensing data), so that the processor 108 can receive road environment information sent from multiple sensors to perform synchronous image recognition, data processing and other functions, which helps to improve environmental perception capabilities, achieve pre-collision and post-collision warnings, obtain near real-time road conditions, perform blind spot warnings and improve visibility, thereby realizing automated driving, safe decision-making, etc.

[0044] Figure 4 A schematic diagram of an apparatus 400 for performing time synchronization in a sensing system according to an embodiment is shown. Figure 4 The device 400 shown can be implemented by software, hardware, or a combination of software and hardware.

[0045] like Figure 4 As shown, the apparatus 400 may include a timer module 402 , a precise time acquisition module 404 , a camera hard trigger module 406 , a timestamp recording module 408 , and a sending module 410 .

[0046] The timer module 402 sets a timer for a time synchronization period.

[0047] The module 404 for obtaining precise time is used to obtain precise time from the reference clock source 102 by periodically (for example, according to the time synchronization period of the timer, such as every few seconds) sending synchronization requests to the reference clock source 102 (for example, a satellite clock, a network time protocol (NTP) server, an atomic clock, an observatory clock, and an Internet clock, etc.) so as to calibrate the clock of the device.

[0048] The camera hard trigger module 406 controls the pin level of the microcontroller 106 to flip periodically (e.g., every 100 ms) (e.g., from 0V to 3.3V, or from 3.3V to 0V), triggering the shutters of multiple cameras to capture images at the rising and / or falling edges of the pin level.

[0049] The timestamp recording module 408 records the pin level flip timestamp of the single-chip microcomputer 106 according to the calibrated clock.

[0050] The sending module 410 sends the pin level flip timestamp of the microcontroller 106 to the processor 108 via a user datagram protocol (UDP) data packet, so as to associate each pin level flip timestamp with the image captured by multiple cameras at the corresponding pin level flip timestamp (for example, each pin level flip timestamp is used to name the image captured at the corresponding pin level flip time; or, (for example, using a table) establish a one-to-one correspondence between each pin level flip timestamp and the image captured at the corresponding pin level flip time).

[0051] Figure 5 A block diagram of a computer system 500 for time synchronization is shown according to one embodiment.

[0052] According to one embodiment, the computer system 500 may include one or more processors 510 that execute one or more processor-executable instructions (i.e., the above-described elements implemented in software form) stored or encoded in a processor-readable storage medium (i.e., memory 520). Figure 5 However, those skilled in the art will appreciate that the computer system 500 may include various other components, such as various communication modules, bus modules, and possible user interface modules.

[0053] In one embodiment, the computer system 500 may be implemented in Figure 1 The time synchronization system 100 shown or Figure 4 In the device 400 shown in the figure, the processor executable instructions are stored in the memory 520, and when one or more processors 510 execute the instructions, the one or more processors 510 execute the above-mentioned combined Figure 2-Figure 3 Describes the various operations.

[0054] According to one embodiment, a non-transitory machine-readable storage medium is provided. The non-transitory machine-readable storage medium may have instructions (ie, the above-mentioned elements implemented in software form), which, when executed by a machine, causes, for example, Figure 1 , Figure 4 and Figure 5 The device shown in the figure performs the above combination in each embodiment of the present application. Figure 1-Figure 5 Describes the various operations and functions.

[0055] According to one embodiment, a computer program product is provided. The computer program product may include instructions (i.e., the above-mentioned elements implemented in software form), which, when executed by a machine, enable, for example Figure 1 , Figure 4 and Figure 5 The device shown in the figure performs the above combination in each embodiment of the present application. Figure 1-5Describes the various operations and functions.

[0056] The specific embodiments described above in conjunction with the accompanying drawings describe exemplary embodiments, but do not represent all embodiments that can be implemented or fall within the scope of protection of the claims. The term "example" used throughout this specification may mean "used as an example, instance or illustration" and does not mean "preferred" or "having advantages" over other embodiments. For the purpose of providing an understanding of the described technology, the specific embodiments include specific details. However, these technologies can be implemented without these specific details. In some instances, in order to avoid making the concepts of the described embodiments difficult to understand, well-known structures and devices are shown in block diagram form.

[0057] The above description of the present disclosure is provided to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications to the present disclosure will be apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other variations without departing from the scope of protection of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the widest range of principles and novel features disclosed herein.

Claims

1. A device for time synchronization, comprising: The microcontroller is configured as: Obtaining accurate time from a reference clock source to calibrate the clock of the microcontroller, Synchronously trigger multiple cameras to acquire images and record the triggering time, and sending the trigger time; The processor is configured to: receive the trigger time from the single-chip microcomputer, receive the images captured by the multiple cameras, and associate the trigger time with the images captured by the multiple cameras at the trigger time.

2. The device according to claim 1, wherein: The single-chip microcomputer synchronously triggers the multiple cameras to capture images, including: controlling the level of the pin of the single-chip microcomputer to flip periodically, and synchronously triggering the multiple cameras to capture images at the flipping edge of the level.

3. The device according to claim 2, wherein: The flip edge includes a rising edge, a falling edge, or both.

4. The device according to claim 1, wherein: The processor configured to associate the trigger time with images acquired by the plurality of cameras at the trigger time is further configured to: naming the image acquired at the trigger time with the trigger time; or A one-to-one correspondence is established between the trigger time and the image collected at the trigger time.

5. The device according to claim 1, wherein: The reference clock source includes a satellite clock, a Network Time Protocol (NTP) server, an atomic clock, an observatory clock, and an Internet clock, and the precise time includes Coordinated Universal Time (UTC).

6. The device according to claim 1, wherein: The single chip microcomputer includes one of the following: a microcontroller unit (MCU), a system on a chip (SOC), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and an Arduino.

7. A system for time synchronization, comprising: Reference clock source, used to provide accurate time; multiple sensors, including multiple cameras for acquiring images; as well as A device according to any one of claims 1 to 6.

8. The system according to claim 7, wherein: The multiple sensors also include multiple radars, which include millimeter wave radars, ultrasonic radars, and laser radars.

9. The system according to claim 8, wherein: The multiple radars receive the precise time to calibrate clocks of the multiple radars.

10. A method for time synchronization, comprising: The single-chip microcomputer receives the precise time from the reference clock source to calibrate the clock of the single-chip microcomputer; Using the single chip microcomputer to synchronously trigger multiple cameras to collect images and simultaneously record the triggering time; as well as The trigger time is associated with the images acquired by the multiple cameras at the trigger time.

11. The method according to claim 10, wherein: The method of synchronously triggering a plurality of cameras to collect images by using the single chip microcomputer comprises: The level of the pin of the single-chip microcomputer is controlled to flip periodically, and the multiple cameras are synchronously triggered to collect images at the flipping edge of the level.

12. The method according to claim 11, wherein: The flip edge includes a rising edge, a falling edge, or both.

13. The method according to claim 10, wherein: Associating the trigger time with the images acquired by the multiple cameras at the trigger time includes: naming the image acquired at the trigger time with the trigger time; or A one-to-one correspondence is established between the trigger time and the image collected at the trigger time.

14. The method according to claim 10, wherein: The reference clock source includes a satellite clock, a Network Time Protocol (NTP) server, an atomic clock, an observatory clock, and an Internet clock, and the precise time includes Coordinated Universal Time (UTC).

15. The method according to claim 10, wherein: The single chip microcomputer includes one of the following: a microcontroller unit (MCU), a system on a chip (SOC), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), and an Arduino.

16. The method according to claim 10, further comprising: The precise time is received by a plurality of radars to calibrate clocks of the plurality of radars.

17. A device for synchronously capturing images with multiple cameras, comprising: A timer module, configured to set a timer for a time synchronization period; A module for obtaining precise time, configured to obtain precise time from a reference clock source so as to calibrate the clock of the device; The camera hard trigger module is configured to control the pin level of the single-chip microcomputer to flip periodically, and trigger the shutters of multiple cameras to capture images at the flip edge of the pin level; A timestamp recording module, configured to record a timestamp of the flip of the pin level according to the calibrated clock; The sending module is configured to send a timestamp of the flipping of the pin level.

18. The device according to claim 17, wherein: The reference clock source includes a satellite clock, a Network Time Protocol (NTP) server, an atomic clock, an observatory clock, and an Internet clock, and the precise time includes Coordinated Universal Time (UTC).

19. The device according to claim 17, wherein: The module for obtaining precise time is further configured to: periodically send the synchronization request to the reference clock source according to the time synchronization period of the timer, and obtain precise time from the reference clock source.

20. The device according to claim 17, wherein: The flip edge includes a rising edge, a falling edge, or both.

21. A device for synchronously acquiring images by multiple cameras, comprising: one or more processors; as well as A memory storing processor-executable instructions which, when executed by the one or more processors, cause the one or more processors to perform a method according to any one of claims 10 to 16.

22. A machine-readable storage medium storing processor-executable instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 10 to 16.

23. A computer program product comprising processor-executable instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 10 to 16.

Citation Information

Cited By

  • Acousto-optic linkage composite blade fluid-solid coupling synchronous monitoring device and method

    CN122217777A