Multi-camera synchronization method, device, equipment and computer-readable storage medium

Abstract

A multi-camera synchronization method, apparatus, device, and computer-readable storage medium. The method includes: a frame synchronization signal generating unit generates a frame synchronization signal and sends it to a timestamp generating unit and N cameras, where N is a positive integer greater than or equal to 2; when the timestamp generating unit receives the frame synchronization signal, it places the timestamp of the current moment into the FIFO queue corresponding to each camera; when each camera receives the frame synchronization signal, it performs a shooting action; when a forwarding unit receives image data obtained by each camera based on the shooting action, it reads the timestamp from the FIFO queue corresponding to the camera, and sends the image data and the read timestamp to the image processing end. Through this application, multiple cameras can synchronously perform shooting actions and the back-end image processing end can accurately determine the image data for synchronous exposure.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-camera synchronization method, apparatus, device, and computer-readable storage medium. Background Art

[0002] In an autonomous driving system, after the back-end processing module receives image data captured by multiple cameras, it generally determines the image data exposed at the same time based on the timestamps attached to the image data for fusion perception.

[0003] However, in the existing solution, each camera independently obtains a timestamp and binds it to the image based on the time point when the output image data is monitored. Due to the different connection line delays between different cameras and the back-end processing module or the differences in exposure time of different cameras, the timestamps corresponding to images exposed at the same time may be different, or the timestamps corresponding to images exposed at different times may be the same. This will cause errors in subsequent fusion perception and threaten driving safety. Summary of the Invention

[0004] The present application provides a multi-camera synchronization method, apparatus, device and computer-readable storage medium, which can solve the technical problem of time asynchrony between multiple cameras existing in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a multi-camera synchronization method, the multi-camera synchronization method comprising:

[0006] The frame synchronization signal generating unit generates a frame synchronization signal and sends it to the timestamp generating unit and N cameras, where N is a positive integer greater than or equal to 2;

[0007] When the timestamp generation unit receives the frame synchronization signal, it puts the timestamp of the current moment into the FIFO queue corresponding to each camera;

[0008] When each camera receives the frame synchronization signal, it performs a shooting action;

[0009] When the forwarding unit receives the image data obtained by each camera based on the shooting action, it reads the timestamp from the FIFO queue corresponding to the camera and sends the image data and the read timestamp to the image processing end.

[0010] In combination with the first aspect, in one embodiment, the frame synchronization signal is a pulse signal, and the frame synchronization signal generating unit generates a frame synchronization signal with a preset pulse width at a preset frequency according to the value of the first counter, and the value of the first counter increases by one every first clock cycle.

[0011] In combination with the first aspect, in one implementation, the timestamp is a value of a second counter, and the value of the second counter is incremented by one every second clock cycle.

[0012] In combination with the first aspect, in one embodiment, the timestamp is the time of the timestamp generation unit, wherein each time the value of the third counter is increased by one, the time of the timestamp generation unit is increased by a third clock cycle, the timestamp generation unit is synchronized with the external reference clock at a preset time interval, and the value of the third counter is increased by one each time the third clock cycle passes.

[0013] In combination with the first aspect, in one implementation, the depth of the FIFO queue is 4.

[0014] In a second aspect, an embodiment of the present application provides a multi-camera synchronization device, the multi-camera synchronization device comprising:

[0015] a frame synchronization signal generating unit, configured to generate a frame synchronization signal and send the signal to the timestamp generating unit and N cameras, where N is a positive integer greater than or equal to 2;

[0016] A timestamp generating unit, configured to, upon receiving the frame synchronization signal, place the current timestamp into the FIFO queue corresponding to each camera;

[0017] The camera is configured to execute a shooting action upon receiving the frame synchronization signal;

[0018] The forwarding unit is used to read the timestamp from the FIFO queue corresponding to the camera when receiving the image data obtained by each camera based on the shooting action, and send the image data and the read timestamp to the image processing end.

[0019] In combination with the second aspect, in one embodiment, the frame synchronization signal is a pulse signal, and the frame synchronization signal generating unit generates a frame synchronization signal with a preset pulse width at a preset frequency according to the value of the first counter, and the value of the first counter increases by one every first clock cycle.

[0020] In combination with the second aspect, in one implementation, the timestamp is a value of a second counter, and the value of the second counter is incremented by one every second clock cycle.

[0021] In a third aspect, an embodiment of the present application provides a multi-camera synchronization device, which includes a processor, a memory, and a multi-camera synchronization program stored on the memory and executable by the processor, wherein when the multi-camera synchronization program is executed by the processor, the steps of the multi-camera synchronization method described in the first aspect are implemented.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a multi-camera synchronization program is stored, wherein when the multi-camera synchronization program is executed by a processor, the steps of the multi-camera synchronization method described in the first aspect are implemented.

[0023] The beneficial effects of the technical solutions provided in the embodiments of the present application include:

[0024] In an embodiment of the present application, a frame synchronization signal generating unit generates a frame synchronization signal and sends it to a timestamp generating unit and N cameras, where N is a positive integer greater than or equal to 2; when the timestamp generating unit receives the frame synchronization signal, it puts the timestamp of the current moment into the FIFO queue corresponding to each camera; when each camera receives the frame synchronization signal, it performs a shooting action; when the forwarding unit receives the image data obtained by each camera based on the shooting action, it reads the timestamp from the FIFO queue corresponding to the camera, and sends the image data and the read timestamp to the image processing end. Through an embodiment of the present application, a frame synchronization signal is sent to multiple cameras so that the multiple cameras can synchronously perform the shooting action; in addition, the frame synchronization signal is synchronously sent to the timestamp generating unit, and the timestamp when the timestamp generating unit receives the frame synchronization signal is used as the starting moment for each camera to perform the shooting action, and is bound to the image data obtained by each camera, so that the back-end image processing end can accurately determine the synchronously exposed image data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of an embodiment of a multi-camera synchronization method of the present application;

[0026] Figure 2 This is a schematic diagram of the functional modules of an embodiment of a multi-camera synchronization device of the present application;

[0027] Figure 3 This is a schematic diagram of the hardware structure of the multi-camera synchronization device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0030] In a first aspect, an embodiment of the present application provides a multi-camera synchronization method.

[0031] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the multi-camera synchronization method of the present application. Figure 1 As shown, the multi-camera synchronization method includes:

[0032] Step S10: The frame synchronization signal generating unit generates a frame synchronization signal and sends it to the timestamp generating unit and N cameras, where N is a positive integer greater than or equal to 2;

[0033] Step S20, when the timestamp generating unit receives the frame synchronization signal, it puts the timestamp of the current moment into the FIFO queue corresponding to each camera;

[0034] Step S30, when each camera receives the frame synchronization signal, it performs a shooting action;

[0035] In step S40 , when the forwarding unit receives the image data obtained by each camera based on the shooting action, it reads the timestamp from the FIFO queue corresponding to the camera and sends the image data and the read timestamp to the image processing end.

[0036] In this embodiment, the hardware structure relied upon by the embodiment of the multi-camera synchronization method of this application is described:

[0037] First, for each camera, the camera's FSYNC pin is connected to the main platform's GPIO pin. The main platform's frame synchronization signal generation unit sends frame synchronization signals at certain time intervals based on the internal clock module. The timing is accurate and does not require CPU intervention. At the same time, the frame synchronization signal generation unit can configure the number of cameras that need to be exposed synchronously through the MUX function. This does not require hardware modification and allows for flexible configuration.

[0038] Among them, the number of frame synchronization signal generating units can be one or more, and each frame synchronization signal generating unit can independently generate a frame synchronization signal and send it to the corresponding camera group; or the signals generated by multiple frame synchronization signal generating units can be merged together to generate a frame synchronization signal, and sent to the corresponding camera group.

[0039] Furthermore, a communication connection is established between the timestamp generation unit and the frame synchronization signal generation unit. The frame synchronization signal is synchronously sent to the timestamp generation unit and N cameras. When each camera receives the frame synchronization signal, it performs a shooting action (i.e., starts exposure). When the timestamp generation unit receives the frame synchronization signal, it puts the timestamp of the current moment into the FIFO queue corresponding to each camera.

[0040] For each camera, image data can be obtained by executing the shooting action, and the image data is sent to the forwarding unit through the communication connection between the forwarding unit and the camera. When the forwarding unit receives a complete frame of image data, it reads the timestamp from the FIFO queue corresponding to the camera and sends the image data and the read timestamp to the image processing end.

[0041] For example, when cameras 1-3 receive frame synchronization signal S, they begin capturing. Correspondingly, upon receiving frame synchronization signal S, the timestamp generation unit places the current timestamp t0 into the corresponding FIFO queues FIFO_1 through FIFO_3 for cameras 1-3. Subsequently, when the forwarding unit receives image data 1 from camera 1, it uses the first-in, first-out (FIFO) nature of the FIFO queue to read t0 from FIFO_1, binds image data 1 with t0, and sends the data to the image processing end. Similarly, when the forwarding unit receives image data 2 from camera 2, it uses the first-in, first-out (FIFO) nature of the FIFO queue to read t0 from FIFO_2, binds image data 2 with t0, and sends the data to the image processing end. When the forwarding unit receives image data 3 from camera 3, it uses the first-in, first-out (FIFO) nature of the FIFO queue to read t0 from FIFO_3, binds image data 3 with t0, and sends the data to the image processing end. This ensures that image data 1, 2, and 3 were exposed simultaneously, allowing them to be used for fusion perception, ensuring the accuracy of fusion perception.

[0042] In an embodiment of the present application, a frame synchronization signal generating unit generates a frame synchronization signal and sends it to a timestamp generating unit and N cameras, where N is a positive integer greater than or equal to 2; when the timestamp generating unit receives the frame synchronization signal, it puts the timestamp of the current moment into the FIFO queue corresponding to each camera; when each camera receives the frame synchronization signal, it performs a shooting action; when the forwarding unit receives the image data obtained by each camera based on the shooting action, it reads the timestamp from the FIFO queue corresponding to the camera, and sends the image data and the read timestamp to the image processing end. Through an embodiment of the present application, a frame synchronization signal is sent to multiple cameras so that the multiple cameras can synchronously perform the shooting action; in addition, the frame synchronization signal is synchronously sent to the timestamp generating unit, and the timestamp when the timestamp generating unit receives the frame synchronization signal is used as the starting moment for each camera to perform the shooting action, and is bound to the image data obtained by each camera, so that the back-end image processing end can accurately determine the synchronously exposed image data.

[0043] Furthermore, in one embodiment, the frame synchronization signal is a pulse signal, and the frame synchronization signal generating unit generates a frame synchronization signal of a preset pulse width at a preset frequency according to the value of the first counter, and the value of the first counter increases by one every first clock cycle.

[0044] In this embodiment, the value of the first counter increments by one every first clock cycle. For example, if the first clock cycle is the clock cycle of a clock unit with a 666M clock cycle, the value of the first counter increments by one every approximately 1.5 nanoseconds. The frame synchronization signal generating unit is configured to output a high-level signal when the value of the first counter reaches a specific value, based on the camera's required frame rate and exposure duration. For example, after the first counter reaches Y, it is reset and restarted. The frame synchronization signal generating unit can be configured to output a high-level signal (i.e., output a frame synchronization signal) when the first counter value ranges from X1 to X2. The values ​​of X1 and X2 are configured based on the camera's required frame rate and exposure duration, thereby generating a frame synchronization signal with a preset pulse width at a preset frequency. Alternatively, if the first counter value continuously accumulates without being reset, the frame synchronization signal generating unit can output a high-level signal (i.e., output a frame synchronization signal) when the first counter value is configured based on the first clock cycle, based on the camera's required frame rate and exposure duration, thereby generating a frame synchronization signal with a preset pulse width at a preset frequency.

[0045] Furthermore, in one embodiment, the timestamp is a value of a second counter, and the value of the second counter is incremented by one every second clock cycle.

[0046] In this embodiment, the timestamp generation unit includes a second clock unit and a second counter, wherein the value of the second counter is incremented by one every second clock cycle, where the second clock cycle is the clock cycle of the second clock unit. For example, if the frequency of the second clock unit is 1 GHz, the value of the second counter is incremented by one every 1 ns.

[0047] Furthermore, in one embodiment, the timestamp is the time of the timestamp generation unit, wherein each time the value of the third counter increases by one, the time of the timestamp generation unit increases by a third clock cycle, the timestamp generation unit synchronizes time with the external reference clock at a preset time interval, and the value of the third counter increases by one each time the third clock cycle passes.

[0048] In this embodiment, the third counter counts based on a third clock cycle. For example, if the clock frequency is 1 GHz, the counter increments by one every 1 ns. The timestamp generation unit synchronizes the standard UTC time from an external reference clock at a certain frequency, such as every 1 ms. It then increments the third counter by one, incrementing by 1 ns. Upon receiving a frame synchronization signal, the timestamp at that moment is stored in the corresponding FIFO queue.

[0049] Furthermore, in one embodiment, the depth of the FIFO queue is 4.

[0050] In this embodiment, for the autonomous driving scenario, using 4 consecutive frames of image data is sufficient, so in order to save chip internal registers or RAM resources, the depth of the FIFO queue can be set to 4.

[0051] In a second aspect, an embodiment of the present application also provides a multi-camera synchronization device.

[0052] In one embodiment, referring to Figure 3 , Figure 3 This is a functional module diagram of an embodiment of the multi-camera synchronization device of the present application. Figure 3 As shown, the multi-camera synchronization device includes:

[0053] A frame synchronization signal generating unit 10 is configured to generate a frame synchronization signal and send the signal to a timestamp generating unit 20 and N cameras 30, where N is a positive integer greater than or equal to 2;

[0054] The timestamp generating unit 20 is configured to put the timestamp of the current moment into the FIFO queue corresponding to each camera upon receiving the frame synchronization signal;

[0055] The camera 30 is configured to execute a shooting action upon receiving the frame synchronization signal;

[0056] The forwarding unit 40 is configured to read a timestamp from a FIFO queue corresponding to each camera upon receiving image data obtained based on a shooting action from each camera 30 , and send the image data and the read timestamp to the image processing end.

[0057] Furthermore, in one embodiment, the frame synchronization signal is a pulse signal, and the frame synchronization signal generating unit 10 generates a frame synchronization signal of a preset pulse width at a preset frequency according to the value of the first counter, and the value of the first counter increases by one every first clock cycle.

[0058] Furthermore, in one embodiment, the timestamp is a value of a second counter, and the value of the second counter is incremented by one every second clock cycle.

[0059] Furthermore, in one embodiment, the timestamp is the time of the timestamp generation unit 20, wherein each time the value of the third counter increases by one, the time of the timestamp generation unit 20 increases by a third clock cycle, the timestamp generation unit 20 performs time synchronization with the external reference clock at a preset time interval, and the value of the third counter increases by one each time the third clock cycle passes.

[0060] Furthermore, in one embodiment, the depth of the FIFO queue is 4.

[0061] Among them, the functional implementation of each module in the above-mentioned multi-camera synchronization device corresponds to the various steps in the above-mentioned multi-camera synchronization method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0062] In a third aspect, an embodiment of the present application provides a multi-camera synchronization device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0063] Reference Figure 3 , Figure 3 Schematic diagram of the hardware structure of the multi-camera synchronization device involved in the embodiment of the present application. In the embodiment of the present application, the multi-camera synchronization device may include a processor, a memory, a communication interface and a communication bus.

[0064] The communication bus may be of any type and is used to interconnect the processor, memory, and communication interface.

[0065] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces. These interfaces interconnect components within the multi-camera synchronization device and connect the device to other devices, such as other computing devices or user devices. Physical interfaces can include Ethernet, fiber optic, and ATM interfaces; user devices can include displays and keyboards.

[0066] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0067] The processor may be a general-purpose processor that can invoke a multi-camera synchronization program stored in a memory and execute the multi-camera synchronization method provided in the embodiments of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the multi-camera synchronization program is invoked can be referenced in the various embodiments of the multi-camera synchronization method of the present application and will not be further described here.

[0068] Those skilled in the art will understand that Figure 3 The hardware structure shown in the figure does not constitute a limitation to the present application and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0069] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0070] The computer-readable storage medium of the present application stores a multi-camera synchronization program, wherein when the multi-camera synchronization program is executed by a processor, the steps of the multi-camera synchronization method as described above are implemented.

[0071] Among them, the method implemented when the multi-camera synchronization program is executed can refer to the various embodiments of the multi-camera synchronization method of this application, and will not be repeated here.

[0072] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0073] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.

[0074] In the description of the embodiments of this application, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0075] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0076] In some processes described in the embodiments of the present application, multiple operations or steps are included that appear in a specific order. However, it should be understood that these operations or steps may not be performed in the order in which they appear in the embodiments of the present application or may be performed in parallel. The sequence numbers of the operations are only used to distinguish between different operations, and the sequence numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be performed in sequence or in parallel, and these operations or steps may be combined.

[0077] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above and includes a number of instructions for enabling a terminal device to execute the methods described in each embodiment of this application.

[0078] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-camera synchronization method, characterized in that: The multi-camera synchronization method comprises: The frame synchronization signal generating unit generates a frame synchronization signal and sends it to the timestamp generating unit and N cameras, where N is a positive integer greater than or equal to 2; When the timestamp generation unit receives the frame synchronization signal, it puts the timestamp of the current moment into the FIFO queue corresponding to each camera, wherein the timestamp is the time of the timestamp generation unit, wherein each time the value of the third counter increases by one, the time of the timestamp generation unit increases by a third clock cycle, the timestamp generation unit synchronizes time with the external reference clock according to a preset time interval, and the value of the third counter increases by one each time the third clock cycle passes; When each camera receives the frame synchronization signal, it performs a shooting action; When the forwarding unit receives the image data obtained by each camera based on the shooting action, it reads the timestamp from the FIFO queue corresponding to the camera and sends the image data and the read timestamp to the image processing end.

2. The multi-camera synchronization method according to claim 1, wherein: The frame synchronization signal is a pulse signal. The frame synchronization signal generating unit generates a frame synchronization signal with a preset pulse width at a preset frequency according to the value of the first counter. The value of the first counter increases by one every first clock cycle.

3. The multi-camera synchronization method according to claim 1 or 2, wherein: The depth of the FIFO queue is 4.

4. A multi-camera synchronization device, characterized in that: The multi-camera synchronization device comprises: a frame synchronization signal generating unit, configured to generate a frame synchronization signal and send the signal to the timestamp generating unit and N cameras, where N is a positive integer greater than or equal to 2; a timestamp generating unit, configured to, upon receiving the frame synchronization signal, place a timestamp of the current moment into a FIFO queue corresponding to each camera, wherein the timestamp is a time of the timestamp generating unit, wherein each time the value of the third counter increases by one, the time of the timestamp generating unit increases by a third clock cycle, the timestamp generating unit synchronizes time with the external reference clock at a preset time interval, and the value of the third counter increases by one each time the third clock cycle passes; The camera is configured to execute a shooting action upon receiving the frame synchronization signal; The forwarding unit is used to read the timestamp from the FIFO queue corresponding to the camera when receiving the image data obtained by each camera based on the shooting action, and send the image data and the read timestamp to the image processing end.

5. The multi-camera synchronization device according to claim 4, wherein: The frame synchronization signal is a pulse signal. The frame synchronization signal generating unit generates a frame synchronization signal with a preset pulse width at a preset frequency according to the value of the first counter. The value of the first counter increases by one every first clock cycle.

6. A multi-camera synchronization device, characterized in that: The multi-camera synchronization device includes a processor, a memory, and a multi-camera synchronization program stored on the memory and executable by the processor, wherein when the multi-camera synchronization program is executed by the processor, the steps of the multi-camera synchronization method as described in any one of claims 1 to 3 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a multi-camera synchronization program, wherein when the multi-camera synchronization program is executed by a processor, the steps of the multi-camera synchronization method according to any one of claims 1 to 3 are implemented.

Images