Multi-camera frame synchronization control method and self-propelled device

Abstract

The present application provides a multi-camera frame synchronization control method, which is used for performing frame synchronization control on images of a first camera and a second camera of a self-propelled device, the method including: sending a synchronization command when the first camera captures a first image frame; determining a transmission time of a frame synchronization signal based on a time offset of a time when the first camera captures the first image frame in response to receiving the synchronization command; and transmitting the frame synchronization signal to the second camera at the transmission time to control the second camera to capture the second image frame. The multi-camera frame synchronization control method ensures that multiple cameras capture images staggered within the same clock cycle, achieving the purpose of time division multiplexing.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority from Chinese Patent Application No. 202210005387.4, filed on January 4, 2022, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of smart devices, and in particular to a multi-camera frame synchronization control method, a self-propelled device, a storage medium, and an electronic device. [Background technology]

[0003] With the development of artificial intelligence, smart devices are becoming more and more common in various scenarios. For example, smart home devices and self-driving delivery trucks are smart devices that move on their own to perform tasks such as cleaning and transporting goods. Self-driving smart devices are equipped with cameras, and often have two cameras (binocular cameras) or more cameras (multi-lens cameras) to obtain multi-directional and multi-angle images of the surroundings. How to control the clock synchronization of image frames from two or more cameras is a technical challenge that needs to be solved. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of this, the present application provides a multi-camera frame synchronization control method, a self-propelled device, a storage medium and an electronic device, the main purpose of which is to solve the technical problem of how to synchronize and control image frames captured by multiple cameras. [Means for solving the problem]

[0005] According to one aspect of the present application, there is provided a multi-camera frame synchronization control method used to perform frame synchronization control on images from a first camera and a second camera of a self-propelled device, the method including: transmitting a synchronization command when the first camera captures a first image frame; in response to receiving the synchronization command, determining a transmission time of a frame synchronization signal based on a time offset of the time when the first camera captured the first image frame; and transmitting the frame synchronization signal to the second camera at the transmission time, and controlling the second camera to capture the second image frame.

[0006] According to one aspect of the present application, a self-propelled device is provided, the self-propelled device comprising a first camera, a second camera, and a controller, wherein the first camera transmits a synchronization command when capturing a first image frame, and the controller, in response to the received synchronization command, determines a transmission time of a frame synchronization signal based on a time offset of the time when the first camera captured the first image frame, transmits a frame synchronization signal to the second camera at the transmission time, and controls the second camera to capture the second image frame.

[0007] According to one aspect of the present application, there is provided a storage medium having a computer program stored therein, the computer program being configured to execute the above-described multi-camera frame synchronization control method when executed.

[0008] According to one aspect of the present application, there is provided an electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the above-mentioned multi-camera frame synchronization control method when it executes the computer program.

[0009] According to the above technical solutions, in the multi-camera frame synchronization control method, self-propelled device, storage medium, and electronic device provided by the present application, the transmission time of a frame synchronization signal is determined from a time offset based on the imaging time of the first camera, and a frame synchronization signal is transmitted at the transmission time to control the time at which the second camera captures an image, so that each camera captures an image at a different time within the same signal cycle, thereby achieving the purpose of time division multiplexing of images captured by different cameras.

[0010] The above description is only an outline of the technical solution of the present application, in order to more clearly understand the technical means of the present application, and to implement according to the content of the specification, in order to make the above and other objectives, features and advantages of the present application more obvious and easily understood, specific embodiments of the present application are specifically cited hereinafter.

[0011] The accompanying drawings described herein are used for further understanding of the present application and constitute a part of the present application, and the illustrative embodiments and the description thereof are used to explain the present application and are not to be construed as unduly limiting the present application. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram showing an implementation scene of a multi-camera frame synchronization control method provided by an embodiment of the present application; [Diagram 2] 1 is a flowchart showing an example of a multi-camera frame synchronization control method provided by an embodiment of the present application. [Diagram 3] FIG. 1 is a schematic diagram illustrating staggered imaging within the same imaging signal cycle provided by an embodiment of the present application; [Figure 4] 1 is a flowchart showing an example of a multi-camera frame synchronization control method provided by an embodiment of the present application. [Diagram 5] FIG. 1 is a schematic diagram showing the structure of a self-propelled device provided by an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In order to allow those skilled in the art to better understand the solution of the present application, the technical solutions of the embodiments of the present application will be described below in a clear and complete manner in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, and are not all of the embodiments. Based on the embodiments of the present application, other embodiments obtained by those skilled in the art without creative labor are all included in the scope of protection of the present application. It should be noted that the embodiments and features of the embodiments of the present application can be combined with each other as long as they are not inconsistent with each other.

[0014] FIG. 1 is a schematic diagram of an implementation scene of a multi-camera frame synchronization control method provided by an embodiment of the present application, in which a self-propelled device 10 is shown, and the self-propelled device 10 is provided with at least two cameras. FIG. 1 illustrates a first camera 100 and a second camera 200, which cooperate to capture images in multiple directions and angles, so as to more completely and clearly capture the surrounding environment of the self-propelled device 10, and provide a basis for obstacle avoidance during the traveling process of the self-propelled device 10. The self-propelled device 10 may be a smart device such as a cleaning robot, an autonomous delivery vehicle, a companion robot, etc., and is not particularly limited in the embodiment of the present application, and it should be understood that the self-propelled device referred to in the embodiment of the present application may be any existing or future self-propelled smart device.

[0015] FIG. 2 is a flowchart showing an example of a multi-camera frame synchronization control method provided by an embodiment of the present application, which is used to perform frame synchronization control on images from a first camera and a second camera of a self-propelled device, and includes the following steps S201 to S203.

[0016] In S201, a synchronization command is sent when the first camera captures the first image frame.

[0017] In addition, in a self-propelled device using a binocular camera (or a multi-lens camera), it is necessary to obtain images captured by two (or multiple) cameras within the same imaging signal cycle to provide a basis for image synthesis, calibration, and analysis, so it is necessary to synchronize the clocks of the two or multiple cameras, where clock synchronization means controlling at least two images captured by each camera with a lag (a first image frame captured by a first camera, a second image frame captured by a second camera) within the same imaging signal cycle of the imaging clock.

[0018] It should be noted that the multi-camera frame synchronization control method provided by the embodiments of the present application can synchronously control image frames of at least two cameras. For convenience of explanation, the following description will be given using two cameras (a first camera and a second camera) as an example. It should be understood that scenes of two or more cameras can also be processed by adopting the solutions provided by the embodiments of the present application in a similar manner.

[0019] In one embodiment, to synchronize the clocks of two cameras, in the embodiment of the present application, when one camera (e.g., the master camera) captures an image, the other camera (e.g., the slave camera) does not capture an image or discards the captured image, and conversely, when the slave camera captures an image, the master camera does not capture an image or discards the captured image. At the same time, when the first camera captures an image, it records the capture time and sends a synchronization command at this time to later control the clock synchronization of the second camera.

[0020] In S202, in response to receiving a synchronization command, a transmission time of a frame synchronization signal is determined based on a time offset of the time when the first camera captured a first image frame.

[0021] Assuming that the first camera starts capturing the first image frame at time t0, a synchronization command is sent at time t0, and the synchronization command is used to record the capture start time of the first camera, and then the transmission time of the frame synchronization signal is determined based on the time offset of the time when the first camera captures the first image frame.

[0022] In S203, a frame synchronization signal is transmitted to the second camera at the transmission time, and the second camera is controlled to capture the second image frame.

[0023] Assuming that the first camera captures a first image frame at time t0, and the time offset at which the two cameras capture image frames is Δt, the second camera is controlled to capture a second image frame at time “t0+Δt”.

[0024] In this way, to achieve clock synchronization of the different cameras, the first camera captures a first image frame and the second camera captures a second image frame within the same imaging signal cycle, where a time offset is used to control the imaging time of the second image frame to be different from the imaging time of the first image frame.

[0025] FIG. 3 is a schematic diagram of two cameras capturing images with a shift within the same imaging signal cycle. In the example of FIG. 3, the first camera and the second camera use an imaging clock CLK of the same frequency, and are controlled to capture images with a shift at least once within each imaging signal cycle of the CLK, that is, capture at least one first image frame and one second image frame. FIG. 3 shows that each camera captures images twice within each imaging signal cycle, specifically, the first camera captures images at times t1 and t2, and the second camera captures images at times "t1+Δt" and "t2+Δt". As can be seen from the figure, all four times are within the same imaging signal cycle.

[0026] In another embodiment, to synchronize the clocks of two cameras, when one camera (e.g., a master camera) captures an image, the other camera (e.g., a slave camera) does not capture an image or discards the captured image, and conversely, when the slave camera captures an image, the master camera does not capture an image or discards the captured image. At the same time, when the first camera captures an image, the capture time is recorded, and a synchronization command is sent at this time to control the clock synchronization of the second camera later.

[0027] In this way, according to the multi-camera frame synchronization control method provided by the embodiments of the present application, the image capture time of the second camera is controlled from the time offset based on the capture time of the first camera, so that each camera captures images at a different time within the same signal cycle, thereby achieving the purpose of time division multiplexing of images captured by different cameras.

[0028] FIG. 4 is a flowchart showing an example of a multi-camera frame synchronization control method provided by an embodiment of the present application, which is used to perform frame synchronization control on images from a first camera and a second camera of a self-propelled device, and includes the following steps S401 to S405.

[0029] Compared with the example shown in FIG. 2, this example introduces a controller to control the clock synchronization between the first camera and the second camera, and specifically describes a solution for synchronizing image frames of two cameras when the two cameras use clocks with the same or multiple frequencies, under the premise that the initial clocks of the two cameras are aligned.

[0030] In S401, the first camera and the second camera are controlled to perform initial clock alignment.

[0031] For example, initial clock alignment of the first and second cameras can be achieved by controlling the first and second cameras to be turned on simultaneously or within a preset minimum time interval.

[0032] In S402, the first camera and the second camera are controlled so as to use clocks of the same frequency, or the first camera and the second camera are controlled so as to use clock frequencies that are multiples of each other.

[0033] In one embodiment, the first camera and the second camera use clocks of the same frequency, for example, by controlling the first camera and the second camera to use the same clock signal, the first camera and the second camera have clocks of the same frequency. As each clock pulse arrives, an image is captured, for example, assuming that the frame rate of the first camera and the second camera is 30 frames per second (fps), the camera is controlled to capture 30 frame images every second from the time the camera is started.

[0034] In another embodiment, the first and second cameras are controlled to use clock frequencies that are multiple-related, for example, the first camera has a frame rate of 15 frames per second and the second camera has a frame rate of 30 frames per second (fps), i.e., the second camera frame rate is a multiple of 2 of the first camera frame rate.

[0035] In S403, a synchronization command is sent when the first camera captures the first image frame.

[0036] In the method adopted in the embodiment of the present application, when one camera (e.g., the master camera) captures an image, the other camera (e.g., the slave camera) discards the images that are not captured or captured, and conversely, when the slave camera captures an image, the master camera discards the images that are not captured or captured. At the same time, when the first camera captures an image, it records the capture start time and transmits a synchronization command at this time, thereby controlling the second camera to be synchronized later.

[0037] In S404, the controller receives the synchronization command and determines the transmission time of the frame synchronization signal based on the time offset of the time when the first camera captures an image.

[0038] Assuming that the first camera starts capturing the first image frame at time t0, a synchronization command is sent at time t0, and the synchronization command is used to record the time when the first camera actually starts capturing. Then, the transmission time of the frame synchronization signal is determined based on the time offset Δt of the two cameras. Specifically, if the first camera and the second camera use clocks with the same frequency, it is ensured that the two times t0 and "t0+Δt" are within the same clock signal cycle; if the first camera and the second camera use clocks with a multiple relationship, assuming that the first camera clock is CLK1 and the second camera clock is CLK2 and they have a multiple relationship, t0 is first converted to the time under CLK2, and then the transmission time of the frame synchronization signal is determined based on the clock signal cycle of CLK2.

[0039] In S405, a frame synchronization signal is transmitted to the second camera at the transmission time, and the second camera is controlled to capture the second image frame.

[0040] Corresponding to the above-mentioned multi-camera frame synchronization control method, the embodiment of the present application further provides a self-propelled device. Figure 5 shows a structural schematic diagram of the self-propelled device provided by the embodiment of the present application.

[0041] The self-propelled device comprises at least a first camera 501, a second camera 502, and a controller 503, wherein the first camera 501 transmits a synchronization command when capturing a first image frame, and in response to receiving the synchronization command from the first camera 501, the controller 503 determines a transmission time of a frame synchronization signal based on a time offset of the time at which the first camera captures the first image frame, transmits a frame synchronization signal to the second camera 502 at the transmission time, and controls the second camera to capture the second image frame.

[0042] In one embodiment, the controller 503 is further used to determine the transmission time of the frame synchronization signal based on a preset operation clock difference between the first camera and the second camera.

[0043] In one embodiment, the controller 503 is further used to control alignment of the initial clocks of the first camera 501 and the second camera 502 .

[0044] In one embodiment, the controller 503 is further used to control the first camera 501 and the second camera 502 to meet initial clock alignment by controlling the first camera 501 and the second camera 502 to be turned on simultaneously or within a preset minimum time interval.

[0045] In one embodiment, the controller 503 is further used to control the first camera 501 and the second camera 502 to use clocks of the same frequency, or to control the first camera 501 and the second camera 502 to use clock frequencies that are multiples of each other.

[0046] In one embodiment, the controller 503 is further used to control that the capturing of the first image frame by the first camera and the capturing of the second image frame by the second camera are performed within the same imaging signal cycle, wherein the time offset is used to control the capturing time of the second image frame to be different from the capturing time of the first image frame.

[0047] In one embodiment, the controller 503 is further used to control the second camera not to capture an image or to discard the captured image of the second camera when the first camera captures a first image frame, and to control the first camera not to capture an image or to discard the captured image of the first camera when the second camera captures a second image.

[0048] An embodiment of the present application further provides a storage medium, having a computer program stored thereon, wherein the computer program is configured to perform the steps of any one of the method embodiments above when executed.

[0049] Optionally, in this embodiment, the storage medium is configured to store a computer program that executes the following steps: (1) sending a synchronization command when a first camera captures a first image frame; (2) in response to receiving a synchronization command, determining a transmission time of a frame synchronization signal based on a time offset of a first image frame capture time of the first camera; (3) The frame synchronization signal is transmitted to a second camera at the transmission time, and the second camera is controlled so as to capture a second image frame.

[0050] Optionally, in this embodiment, the storage medium includes, but is not limited to, various media capable of storing computer programs, such as a USB disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.

[0051] An embodiment of the present application further provides an electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to perform the steps of any one of the method embodiments described above when executing the computer program.

[0052] Optionally, the electronic device may further comprise a transmitting device and an input / output device, where the transmitting device is connected to the processor and the input / output device is connected to the processor.

[0053] Optionally, in this embodiment, the processor executes the following steps by a computer program: (1) sending a synchronization command when a first camera captures a first image frame; (2) in response to receiving a synchronization command, determining a transmission time of a frame synchronization signal based on a time offset of a first image frame capture time of the first camera; (3) A frame synchronization signal is transmitted to a second camera at the transmission time, and the second camera is controlled so as to capture a second image frame.

[0054] Optionally, specific examples of this embodiment may refer to the examples described in the above embodiments and optional embodiments, which will not be repeated in this embodiment.

[0055] The numbering of the examples in the above application is used for illustrative purposes and does not indicate the merits of the examples.

[0056] In the above embodiments of the present application, the description of each embodiment is focused, and the parts not described in detail in one embodiment may be referred to the related descriptions of other embodiments.

[0057] It should be understood that in some embodiments provided by the present application, the technical contents disclosed may be realized by other means. Here, the above-mentioned device embodiments are merely examples, for example, the division of the units is merely a logical division of functions, and the actual implementation may be divided in other ways, for example, multiple units or assemblies may be combined or integrated into other systems, or some features may be ignored or not implemented. Alternatively, the illustrated or discussed mutual couplings or direct couplings or communication connections may be indirect couplings or communication connections of some interfaces, units or modules, and may be electrical or other methods.

[0058] The units described as separate components may or may not be physically separated, and the components illustrated as units may or may not be physical units, i.e., they may be located in one place or distributed among multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of this embodiment.

[0059] Furthermore, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be implemented in the form of hardware or software functional unit.

[0060] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in one computer-readable storage medium. Based on this understanding, the technical solution of the present application may be essentially or in part embodied in the form of a software product, and the computer software product is stored in one storage medium and contains a number of instructions to make a computer device (such as a personal computer, a server or a network device) execute all or part of the steps of the method described in each embodiment of the present application. The storage medium includes, but is not limited to, various media that can store program code, such as a USB disk, a read-only memory (ROM), a random access memory (RAM), a removable hard disk, a magnetic disk or an optical disk.

[0061] Although the preferred embodiments of the present application have been described above, it should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present application, and all such improvements and modifications shall fall within the scope of protection of the present application.

Claims

1. A multi-camera frame synchronization control method for performing frame synchronization control on images of a first camera and a second camera of a self-propelled device, comprising: transmitting a synchronization command when the first camera captures a first image frame; In response to receiving the synchronization command, determining a transmission time of a frame synchronization signal based on a time offset of a time when the first camera captured a first image frame; transmitting the frame synchronization signal to the second camera at the transmission time to control the second camera to capture a second image frame. A multi-camera frame synchronization control method comprising:

2. Determining a transmission time of a frame synchronization signal based on a time offset of a time when the first camera captured a first image frame determining a transmission time of the frame synchronization signal based on a preset operation clock difference between the first camera and the second camera; 2. The multi-camera frame synchronization control method according to claim 1 .

3. and controlling the first camera and the second camera to perform initial clock alignment.

2. The multi-camera frame synchronization control method according to claim 1 .

4. Controlling the first camera and the second camera to align an initial clock includes: controlling the first camera and the second camera to be turned on simultaneously or within a preset minimum time interval to ensure that the first camera and the second camera meet initial clock alignment.

4. The multi-camera frame synchronization control method according to claim 3.

5. The method further includes controlling the first camera and the second camera to use clocks of the same frequency, or controlling the first camera and the second camera to use clock frequencies that are multiples of each other.

2. The multi-camera frame synchronization control method according to claim 1 .

6. imaging the first image frame by the first camera and the second image frame by the second camera within the same imaging signal cycle, and controlling the time offset so that an imaging time of the second image frame is different from an imaging time of the first image frame.

6. The multi-camera frame synchronization control method according to claim 1, wherein:

7. Controlling the second camera so that, when the first camera captures a first image frame, the second camera does not capture an image or discards an image captured by the second camera; and and controlling the first camera not to capture an image or to discard an image captured by the first camera when the second camera captures a second image.

6. The multi-camera frame synchronization control method according to claim 1, wherein:

8. A self-propelled device comprising a first camera, a second camera, and a controller, the first camera transmitting a synchronization command when capturing a first image frame; the controller, in response to the received synchronization command, determines a transmission time of a frame synchronization signal based on a time offset of a time when the first camera captured a first image frame, and transmits a frame synchronization signal to the second camera at the transmission time, thereby controlling the second camera to capture a second image frame. A self-propelled device comprising:

9. The controller is used to determine a transmission time of the frame synchronization signal based on a preset operation clock difference between the first camera and the second camera.

9. The self-propelled device according to claim 8.

10. The controller is further used to control the first camera and the second camera to perform initial clock alignment.

9. The self-propelled device according to claim 8.

11. the controller is further used to control the first camera and the second camera to be turned on simultaneously or within a preset minimum time interval to ensure that the first camera and the second camera meet initial clock alignment.

11. The self-propelled device according to claim 10.

12. The controller is used to control the first camera and the second camera to use clocks of the same frequency, or to control the first camera and the second camera to use clock frequencies that are multiples of each other.

9. The self-propelled device according to claim 8.

13. The controller further controls the first camera to capture the first image frame and the second camera to capture the second image frame within the same imaging signal cycle, and controls the time offset so that an imaging time of the second image frame is different from an imaging time of the first image frame. A self-propelled device according to any one of claims 8 to 12.

14. the controller is further used to control the second camera not to capture an image or to discard an image captured by the second camera when the first camera captures a first image frame, and to control the first camera not to capture an image or to discard an image captured by the first camera when the second camera captures a second image. A self-propelled device according to any one of claims 8 to 12.

15. a computer program stored thereon, the computer program being configured to carry out the method according to any one of claims 1 to 7 when the computer program is executed; A storage medium comprising:

16. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor being configured to execute the method according to any one of claims 1 to 7 when executing the computer program. An electronic device comprising:

Images