Multi-camera frame synchronization method and apparatus and storage medium

Abstract

Embodiments of the present application disclose a multi-camera frame synchronization method and device and a storage medium, and are applied to the technical field of images. A slave camera in a multi-camera system acquires first timestamp information of each frame in a master video frame sequence and second timestamp information of each frame in a slave video frame sequence, and calculates a first time interval of adjacent frames in the master video frame sequence and a second time interval of adjacent frames in the slave video frame sequence based on the information, so that the frame rate of the slave camera is set to be the same as that of the master camera. On this basis, the frame rate of the slave camera is adjusted in combination with a timestamp difference between the second timestamp information of a specific frame in the slave video frame sequence and the first timestamp information of a first reference frame in the master video frame sequence and the frame interval of adjacent frames, so as to gradually realize frame synchronization of the master and slave cameras, so that each slave camera and the master camera can simultaneously capture a target scene at the same time, and the phenomenon of splitting in later picture splicing is avoided.

Description

Technical Field

[0001] This invention relates to the field of image technology, and in particular to a multi-camera frame synchronization method, apparatus, and storage medium. Background Technology

[0002] Panoramic cameras and free-view camera arrays are devices capable of capturing panoramic images or videos. They use multiple lenses to simultaneously capture the surrounding environment, generating a 360-degree panoramic view. Content captured by panoramic cameras allows users to experience virtual reality content immersively, as if they were actually in the scene being filmed. Whether used for recording life, creating content, or professional applications, they demonstrate unique advantages.

[0003] In general, multiple cameras are used in a camera array to take pictures, especially in dynamic scenes. This requires the multiple cameras to synchronize frames to avoid disjointed images in the later stitching process. Summary of the Invention

[0004] This invention provides a multi-camera frame synchronization method, apparatus, and storage medium, which avoids the phenomenon of fragmentation in the post-processing stitching of images from a multi-camera system.

[0005] One embodiment of the present invention provides a multi-camera frame synchronization method for a multi-camera system having one master camera and multiple slave cameras, comprising:

[0006] Obtain the first timestamp information of each frame in the main video frame sequence captured by the main camera of the multi-camera system for the target scene;

[0007] Obtain the second timestamp information of each frame in the video frame sequence captured by the camera for the target scene;

[0008] Calculate the first time interval between adjacent frames in the main video frame sequence and the second time interval between adjacent frames in the secondary video frame sequence, and based on the first time interval and the second time interval, set the frame rate of the secondary camera to be the same as the frame rate of the main camera to obtain the set frame rate of the secondary camera;

[0009] Calculate the timestamp difference between the second timestamp information of a specific frame in the video frame sequence based on the set frame rate and the first timestamp information of the first reference frame in the main video frame sequence, and calculate the frame interval between adjacent frames in the video frame sequence based on the set frame rate; the first reference frame is the frame in the main video frame sequence that is closest to the specific frame before the specific frame.

[0010] Based on the timestamp difference and frame interval, the frame rate of the slave camera is adjusted so that subsequent frames of a specific frame in the slave video frame sequence are gradually synchronized with subsequent frames of the first reference frame in the main video frame sequence.

[0011] Another aspect of the present invention provides a multi-camera frame synchronization device for a multi-camera system having one master camera and multiple slave cameras, comprising:

[0012] The first acquisition unit is used to acquire the first timestamp information of each frame in the main video frame sequence captured by the main camera in the multi-camera system for the target scene.

[0013] The second acquisition unit is used to acquire the second timestamp information of each frame in the video frame sequence captured by the camera for the target scene;

[0014] A frame rate setting unit is used to calculate a first time interval between adjacent frames in the main video frame sequence and a second time interval between adjacent frames in the secondary video frame sequence, and based on the first time interval and the second time interval, set the frame rate of the secondary camera to be the same as the frame rate of the main camera, thereby obtaining the set frame rate of the secondary camera.

[0015] The difference calculation unit is used to calculate the timestamp difference between the second timestamp information of a specific frame in the video frame sequence based on the set frame rate and the first timestamp information of the first reference frame in the main video frame sequence, and to calculate the frame interval between adjacent frames in the video frame sequence based on the set frame rate; the first reference frame is the frame in the main video frame sequence that is closest to the specific frame before the specific frame.

[0016] The adjustment unit is used to adjust the frame rate of the slave camera according to the timestamp difference and the frame interval, so that the subsequent frames of a specific frame in the slave video frame sequence are gradually synchronized with the subsequent frames of the first reference frame in the main video frame sequence.

[0017] Another aspect of the present invention provides a computer-readable storage medium storing a plurality of computer programs adapted for loading by a processor and executing the multi-camera frame synchronization method as described in one aspect of the present invention.

[0018] As can be seen, in the method of this embodiment, the slave camera in the multi-camera system acquires the first timestamp information of each frame in the master video frame sequence of the master camera, and the second timestamp information of each frame in the slave video frame sequence of the slave camera. Based on this, it calculates the first time interval between adjacent frames in the master video frame sequence and the second time interval between adjacent frames in the slave video frame sequence, thereby setting the frame rate of the slave camera to be the same as that of the master camera. Further, based on the same frame rate between the master and slave cameras, the frame rate of the slave camera is adjusted by combining the timestamp difference between the second timestamp information of a specific frame in the slave video frame sequence and the first timestamp information of the first reference frame in the master video frame sequence, and the frame interval between adjacent frames in the slave video frame sequence, so as to gradually achieve frame synchronization between the master and slave cameras. In this way, by further synchronizing related frames on the basis of frame rate synchronization between the master and slave cameras, each slave camera and the master camera can simultaneously shoot the target scene at the same time, avoiding the phenomenon of disjointed images during post-processing. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a multi-camera system provided in an embodiment of the present invention;

[0021] Figure 2 This is a flowchart of a multi-camera frame synchronization method provided in an embodiment of the present invention;

[0022] Figure 3a This is a schematic diagram illustrating a relationship between a specific frame and a first reference frame in an embodiment of the present invention;

[0023] Figure 3b This is a schematic diagram illustrating another relationship between a specific frame and a first reference frame in an embodiment of the present invention;

[0024] Figure 4a This is a schematic diagram illustrating the relationship between the main video frame sequence and the slave video frame sequence after increasing the slave camera frame rate in an embodiment of the present invention.

[0025] Figure 4b This is a schematic diagram illustrating the relationship between the main video frame sequence and the slave video frame sequence after reducing the slave camera frame rate in an embodiment of the present invention.

[0026] Figure 5 This is a flowchart of a multi-camera frame synchronization method in one application embodiment of the present invention;

[0027] Figure 6 This is a schematic diagram of the logic structure of a multi-camera frame synchronization device provided in an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the logical structure of a terminal device provided in an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] This invention provides a multi-camera frame synchronization method, mainly applied to applications such as... Figure 1 The image shown is from a multi-camera system, where each camera is positioned at a different location and captures the scene within its field of view. This allows multiple cameras to capture the same scene from different angles. Figure 1 Taking a circle of multiple cameras as an example, in other embodiments, the cameras can be placed in other positions, which is not limited here, nor is the number of cameras limited. Wherein:

[0032] Each camera captures an image at a set frame rate interval, and consecutive images are combined to form a video. For example, the video frames use a fixed field synchronization signal as a reference and begin exposure according to a set exposure time. If the exposure start position and exposure duration of video frames from multiple cameras at the same time are consistent, the objects captured in the same scene will be consistent, and the resulting image obtained by stitching together the images captured by each camera will not have any fragmentation.

[0033] To avoid subsequent image fragmentation, it's generally necessary to ensure that the frame intervals (i.e., frame rates), initial exposure times, and exposure durations of frames at the same time are consistent across all cameras. This achieves frame synchronization and effectively resolves the subsequent image fragmentation problem. When the frame rates of multiple cameras are consistent, only synchronization settings are needed to ensure the initial exposure time and frame exposure duration are identical. Therefore, achieving frame rate synchronization is crucial in the frame synchronization process.

[0034] In the multi-camera frame synchronization process of this invention embodiment, a master camera and slave cameras are specifically set up among multiple cameras in the multi-camera system. For any slave camera, such as Figure 2 As shown, frame synchronization can be achieved using the following method:

[0035] Step 101: Obtain the first timestamp information of each frame in the main video frame sequence captured by the main camera of the multi-camera system for the target scene.

[0036] In this embodiment, any camera in the multi-camera system can be set as the master camera, or a camera in an important position can be set as the master camera, such as the camera in front of the target scene. The remaining cameras are slave cameras. Each slave camera needs to actively synchronize frames with the master camera based on the master camera, thereby achieving frame synchronization among all cameras.

[0037] The main video frame sequence captured by the main camera includes multiple frames of images of the target scene based on time information. Thus, the first timestamp information of each frame in the main video frame sequence obtained from the camera is used to describe the starting shooting time of each frame of the target scene captured by the main camera.

[0038] In a specific implementation, when the main camera begins capturing each frame of the main video frame sequence, it can send a signal, such as an interrupt signal or a dot signal, to each slave camera. Thus, the slave cameras can obtain the aforementioned first timestamp information in the following way:

[0039] Receive signals transmitted by the main camera at the start time of each frame in the main video frame sequence;

[0040] Based on the transmitted signal, the current system timestamp is recorded as the first timestamp information.

[0041] In another specific implementation, instead of having the main camera actively transmit signals, a simplified setup is implemented where each camera in the multi-camera system is configured to generate a specific signal, such as an interrupt signal, when it begins capturing each frame in its corresponding video frame sequence. This allows the slave cameras to obtain the aforementioned first timestamp information using the following method:

[0042] Real-time monitoring of specific signals generated by the main camera;

[0043] When a specific signal is detected, the current system timestamp is recorded as the first timestamp information.

[0044] In the process of obtaining the first timestamp information, each slave camera needs to detect the main camera in real time, such as detecting the signal transmitted by the main camera or the specific signal generated by the main camera, and record it in real time. This requires maintaining real-time communication between the main camera and each slave camera.

[0045] Step 102: Obtain the second timestamp information of each frame in the video frame sequence captured by the camera for the target scene.

[0046] Specifically, when the camera starts capturing each frame of the video frame sequence, the system timestamp at that time can be recorded as the second timestamp information for each frame. In this way, the second timestamp information is used to describe the start time of the camera capturing each frame of the target scene.

[0047] In one specific embodiment, the camera can be configured to generate a specific signal, such as an interrupt signal, when it begins capturing each frame of the video frame sequence. In this way, the camera can directly record the system time of each specific signal as a second timestamp information.

[0048] It should be noted that the execution order of steps 101 and 102 above is not limited; they can be executed simultaneously or sequentially. Figure 2 The example shown is just one specific implementation method.

[0049] Step 103: Calculate the first time interval between adjacent frames in the main video frame sequence and the second time interval between adjacent frames in the secondary video frame sequence. Based on the first and second time intervals, set the frame rate of the secondary camera to be the same as that of the main camera to obtain the frame rate of the secondary camera after setting.

[0050] Specifically, in the main video frame sequence, the first time interval between adjacent frames is the difference between the first timestamp information Mn of one frame and the first timestamp information Mn-1 of the previous frame, Mn-Mn-1; and in the secondary video frame sequence, the second time interval between adjacent frames is the difference between the second timestamp information Sn of one frame and the second timestamp information Sn-1 of the previous frame, Sn-Sn-1.

[0051] Since the frame rate of a camera has a certain linear relationship with the time interval between adjacent frames, if the first time interval is the same as the second time interval, the frame rate of the slave camera is the same as the frame rate of the master camera. If the first time interval is different from the second time interval, the frame rate of the slave camera is inconsistent with the frame rate of the master camera, and the frame rate of the slave camera needs to be set to be the same as the frame rate of the master camera.

[0052] Step 104: Calculate the timestamp difference between the second timestamp information of a specific frame in the video frame sequence based on the set frame rate and the first timestamp information of the first reference frame in the main video frame sequence, and calculate the frame interval between adjacent frames in the video frame sequence based on the set frame rate; the first reference frame is the frame in the main video frame sequence that is closest to the specific frame before the specific frame.

[0053] After setting the frame rate from the camera, a specific frame is selected from the sequence of video frames based on the set frame rate to calculate the timestamp difference between it and a first reference frame. The first reference frame is the frame preceding the specific frame and has the smallest time interval between it and the specific frame in the main video frame sequence. For example... Figure 3a In the main video frame sequence, the timestamps of each frame are M1, M2, ..., Mn, while the timestamps of each frame are S1, S2, ..., Sn. Among them, a specific frame S1 in the secondary video frame sequence corresponds to the first reference frame M1 in the main video frame sequence; a specific frame S4 ​​in the secondary video frame sequence corresponds to the first reference frame M4 in the main video frame sequence, and so on.

[0054] For example Figure 3b In the video frame sequence, a specific frame S1 corresponds to the first reference frame M2 in the main video frame sequence; a specific frame Sn-1 corresponds to the first reference frame Mn in the main video frame sequence, and so on.

[0055] Step 105: Based on the timestamp difference and frame interval calculated in step 104 above, adjust the frame rate of the camera so that the subsequent frames of a specific frame in the video frame sequence are gradually synchronized with the subsequent frames of the first reference frame in the main video frame sequence.

[0056] It can be understood that when the timestamp difference Δt calculated in step 104 above is less than or equal to a critical value δ, the master video frame sequence and the slave video frame sequence are considered to be frame synchronized. That is, the master and slave cameras will simultaneously capture images of the target scene from various angles at almost the same time. If the timestamp difference Δt is greater than this critical value, frame synchronization can be achieved in the following way:

[0057] If the timestamp difference is less than half of the frame interval but greater than a threshold, increase the frame rate from the camera to obtain the increased frame rate.

[0058] If the timestamp difference is greater than half of the frame interval but less than the difference between the frame interval and the threshold, reduce the frame rate from the camera to obtain the reduced frame rate.

[0059] Specifically, when increasing the frame rate of the camera, if the timestamp difference is greater than a set difference, the frame rate of the camera will be increased by a first change value, such as 1 fps, to obtain the increased frame rate; if the timestamp difference is less than or equal to the first frame rate, the frame rate of the camera will be increased by a second change value, such as 0.5 fps, to obtain the increased frame rate, wherein the first change value is greater than the second change value.

[0060] Specifically, when reducing the frame rate of the camera, if the absolute difference between the timestamp difference and the frame interval is greater than a set difference, the frame rate of the camera will be reduced by a first change value, such as 1 fps, to obtain the reduced frame rate; if the absolute difference between the timestamp difference and the frame interval is less than or equal to the set difference, the frame rate of the camera will be reduced by a second change value, such as 0.5 fps, to obtain the reduced frame rate.

[0061] In this way, different adjustment ranges (first change value or second change value) are selected based on the magnitude of the timestamp difference. The timestamp difference reflects the gap between the two closest frames in the master and slave video frame sequences. The frame rate of the slave camera is adjusted according to this gap. When the gap is large, a larger adjustment is made to the frame rate of the slave camera, which can quickly achieve frame synchronization between the slave and master cameras. When the gap is small, a smaller adjustment is made, i.e., fine-grained adjustment, so that even small differences between the frames of the master and slave cameras are synchronized. In this way, higher precision and more efficient frame synchronization can be achieved by changing the frame rate adjustment range.

[0062] It should be noted that when performing steps 104 to 105 in the above process, after obtaining the second timestamp information St of any frame P1 (i.e. the specific frame mentioned above) in the video frame sequence from the camera based on the set frame rate, step 105 can be performed for the first timestamp information Mt of the corresponding first reference frame P2 in the main video frame sequence.

[0063] Furthermore, after adjusting the slave camera frame rate for a specific pair of frames and the first reference frame, the slave camera frame rate can be adjusted again for the next specific pair of frames and the first reference frame. That is, step 105 above is repeated to dynamically adjust the slave camera frame rate, so as to ultimately achieve frame synchronization between the slave camera and the master camera. For example:

[0064] (1) The above timestamp difference is Δt = St - Mt. If Δt < Ft / 2, the frame rate of the slave camera is increased. Here, St is the second timestamp information of a specific frame, Mt is the first timestamp information of the first reference frame, and the frame interval Ft is the time interval between adjacent frames in the slave video frame sequence based on the set frame rate. Specifically, the frame interval Ft = 1000 / Mf, and Mf = 1000 / (Sn - Sn - 1) = 1000 / (Mn - Mn - 1), where Sn - Sn - 1 is the time interval between adjacent frames in the slave video frame sequence based on the set frame rate, and Mn - Mn - 1 is the time interval between adjacent frames in the main video frame sequence.

[0065] As Figure 4a shown, when increasing the frame rate of the slave camera, different increasing amplitudes need to be adopted according to the magnitude of the timestamp difference Δt to increase the frame rate of the slave camera. For example, when the timestamp difference is greater than a set difference, an increasing amplitude of 1 fps is adopted. When the timestamp difference is less than or equal to the set difference, an increasing amplitude of 0.5 fps is adopted. Here, the set difference needs to be greater than the above critical value δ.

[0066] After increasing the frame rate of the slave camera, the second timestamp information of each frame in the subsequent generated slave video frame sequence will be slightly advanced, that is, the timestamp difference Δt value between St - Mt will gradually decrease to 0. In this process, when the timestamp difference is less than or equal to a critical value, that is, |Δt| ≤ δ, the frame synchronization between the slave camera and the master camera is achieved. At this time, the increased frame rate of the slave camera needs to be adjusted back to the above set frame rate. That is to say, after increasing the frame rate of the slave camera, the slave camera will:

[0067] Obtain the second timestamp information of the subsequent frame of the specific frame according to the increased frame rate, and update the above timestamp difference based on this;

[0068] When the timestamp difference between the second timestamp of the subsequent frame and the first timestamp information of the corresponding reference frame in the main video frame sequence is less than or equal to the critical value, adjust the increased frame rate back to the set frame rate of the slave camera; where the corresponding reference frame for the subsequent frame is the frame that is before the subsequent frame and has the closest distance to the subsequent frame in the main video frame sequence;

[0069] When the timestamp difference between the second timestamp of the subsequent frame and the first timestamp information of the corresponding reference frame in the main video frame sequence is greater than the critical value, based on the updated timestamp difference and the frame interval, return to execute the above step of increasing the frame rate of the slave camera.

[0070] (2) If Δt > Ft / 2 and (Ft - Δt) > δ, that is, Δt < Ft - δ, the frame rate of the slave camera is decreased. As Figure 4bAs shown, when reducing the frame rate of a slave camera, different reduction rates need to be applied based on the magnitude of the absolute difference |Δt - Ft| between the timestamp difference Δt and the frame interval Ft. For example, if the absolute difference between the timestamp difference and the frame interval is greater than a set value, an increase of 1 fps is applied; if the absolute difference is less than or equal to the set value, an increase of 0.5 fps is applied. This set value must be greater than the aforementioned threshold.

[0071] When the frame rate of the slave camera is reduced, the second timestamp information of each subsequent frame in the slave video will be slightly delayed. This causes the timestamp difference Δt between St and Mt to gradually approach Ft, and the difference between Mt+1 and St to gradually approach 0. During this process, the absolute difference between the timestamp difference Δt and the frame interval Ft is less than or equal to a critical value, i.e., |Δt - Ft| ≤ δ, thus achieving frame synchronization between the slave and master cameras. In other words, after reducing the frame rate of the slave camera, the slave camera will:

[0072] Based on reducing the frame rate, obtain the second timestamp information of the subsequent frames of a specific frame, and update the timestamp difference accordingly;

[0073] When the absolute difference between the second timestamp difference of the subsequent frame and the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence is less than or equal to the frame interval, the frame rate will be reduced and adjusted back to the frame rate set by the camera; wherein, the reference frame corresponding to the subsequent frame is the frame before the subsequent frame and the frame closest to the subsequent frame in the main video frame sequence.

[0074] When the absolute difference between the second timestamp difference of the subsequent frame and the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence, and the frame interval is greater than the critical value, the step of reducing the frame rate of the slave camera is returned based on the updated timestamp difference and the frame interval.

[0075] (3) In other cases besides the two mentioned above, such as Δt ≤ δ or |Ft - Δt| ≤ δ, the master and slave cameras have already achieved frame synchronization. The current frame rate is maintained and no further processing is required.

[0076] It should be noted that after setting the frame rates of the master and slave cameras to be consistent, frame synchronization between the master and slave cameras can be gradually achieved through the above dynamic adjustment process. During the dynamic adjustment process, that is, during the repeated execution of step 105 above, the step size of each adjustment can be different. That is, the first change value and the second change value mentioned above can change dynamically. Specifically:

[0077] During the dynamic adjustment process, the frame rate of the slave camera can be adjusted by using a decreasing first change value and a second change value. That is, the first adjustment is made by using the first change value or the second change value, the second adjustment is made by using the difference between the first change value and a set value, or the difference between the second change value and a set value, and so on, until the frame synchronization between the master and slave cameras is achieved.

[0078] As can be seen, in the method of this embodiment, the slave camera in the multi-camera system acquires the first timestamp information of each frame in the master video frame sequence of the master camera, and the second timestamp information of each frame in the slave video frame sequence of the slave camera. Based on this, it calculates the first time interval between adjacent frames in the master video frame sequence and the second time interval between adjacent frames in the slave video frame sequence, thereby setting the frame rate of the slave camera to be the same as that of the master camera. Further, based on the same frame rate between the master and slave cameras, the frame rate of the slave camera is adjusted by combining the timestamp difference between the second timestamp information of a specific frame in the slave video frame sequence and the first timestamp information of the first reference frame in the master video frame sequence, and the frame interval between adjacent frames in the slave video frame sequence, so as to gradually achieve frame synchronization between the master and slave cameras. In this way, by further synchronizing related frames on the basis of frame rate synchronization between the master and slave cameras, each slave camera and the master camera can simultaneously shoot the target scene at the same time, avoiding the phenomenon of disjointed images during post-processing.

[0079] The following is a specific application example to illustrate the multi-camera frame synchronization method of the present invention, which can be applied as described above. Figure 1 In the multi-camera system shown, specifically as follows: Figure 5 As shown, any slave camera can achieve multi-camera frame synchronization in the following way:

[0080] Step 201: Obtain the first timestamp information M1, M2, ..., Mn of each frame in the main video frame sequence captured by the main camera for the target scene in the multi-camera system.

[0081] Step 202: Obtain the second timestamp information S1, S2, ..., Sn of each frame in the video frame sequence captured by the camera for the target scene.

[0082] Step 203: Calculate the first time interval between adjacent frames based on the first timestamp information of each frame in the main video frame sequence, and calculate the second time interval between adjacent frames based on the second timestamp information of each frame in the secondary video frame sequence.

[0083] Step 204: Based on the first time interval and the second time interval, the frame rate of the slave camera is set to be the same as that of the master camera to obtain the set frame rate of the slave camera.

[0084] Step 205: Obtain the second timestamp information of each frame in the video frame sequence based on the set frame rate.

[0085] Step 206: Calculate the timestamp difference between the second timestamp information of a specific frame in the video frame sequence and the first timestamp information of the first reference frame in the main video frame sequence, and the frame interval between adjacent frames in the video frame sequence; the first reference frame is the frame in the main video frame sequence that is closest to the specific frame before the specific frame.

[0086] Step 207: If the timestamp difference calculated in step 206 is less than half of the frame interval and greater than a critical value, increase the frame rate of the camera to obtain the increased frame rate. The specific method for increasing the frame rate is as described in the above embodiments and will not be repeated here.

[0087] Furthermore, regarding the second timestamp information of the subsequent frames of a specific frame in the video frame sequence obtained by increasing the frame rate, if the timestamp difference between the second timestamp information of the subsequent frame and the first timestamp information of the corresponding reference frame in the main video frame sequence reaches the first condition, that is, less than or equal to the critical value, the following step 208 will be executed; if the first condition is not met, then based on increasing the frame rate, the above step 206 will be returned to be executed.

[0088] Step 208 involves adjusting the frame rate after increasing it back to the frame rate set by the slave camera, thereby achieving frame synchronization between the slave camera and the master camera.

[0089] Step 209: If the timestamp difference calculated in step 206 is greater than half of the frame interval and less than the difference between the frame interval and the threshold, reduce the frame rate of the camera to obtain the reduced frame rate. The specific method for reducing the frame rate is as described in the above embodiments and will not be repeated here.

[0090] Furthermore, regarding the second timestamp information of subsequent frames of a specific frame in the video frame sequence obtained after reducing the frame rate, if the timestamp difference between the second timestamp information of the subsequent frame and the first timestamp information of the corresponding reference frame in the main video frame sequence reaches the second condition, that is, the absolute difference between the timestamp difference and the frame interval is less than or equal to the critical value, the following step 210 will be executed; if the second condition is not met, then based on the reduced frame rate, the above step 206 will be executed again.

[0091] Step 210: After reducing the frame rate, adjust it back to the frame rate set by the slave camera, thereby achieving frame synchronization between the slave camera and the master camera.

[0092] As can be seen, by further synchronizing related frames on the basis of frame rate synchronization between master and slave cameras, each slave camera and the master camera can simultaneously capture the target scene at the same time, avoiding the phenomenon of fragmentation when stitching the images in the later stage.

[0093] Furthermore, in this embodiment, different adjustment ranges (first change value or second change value) are selected based on the magnitude of the timestamp difference. The timestamp difference reflects the gap between the two frames with the closest time distance in the master and slave video frame sequences. The frame rate of the slave camera is adjusted according to this gap. That is, when the gap is large, a larger adjustment is made to the frame rate of the slave camera, which can quickly achieve frame synchronization between the slave camera and the master camera. When the gap is small, a smaller adjustment is made, that is, a fine-grained adjustment, so that even small differences between the frames of the master and slave cameras are synchronized. In this way, higher precision and more efficient frame synchronization can be achieved by changing the frame rate adjustment range.

[0094] This invention also provides a multi-camera frame synchronization device, mainly applied to the slave camera in the aforementioned multi-camera system, as shown in the schematic diagram below. Figure 6 As shown, it can specifically include:

[0095] The first acquisition unit 10 is used to acquire the first timestamp information of each frame in the main video frame sequence captured by the main camera for the target scene in the multi-camera system.

[0096] The first acquisition unit 10 is specifically used to receive the signal transmitted by the main camera at the start time of shooting each frame in the main video frame sequence; and to record the current system timestamp as the first timestamp information according to the transmitted signal.

[0097] The second acquisition unit 11 is used to acquire the second timestamp information of each frame in the video frame sequence captured by the camera for the target scene;

[0098] The frame rate setting unit 12 is used to calculate the first time interval between adjacent frames in the main video frame sequence and the second time interval between adjacent frames in the secondary video frame sequence based on the first timestamp information and the second timestamp information obtained by the first acquisition unit 10 and the second acquisition unit 11 respectively, and set the frame rate of the secondary camera to be the same as the frame rate of the main camera based on the first time interval and the second time interval, so as to obtain the set frame rate of the secondary camera.

[0099] The difference calculation unit 13 is used to calculate the timestamp difference between the second timestamp information of a specific frame in the video frame sequence based on the set frame rate and the first timestamp information of the first reference frame in the main video frame sequence, and to calculate the frame interval between adjacent frames in the video frame sequence based on the set frame rate; the first reference frame is the frame in the main video frame sequence that is closest to the specific frame before the specific frame.

[0100] The adjustment unit 14 is used to adjust the frame rate of the slave camera based on the timestamp difference and frame interval calculated by the difference calculation unit 13, so that the subsequent frames of a specific frame in the slave video frame sequence are gradually synchronized with the subsequent frames of the first reference frame in the main video frame sequence.

[0101] The adjustment unit 14 is specifically used to increase the frame rate of the slave camera if the timestamp difference is less than half of the frame interval and greater than a threshold value, thereby obtaining an increased frame rate. If the timestamp difference is greater than half of the frame interval and less than the difference between the frame interval and the threshold value, the frame rate of the slave camera is decreased, thereby obtaining a decreased frame rate.

[0102] Specifically, when adjusting the frame rate, if the timestamp difference is greater than a set difference, the adjustment unit 14 increases the frame rate of the slave camera by a first change value to obtain an increased frame rate; if the timestamp difference is less than or equal to the set difference, the adjustment unit 14 increases the frame rate of the slave camera by a second change value to obtain an increased frame rate, wherein the first change value is greater than the second change value. When adjusting the frame rate, if the absolute difference between the timestamp difference and the frame interval is greater than a set difference, the adjustment unit 14 decreases the frame rate of the slave camera by a first change value to obtain a decreased frame rate; if the absolute difference between the timestamp difference and the frame interval is less than or equal to the set difference, the adjustment unit 14 decreases the frame rate of the slave camera by a second change value to obtain a decreased frame rate.

[0103] Furthermore, after increasing the frame rate of the slave camera, the second acquisition unit 11 is further configured to acquire the second timestamp information of the subsequent frames of the specific frame based on the increased frame rate. Thus, the adjustment unit 14 is further configured to adjust the increased frame rate back to the set frame rate of the slave camera when the timestamp difference between the second timestamp difference of the subsequent frames and the first timestamp information of the reference frame corresponding to the subsequent frames in the main video frame sequence is less than or equal to the threshold value; when the timestamp difference between the second timestamp difference of the subsequent frames and the first timestamp information of the reference frame corresponding to the subsequent frames in the main video frame sequence is greater than the threshold value, the step of increasing the frame rate of the slave camera is returned based on the updated timestamp difference and the frame interval.

[0104] Furthermore, after reducing the frame rate of the slave camera, the second acquisition unit 11 is further configured to acquire the second timestamp information of the second subsequent frame of the specific frame based on the reduced frame rate, and update the timestamp difference. Thus, the adjustment unit 14 is further configured to adjust the reduced frame rate back to the set frame rate of the slave camera when the absolute difference between the second timestamp difference of the subsequent frame and the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence is less than or equal to a critical value; when the absolute difference between the second timestamp difference of the subsequent frame and the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence is greater than the critical value, based on the updated timestamp difference and frame interval, return to the step of reducing the frame rate of the slave camera.

[0105] In the device of this embodiment, by further synchronizing related frames on the basis of frame rate synchronization between the master and slave cameras, each slave camera and the master camera can simultaneously capture the target scene at the same time, avoiding the phenomenon of fragmentation when stitching the images later.

[0106] This invention also provides a terminal device, the structural schematic of which is shown below. Figure 7 As shown, the terminal device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 20 (e.g., one or more processors) and memory 21, and one or more storage media 22 (e.g., one or more mass storage devices) for storing application programs 221 or data 222. The memory 21 and storage media 22 can be temporary or persistent storage. The program stored in the storage media 22 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the terminal device. Furthermore, the CPU 20 may be configured to communicate with the storage media 22 and execute the series of instruction operations in the storage media 22 on the terminal device.

[0107] Specifically, the application program 221 stored in the storage medium 22 includes a multi-camera frame synchronization application, and this program may include the first acquisition unit 10, the second acquisition unit 11, the frame rate setting unit 12, the difference calculation unit 13, and the adjustment unit 14 in the aforementioned multi-camera frame synchronization device, which will not be described in detail here. Furthermore, the central processing unit 20 may be configured to communicate with the storage medium 22 and execute a series of operations corresponding to the multi-camera frame synchronization application stored in the storage medium 22 on the terminal device.

[0108] The terminal device may also include one or more power supplies 23, one or more wired or wireless network interfaces 24, one or more input / output interfaces 25, and / or one or more operating systems 223, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0109] The steps performed by the multi-camera frame synchronization device (i.e., the slave camera) in the above method embodiments can be based on this. Figure 7 The structure of the terminal device shown is illustrated.

[0110] Furthermore, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of computer programs adapted for loading by a processor and executing the multi-camera frame synchronization method performed from a camera as described above.

[0111] Furthermore, embodiments of the present invention also provide a terminal device, including a processor and a memory;

[0112] The memory is used to store multiple computer programs, which are loaded and executed by a processor as described above for the multi-camera frame synchronization method executed from the camera; the processor is used to implement each of the multiple computer programs.

[0113] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0114] The foregoing has provided a detailed description of a multi-camera frame synchronization method, apparatus, and storage medium provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-camera frame synchronization method for a multi-camera system having one master camera and multiple slave cameras, characterized in that, include: Obtain the first timestamp information of each frame in the main video frame sequence captured by the main camera of the multi-camera system for the target scene; Obtain the second timestamp information of each frame in the video frame sequence captured by the camera for the target scene; Calculate the first time interval between adjacent frames in the main video frame sequence and the second time interval between adjacent frames in the secondary video frame sequence, and based on the first time interval and the second time interval, set the frame rate of the secondary camera to be the same as the frame rate of the main camera to obtain the set frame rate of the secondary camera; Calculate the timestamp difference between the second timestamp information of a specific frame in the video frame sequence based on the set frame rate and the first timestamp information of the first reference frame in the main video frame sequence, and calculate the frame interval between adjacent frames in the video frame sequence based on the set frame rate; the first reference frame is the frame in the main video frame sequence that is closest to the specific frame before the specific frame. Based on the timestamp difference and frame interval, the frame rate of the slave camera is adjusted so that subsequent frames of a specific frame in the slave video frame sequence are gradually synchronized with subsequent frames of the first reference frame in the main video frame sequence. The step of adjusting the frame rate of the camera based on the timestamp difference and the frame interval specifically includes: If the timestamp difference is less than half of the frame interval and greater than a threshold value, the frame rate of the slave camera is increased to obtain the increased frame rate. Based on the increased frame rate, obtain the second timestamp information of the subsequent frames of the specific frame, and update the timestamp difference; When the second timestamp difference of the subsequent frame is less than or equal to the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence, the increased frame rate is adjusted back to the frame rate set by the camera. When the second timestamp difference of the subsequent frame is greater than the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence, the step of increasing the frame rate of the slave camera is returned based on the updated timestamp difference.

2. The method as described in claim 1, characterized in that, The acquisition of the first timestamp information of each frame in the main video frame sequence captured by the main camera in the multi-camera system for the target scene specifically includes: Receive the signal transmitted by the main camera at the start time of capturing each frame in the main video frame sequence; Based on the transmitted signal, the current system timestamp is recorded as the first timestamp information.

3. The method as described in claim 1, characterized in that, The process of increasing the frame rate from the camera to obtain the increased frame rate specifically includes: If the timestamp difference is greater than a set difference, the frame rate of the slave camera is increased by a first change value to obtain an increased frame rate; if the timestamp difference is less than or equal to the set difference, the frame rate of the slave camera is increased by a second change value to obtain an increased frame rate; wherein the first change value is greater than the second change value.

4. The method as described in claim 1 or 2, characterized in that, The step of adjusting the frame rate of the camera based on the timestamp difference and the frame interval specifically includes: If the timestamp difference is greater than half of the frame interval but less than the difference between the frame interval and the threshold, the frame rate of the slave camera is reduced to obtain the reduced frame rate.

5. The method as described in claim 4, characterized in that, The reduction of the frame rate from the camera to obtain the reduced frame rate specifically includes: If the absolute difference between the timestamp difference and the frame interval is greater than a set difference, the frame rate of the slave camera is reduced by a first change value to obtain a reduced frame rate; if the absolute difference between the timestamp difference and the frame interval is less than or equal to the set difference, the frame rate of the slave camera is reduced by a second change value to obtain a reduced frame rate; wherein, the first change value is greater than the second change value.

6. The method as described in claim 4, characterized in that, The method further includes: Based on the reduced frame rate, obtain the second timestamp information of the subsequent frames of the specific frame, and update the timestamp difference; When the absolute difference between the second timestamp difference of the subsequent frame and the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence is less than or equal to the threshold value, the reduced frame rate is adjusted back to the frame rate set by the camera. When the absolute difference between the second timestamp difference of the subsequent frame and the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence, and the frame interval is greater than the threshold value, the step of reducing the slave camera frame rate is returned based on the updated timestamp difference.

7. A multi-camera frame synchronization device for a multi-camera system having one master camera and multiple slave cameras, characterized in that, include The first acquisition unit is used to acquire the first timestamp information of each frame in the main video frame sequence captured by the main camera in the multi-camera system for the target scene. The second acquisition unit is used to acquire the second timestamp information of each frame in the video frame sequence captured by the camera for the target scene; A frame rate setting unit is used to calculate a first time interval between adjacent frames in the main video frame sequence and a second time interval between adjacent frames in the secondary video frame sequence, and based on the first time interval and the second time interval, set the frame rate of the secondary camera to be the same as the frame rate of the main camera, thereby obtaining the set frame rate of the secondary camera. The difference calculation unit is used to calculate the timestamp difference between the second timestamp information of a specific frame in the video frame sequence based on the set frame rate and the first timestamp information of the first reference frame in the main video frame sequence, and to calculate the frame interval between adjacent frames in the video frame sequence based on the set frame rate; the first reference frame is the frame in the main video frame sequence that is closest to the specific frame before the specific frame. The adjustment unit is used to adjust the frame rate of the slave camera according to the timestamp difference and the frame interval, so that the subsequent frames of a specific frame in the slave video frame sequence are gradually synchronized with the subsequent frames of the first reference frame in the main video frame sequence. The adjustment unit is specifically used to increase the frame rate of the camera if the timestamp difference is less than half of the frame interval and greater than a threshold value, thereby obtaining an improved frame rate. Based on the increased frame rate, obtain the second timestamp information of the subsequent frames of the specific frame, and update the timestamp difference; When the second timestamp difference of the subsequent frame is less than or equal to the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence, the increased frame rate is adjusted back to the frame rate set by the camera. When the second timestamp difference of the subsequent frame is greater than the first timestamp information of the reference frame corresponding to the subsequent frame in the main video frame sequence, the step of increasing the frame rate of the slave camera is returned based on the updated timestamp difference.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of computer programs adapted to be loaded by a processor and executed as described in any one of claims 1 to 6.

Images