Video processing method, device and system

Abstract

The embodiment of the invention provides a video processing method, device and system, and relates to the technical field of video processing. The method comprises the following steps: acquiring a video frame group; acquiring a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and the reference time difference of each video shooting device; obtaining a relative time difference of each video frame according to the reference timestamp of each video frame and the reference timestamp of a target video frame in the video frame group; and according to the relative time difference of each video frame, determining whether the video frame group is a synchronous video frame group formed by the video frames shot by the plurality of video shooting devices based on the same trigger signal.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, device and system. Background Art

[0002] With the development of science and technology, three-dimensional video technology continues to develop. At present, three-dimensional reconstruction can be performed based on two-dimensional video frames shot from multiple different perspectives to obtain the corresponding three-dimensional model.

[0003] However, when reconstructing a three-dimensional model based on multiple two-dimensional video frames, it is necessary to reconstruct the three-dimensional model based on multiple two-dimensional video frames shot at the same time. Therefore, how to judge whether multiple video frames shot from different perspectives are video frames shot at the same time becomes a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, embodiments of the present application provide a video processing method, apparatus, and system for determining whether video frames captured by multiple video capture devices are video frames captured at the same time.

[0005] In order to achieve the above purpose, the embodiments of the present application provide the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a video processing method, the method comprising:

[0007] Acquire a video frame group, where the video frame group includes video frames respectively captured by a plurality of video capturing devices, and any video frame includes a capturing timestamp of the video frame;

[0008] Obtaining a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and a reference time difference of each video shooting device, wherein the reference time difference of any video shooting device is a difference between a system time of the video shooting device and a system time of a target video shooting device among the multiple video shooting devices;

[0009] Obtaining a relative time difference of each video frame according to a reference time stamp of each video frame and a reference time stamp of a target video frame in the video frame group; the target video frame is a video frame captured by the target video capturing device;

[0010] According to the relative time difference of each video frame, it is determined whether the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0011] As an optional implementation of the embodiment of the present application, determining whether the video frame group is a synchronous video frame group composed of video frames captured by the multiple video capture devices based on the same trigger signal according to the relative time difference of each video frame in the video frame group includes:

[0012] Determining whether a relative time difference between each video frame in the video frame group is less than or equal to a first threshold time difference;

[0013] If the relative time difference of each video frame in the video frame group is less than or equal to the first threshold time difference, determining that the video frame group is a synchronous video frame group composed of video frames captured by the multiple video capture devices based on the same trigger signal;

[0014] If the relative time difference of one or more video frames in the video frame group is greater than the first threshold time difference, it is determined that the video frame group is not a synchronous video frame group composed of video frames captured by the multiple video capturing devices based on the same trigger signal.

[0015] As an optional implementation of the embodiment of the present application, the method further includes:

[0016] In the case where it is determined that the video frame group is not a synchronous video frame group consisting of video frames shot by the multiple video shooting devices based on the same trigger signal, replacing the first video frame in the video frame group with the second video frame, and reacquiring the relative time difference of each video frame in the video frame group, and determining whether the video frames in the video frame group are video frames shot by the multiple video shooting devices based on the same trigger signal according to the relative time difference of each video frame in the video frame group, until it is determined that the video frame group is a synchronous video frame group consisting of video frames shot by the multiple video shooting devices based on the same trigger signal;

[0017] The first video frame is a video frame with the smallest relative time difference in the video frame group, and the second video frame is a next video frame of the first video frame captured by a video capture device that captured the first video frame.

[0018] As an optional implementation of the embodiment of the present application, the method further includes:

[0019] After determining that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal, the reference time difference of each video shooting device is updated according to the relative time difference of each video frame in the synchronous video frame group.

[0020] As an optional implementation of the embodiment of the present application, before obtaining the reference timestamp of each video frame in the video frame group according to the shooting timestamp of each video frame in the video frame group and the reference time difference of each video shooting device, the method further includes:

[0021] Receiving respectively the video frames captured by the plurality of video capturing devices in response to the initial trigger signal;

[0022] The reference time difference of each video capture device is calculated according to the capture timestamps of the video frames captured by the multiple video capture devices in response to the initial trigger signal and the capture timestamps of the video frames captured by the target video capture device in response to the trigger signal.

[0023] As an optional implementation of the embodiment of the present application, the method further includes:

[0024] In the case where it is determined that the video frame group is a synchronous video frame group, preprocessing the video frames in the synchronous video frame group based on multiple preprocessing modules to obtain a preprocessed video frame group;

[0025] The pre-processed video frame group is sent to a synchronization device.

[0026] In a second aspect, an embodiment of the present application provides a video processing method, the method comprising:

[0027] Acquire a video frame group set, wherein the video frame group set includes a plurality of pre-processed video frame groups respectively sent by a plurality of pre-processing devices in the video acquisition system;

[0028] According to the reference timestamps of the preprocessed video frame groups in the video frame group set and the reference timestamps of the target preprocessed video frame group in the video frame group set, the relative time difference of the preprocessed video frame groups is obtained; the reference timestamp of any preprocessed video frame group is the reference timestamp of any video frame in the preprocessed video frame group;

[0029] According to the relative time difference of each pre-processed video frame group, it is determined whether each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal.

[0030] As an optional implementation of the embodiment of the present application, determining whether each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal according to the relative time difference of each pre-processed video frame group includes:

[0031] Determining whether a relative time difference of each pre-processed video frame group in the video frame group set is less than or equal to a second threshold time difference;

[0032] If the relative time difference of each pre-processed video frame group in the video frame group set is less than or equal to the second threshold time difference, determining that each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal;

[0033] If the relative time difference of one or more preprocessed video frame groups in the video frame group is greater than the second threshold time difference, it is determined that the preprocessed video frame groups in the synchronous video group set are not preprocessed video frame groups obtained based on the same trigger signal.

[0034] As an optional implementation of the embodiment of the present application, the method further includes:

[0035] In the case of determining whether each preprocessed video frame group in the synchronous video group set is not a preprocessed video frame group obtained based on the same trigger signal, replacing the first preprocessed video frame group in the video frame group with the second preprocessed video frame group, and reacquiring the relative time difference of each preprocessed video frame group in the synchronous video group set, and determining whether each preprocessed video frame group in the synchronous video group set is a preprocessed video frame group obtained based on the same trigger signal according to the relative time difference of each preprocessed video frame group, until it is determined that each preprocessed video frame group in the synchronous video group set is a preprocessed video frame group obtained based on the same trigger signal;

[0036] The first preprocessed video frame group is the preprocessed video frame group with the smallest relative time difference in the video frame group set, and the second preprocessed video frame group is the next preprocessed video frame group sent by the preprocessing device that sends the first preprocessed video frame group.

[0037] As an optional implementation of the embodiment of the present application, the method further includes:

[0038] Obtaining the queue length of each preprocessing device, the queue length of any preprocessing device being the length of the input queue of the preprocessing module that takes the longest time for a single video frame in the preprocessing module of the preprocessing device;

[0039] Based on the queue length, preset frame rate and model prediction control algorithm of each pre-processing device, the predicted frame rate of each pre-processing device in the next video frame acquisition cycle is obtained;

[0040] The frame rate of video frame acquisition performed by each video shooting device is controlled based on the minimum value among the predicted frame rates of each pre-processing device.

[0041] As an optional implementation of the embodiment of the present application, the target frame rate of each preprocessing device in the next video frame acquisition cycle is obtained based on the queue length, preset frame rate and model prediction control algorithm of each preprocessing device, including:

[0042] The target frame rate of each preprocessing device in the next video frame acquisition cycle is obtained based on the queue length of each preprocessing device, the preset frame rate, and the model predictive control algorithm and the following formula:

[0043]

[0044] Among them, k, α i , β i , γ i are preset constants, L t0+it is the queue length of the preprocessing device in the i-th video frame acquisition cycle, R t0+it is the predicted frame rate of the preprocessing device in the i-th video frame acquisition cycle, is the preset frame rate.

[0045] In a third aspect, an embodiment of the present application provides a video processing device, the device comprising:

[0046] A first acquisition unit, configured to acquire a video frame group, wherein the video frame group includes video frames respectively captured by a plurality of video capture devices, and any video frame includes a capture timestamp of the video frame;

[0047] A second acquisition unit is used to acquire a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and the reference time difference of each video shooting device, wherein the reference time difference of any video shooting device is the difference between the system time of the video shooting device and the system time of a target video shooting device among the multiple video shooting devices;

[0048] A third acquisition unit is used to acquire a relative time difference of each video frame according to a reference time stamp of each video frame and a reference time stamp of a target video frame in the video frame group; the target video frame is a video frame captured by the target video capture device;

[0049] The determination unit is used to determine whether the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal according to the relative time difference of each video frame.

[0050] As an optional implementation of the embodiment of the present application, the determination unit is specifically used to determine whether the relative time difference of each video frame in the video frame group is less than or equal to a first threshold time difference; if the relative time difference of each video frame in the video frame group is less than or equal to the first threshold time difference, then it is determined that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal; if the relative time difference of one or more video frames in the video frame group is greater than the first threshold time difference, then it is determined that the video frame group is not a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0051] As an optional implementation of the embodiment of the present application, the determination unit is also used to replace the first video frame in the video frame group with the second video frame when it is determined that the video frame group is not a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal, and re-acquire the relative time difference of each video frame in the video frame group, and determine whether the video frames in the video frame group are video frames shot by the multiple video shooting devices based on the same trigger signal according to the relative time difference of each video frame in the video frame group, until it is determined that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal; wherein, the first video frame is the video frame with the smallest relative time difference in the video frame group, and the second video frame is the next video frame of the first video frame shot by the video shooting device that shot the first video frame.

[0052] As an optional implementation of the embodiment of the present application, the determination unit is also used to update the reference time difference of each video shooting device according to the relative time difference of each video frame in the synchronous video frame group after determining that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0053] As an optional implementation of the embodiment of the present application, the first acquisition unit is also used to respectively receive the video frames shot by the multiple video shooting devices in response to the initial trigger signal; and calculate the reference time difference of each video shooting device based on the shooting timestamps of the video frames shot by the multiple video shooting devices in response to the initial trigger signal, and the shooting timestamps of the video frames shot by the target video shooting device in response to the trigger signal.

[0054] As an optional implementation of the embodiment of the present application, the determination unit is also used to, when determining that the video frame group is a synchronous video frame group, preprocess the video frames in the synchronous video frame group based on multiple preprocessing modules to obtain a preprocessed video frame group; and send the preprocessed video frame group to the synchronization device.

[0055] In a fourth aspect, an embodiment of the present application provides a video processing device, the device comprising:

[0056] An acquisition unit, configured to acquire a video frame group set, wherein the video frame group set includes a plurality of pre-processed video frame groups respectively sent by a plurality of pre-processing devices in the video acquisition system;

[0057] A processing unit, configured to obtain a relative time difference between each preprocessed video frame group according to a reference time stamp of each preprocessed video frame group in the video frame group set and a reference time stamp of a target preprocessed video frame group in the video frame group set; the reference time stamp of any preprocessed video frame group is a reference time stamp of any video frame in the preprocessed video frame group;

[0058] The determination unit is used to determine whether each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal according to the relative time difference of each pre-processed video frame group.

[0059] As an optional implementation of the embodiment of the present application, the determination unit is specifically used to determine whether the relative time difference of each preprocessed video frame group in the video frame group set is less than or equal to a second threshold time difference; if the relative time difference of each preprocessed video frame group in the video frame group set is less than or equal to the second threshold time difference, then it is determined that each preprocessed video frame group in the synchronized video group set is a preprocessed video frame group obtained based on the same trigger signal; if the relative time difference of one or more preprocessed video frame groups in the video frame group is greater than the second threshold time difference, then it is determined that each preprocessed video frame group in the synchronized video group set is not a preprocessed video frame group obtained based on the same trigger signal.

[0060] As an optional implementation of the embodiment of the present application, the determination unit is also used to replace the first preprocessed video frame group in the video frame group with the second preprocessed video frame group when determining whether the respective preprocessed video frame groups in the synchronized video group set are not preprocessed video frame groups obtained based on the same trigger signal, and reacquire the relative time difference of the respective preprocessed video frame groups in the synchronized video group set, and determine whether the respective preprocessed video frame groups in the synchronized video group set are preprocessed video frame groups obtained based on the same trigger signal according to the relative time difference of the respective preprocessed video frame groups, until it is determined that the respective preprocessed video frame groups in the synchronized video group set are preprocessed video frame groups obtained based on the same trigger signal; wherein, the first preprocessed video frame group is the preprocessed video frame group with the smallest relative time difference in the video frame group set, and the second preprocessed video frame group is the next group of preprocessed video frame groups sent by the preprocessing device that sends the first preprocessed video frame group.

[0061] As an optional implementation of the embodiment of the present application, the video processing device also includes: a calculation unit, which is specifically used to obtain the queue length of each preprocessing device, the queue length of any preprocessing device being the length of the input queue of the preprocessing module in which a single video frame takes the longest time in the preprocessing module of the preprocessing device; based on the queue length of each preprocessing device, a preset frame rate and a model prediction control algorithm, obtaining the predicted frame rate of each preprocessing device in the next video frame acquisition cycle; and controlling the frame rate of each video shooting device for video frame acquisition based on the minimum value of the predicted frame rates of each preprocessing device.

[0062] As an optional implementation of the embodiment of the present application, the computing unit is specifically used for:

[0063] The target frame rate of each preprocessing device in the next video frame acquisition cycle is obtained based on the queue length of each preprocessing device, the preset frame rate, and the model predictive control algorithm and the following formula:

[0064]

[0065] Among them, k, α i , β i , γ i are preset constants, L t0+it is the queue length of the preprocessing device in the i-th video frame acquisition cycle, R t0+it is the predicted frame rate of the preprocessing device in the i-th video frame acquisition cycle, is the preset frame rate.

[0066] In a fifth aspect, an embodiment of the present application provides a video processing system, comprising: a plurality of video shooting devices, a plurality of pre-processing devices, a synchronization device, and a trigger; wherein the plurality of video shooting devices are respectively connected to one of the pre-processing devices, and are all connected to the trigger, the plurality of pre-processing devices are all connected to the synchronization device, and the synchronization device is connected to the trigger;

[0067] The multiple pre-processing devices are respectively used to execute the video processing method described in the first aspect above;

[0068] The synchronization device is used to execute the video processing method described in the second aspect above.

[0069] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program and the processor is used to enable the electronic device to implement the video processing method described in any of the above embodiments when executing the computer program.

[0070] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which, when the computer program is executed by a computing device, enables the computing device to implement the video processing method described in any of the above-mentioned embodiments.

[0071] In an eighth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to implement the video processing method described in any one of the above-mentioned embodiments.

[0072] The video processing method provided by the embodiment of the present application is applied to a preprocessing device in a video acquisition system, including: first obtaining a video frame group, wherein the video frame group includes video frames respectively shot by multiple video shooting devices, and any video frame includes a shooting timestamp of the video frame; according to the shooting timestamps of the respective video frames in the video frame group and the reference time difference of each video shooting device, obtaining the reference timestamp of each video frame, the reference time difference of any video shooting device being the difference between the system time of the video shooting device and the system time of a target video shooting device among the multiple video shooting devices; according to the reference timestamps of the respective video frames and the reference timestamp of the target video frame in the video frame group, obtaining the relative time difference of the respective video frames; the target video frame is a video frame shot by the target video shooting device; according to the relative time difference of the respective video frames, determining whether the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal. The embodiment of the present application uses the difference between the shooting timestamp of each video frame in the video frame group and the system time of the target video shooting device in the video frame group as a reference time difference, and then obtains the reference timestamp of each video frame, and then obtains the relative time difference of each video frame according to the reference timestamp of each video frame and the reference timestamp of the target video frame, and finally determines whether the video frame group is a synchronous video frame group according to the relative time difference of each video frame. Therefore, the embodiment of the present application can determine whether each video frame in the video frame group is shot at the same time according to the relative time difference of each video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0075] Figure 1A scene architecture diagram of the video processing method provided in an embodiment of the present application;

[0076] Figure 2 One of the step flow charts of the video processing method provided in the embodiment of the present application;

[0077] Figure 3 The second step flow chart of the video processing method provided in the embodiment of the present application;

[0078] Figure 4 A schematic diagram of a video processing method provided in an embodiment of the present application;

[0079] Figure 5 The third step flow chart of the video processing method provided in the embodiment of the present application;

[0080] Figure 6 A schematic diagram of a framework of a preprocessing device for a video processing method provided in an embodiment of the present application;

[0081] Figure 7 This is a flowchart of the video processing method provided in the embodiment of the present application;

[0082] Figure 8 The fifth step flow chart of the video processing method provided in the embodiment of the present application;

[0083] Fig. 9 The sixth step flow chart of the video processing method provided in the embodiment of the present application;

[0084] Fig.10 The seventh step flow chart of the video processing method provided in the embodiment of the present application;

[0085] Fig.11 One of the structural schematic diagrams of the video processing device provided in the embodiment of the present application;

[0086] Fig.12 The second structural diagram of the video processing device provided in the embodiment of the present application;

[0087] Fig.13 The third structural diagram of the video processing device provided in the embodiment of the present application;

[0088] Fig.14 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0090] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0091] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0092] The following first describes the scenario architecture of the video processing method provided in the embodiment of the present application.

[0093] Reference Figure 1 As shown, the scene architecture of the video processing method provided in the embodiment of the present application includes: multiple video shooting devices 100, multiple pre-processing devices 200, a synchronization device 300, a trigger 400 and a reconstruction module. Among them, the multiple video shooting devices 100 are respectively connected to one of the preprocessing devices 200, and are all connected to the trigger 400. The multiple preprocessing devices 200 are all connected to the synchronization device 300, and the synchronization device 300 is connected to the trigger 400; the multiple video shooting devices 100 are used to obtain corresponding video frames based on the trigger signal to form a synchronized video frame group, and send the video frame group to the corresponding preprocessing device 200; the multiple preprocessing devices 200 are used to preprocess the video frames in the synchronized video frame group, obtain the preprocessed video frame group, and send the preprocessed video frame group to the synchronization device 300; the synchronization device 300 is used to determine whether each preprocessed video frame group in the synchronized video group set is a preprocessed video frame group obtained based on the same trigger signal according to the relative time difference of each preprocessed video frame group; the trigger 400 is used to send an initial trigger signal to the multiple video shooting devices 100; the reconstruction module 500 is used to obtain the corresponding three-dimensional model according to the preprocessed video frame group.

[0094] Among them, the multiple video shooting devices 100 can be USB industrial cameras or other video shooting devices with shooting functions, and this application does not impose any restrictions on this; wherein, the trigger setting can be implemented based on a 32-bit microcontroller unit (Microcontroller Unit, MCU).

[0095] The present application embodiment provides a video processing method, which is applied to a preprocessing device in a video acquisition system, referring to Figure 2 As shown, the video processing method includes the following steps S201 to S204:

[0096] S201: Obtain a video frame group.

[0097] The video frame group includes video frames respectively shot by a plurality of video shooting devices, and any video frame includes a shooting timestamp of the video frame.

[0098] In the embodiment of the present application, it is necessary to use a plurality of the video shooting devices to shoot the same target area at different angles respectively, so as to more comprehensively capture the features of the target area and reconstruct an accurate three-dimensional model.

[0099] In some embodiments, the shooting timestamp is a timestamp set for a video frame shot by a video shooting device based on local time, and specifically may be a moment when exposure starts during the shooting process.

[0100] Exemplarily, taking the video frame group as a synchronous video frame group as an example, when the video frame group includes: video frame 1, video frame 2, video frame 3; they are from video capture device A, video capture device B, and video capture device C respectively. The system time of video capture device A when shooting video frame 1 is 9:00:00:000, recorded as shooting timestamp T1, the system time of video capture device B when shooting video frame 2 is 9:00:00:003, recorded as shooting timestamp T2, and the system time of video capture device C when shooting video frame 3 is 9:00:00:005, recorded as shooting timestamp T3, then the shooting timestamps of video frame 1, video frame 2, and video frame 3 are T1, T2, and T3.

[0101] S202: Acquire a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and a reference time difference of each video shooting device.

[0102] The reference time difference of any video capture device is the difference between the system time of the video capture device and the system time of a target video capture device among the multiple video capture devices.

[0103] In some embodiments, the target video capture device among the multiple video capture devices can randomly designate one as the target video capture device, use it as the target video capture device, and then obtain the system time of the target video capture device. Exemplarily, based on the embodiment in the above step S21, when the video frame group includes: video frame 1 shot by video capture device A, video frame 2 shot by video capture device B, and video frame 3 shot by video capture device C, video capture device A is used as the target video capture device, then the reference time difference of video capture device A is the system time 9:00:00:000 when shooting video frame 1 and the system time 9:00:00:000 of the target video capture device, which is 0ms (milliseconds); similarly, the reference time difference of video frame 2 is 3ms, and the reference time difference of video frame 2 is 5ms.

[0104] The reference timestamp of each video frame is the difference between the shooting timestamp of each video frame and the reference time difference; specifically, the reference timestamp of video frame 1 is: (9:00:00:000-0), recorded as: T1; the reference timestamp of video frame 2 is: (9:00:00:003-3ms), recorded as: T2-3ms; the reference timestamp of video frame 3 is: (9:00:00:005-5ms), recorded as: T3-5ms; it can be seen that after calculation, the reference timestamps of each video frame are consistent.

[0105] S203: Obtain a relative time difference between each video frame according to a reference time stamp of each video frame and a reference time stamp of a target video frame in the video frame group.

[0106] The target video frame is a video frame captured by the target video capturing device.

[0107] In the above embodiment, video frame 1 in the video frame group is used as the target video frame, and its reference timestamp is: T1; the relative time difference of each video frame is the difference between the reference timestamp of each video frame and the reference timestamp of the target video frame; specifically, the reference timestamp of video frame 1 is: T1, and video frame 1 is used as the target video frame, then the relative time difference of video frame 1 is: T1-T1, which is 0; similarly, the relative time difference of video frame 2 is: (T2-3ms)-T1, which is (9:00:00:003-3ms )-9:00:00:000=0; similarly, the relative time difference of video frame 3 is: (T3-2)-T1, that is, (9:00:00:005-5ms)-9:00:00:003=0; it can be seen that the relative time differences of video frame 1, video frame 2, and video frame 3 in the video frame group are all 0, which means that the different shooting timestamps of each video frame in the current video frame group are caused by the different system times of the shooting devices. In fact, video frame 1, video frame 2, and video frame 3 are shot at the same time.

[0108] S204: Determine, according to the relative time difference of each video frame, whether the video frame group is a synchronous video frame group consisting of video frames captured by the multiple video capturing devices based on the same trigger signal.

[0109] In some embodiments, after the relative time difference of each video frame in the video frame group can be obtained based on the above steps S201 to S203, whether the multiple video frames in the video frame group are synchronized can be determined according to the size of the relative time difference.

[0110] The video processing method provided by the embodiment of the present application is applied to a preprocessing device in a video acquisition system, including: first obtaining a video frame group, wherein the video frame group includes video frames respectively shot by multiple video shooting devices, and any video frame includes a shooting timestamp of the video frame; according to the shooting timestamps of the respective video frames in the video frame group and the reference time difference of each video shooting device, obtaining the reference timestamp of each video frame, the reference time difference of any video shooting device being the difference between the system time of the video shooting device and the system time of a target video shooting device among the multiple video shooting devices; according to the reference timestamps of the respective video frames and the reference timestamp of the target video frame in the video frame group, obtaining the relative time difference of the respective video frames; the target video frame is a video frame shot by the target video shooting device; according to the relative time difference of the respective video frames, determining whether the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal. The embodiment of the present application uses the difference between the shooting timestamp of each video frame in the video frame group and the system time of the target video shooting device in the video frame group as a reference time difference, and then obtains the reference timestamp of each video frame, and then obtains the relative time difference of each video frame according to the reference timestamp of each video frame and the reference timestamp of the target video frame, and finally determines whether the video frame group is a synchronous video frame group according to the relative time difference of each video frame. Therefore, the embodiment of the present application can determine whether each video frame in the video frame group is shot at the same time according to the relative time difference of each video frame.

[0111] As an extension and refinement of the above embodiment, the present application embodiment provides another video processing method, referring to Figure 3 As shown, the video processing method includes the following steps S301 to S307:

[0112] S301: Obtain a video frame group.

[0113] The video frame group includes video frames respectively shot by a plurality of video shooting devices, and any video frame includes a shooting timestamp of the video frame.

[0114] S302: Acquire a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and a reference time difference of each video shooting device.

[0115] The reference time difference of any video capture device is the difference between the system time of the video capture device and the system time of a target video capture device among the multiple video capture devices.

[0116] In some embodiments, before the above step S302, the video processing method further includes the following steps 1 and 2:

[0117] Step 1: respectively receiving video frames captured by the multiple video capturing devices in response to an initial trigger signal.

[0118] In some embodiments, the initial trigger signal can be a pulse trigger signal emitted by a trigger, and the trigger can send the initial trigger signal to the multiple video capture devices simultaneously through a 3.5 mm headphone cable. After the multiple video capture devices simultaneously receive the initial trigger signal, they capture video frames from their respective perspectives.

[0119] Step 2: Calculate the reference time difference of each video capture device based on the capture timestamps of the video frames captured by the multiple video capture devices in response to the initial trigger signal and the capture timestamps of the video frames captured by the target video capture device in response to the trigger signal.

[0120] In some embodiments, after the trigger sends the initial trigger signal, the multiple video capture devices will receive the initial trigger signal at the same time, and then shoot in response to the initial trigger signal, obtain their respective video frames, set corresponding shooting timestamps, and calculate corresponding reference time differences.

[0121] S303 . Obtain a relative time difference between each video frame according to a reference time stamp of each video frame and a reference time stamp of a target video frame in the video frame group.

[0122] The target video frame is a video frame captured by the target video capturing device.

[0123] S304: Determine whether the relative time difference of each video frame in the video frame group is less than or equal to a first threshold time difference.

[0124] In some embodiments, the first threshold time difference can be set by the developer according to actual needs, for example, it can be set to 4ms or 5ms or 6ms, which is not limited in the embodiments of the present application.

[0125] Based on the above steps, after obtaining the relative time difference of each video frame in the video frame group, by comparing the relative time difference with the first threshold time difference, it can be determined whether each video frame in the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0126] In the above step S304, if the relative time difference of each video frame in the video frame group is less than or equal to the first threshold time difference, the following step S305 is performed:

[0127] S305: Determine that the video frame group is a synchronous video frame group consisting of video frames captured by the multiple video capturing devices based on the same trigger signal.

[0128] Exemplarily, when the first threshold time difference is set to 3ms, if the relative time difference of each video frame in the video frame group is less than or equal to 3ms, it is determined that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0129] After the above step S305 is executed, the video processing method further includes the following steps:

[0130] After determining that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal, the reference time difference of each video shooting device is updated according to the relative time difference of each video frame in the synchronous video frame group.

[0131] Reference Figure 4 FIG. 1 is a schematic diagram showing the time offset between the shooting timestamp of the video shooting device and the timestamp of the preprocessing device when two video shooting devices are shooting video frames. Figure 4 The horizontal axis shown is the time in the physical world, and the vertical axis is the difference between the shooting timestamp of the video capture device and the timestamp of the preprocessing device; it can be seen that, at the beginning, the timestamp difference between the two video capture devices and the preprocessing device is 0, and there is no situation where the difference between the shooting timestamp of the two video capture devices and the timestamp of the preprocessing device is different, but as time goes by, the video capture device will have a time offset with the preprocessing device, resulting in inaccurate shooting timestamp; at the same time, the difference between the shooting timestamp and the timestamp of the preprocessing device will also be offset between different video capture devices; therefore, it is necessary to update the reference time difference of each video capture device to ensure the accuracy of the data.

[0132] In some embodiments, when the multiple video capture devices capture video frames, the video capture devices may have the above Figure 4 The problem shown or the problem of timestamp drift causes that the various video frames of the video frame group are not shot at the same time. Therefore, the reference time difference of each video shooting device is updated according to the relative time difference of each video frame in the synchronized video frame group to ensure that under the initial trigger signal, the various video frames in the video frame group shot by the multiple video shooting devices are the same and the multiple video frames shot at the same time are synchronized.

[0133] In some embodiments, if it is determined that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal, it means that the relative time difference obtained by the video frame group shot by the multiple video shooting devices currently meets the current reconstruction requirements, and the reference time difference of each video shooting device can be updated based on the relative time difference, which is used to determine whether the video frame group is a synchronous video frame group next time to improve the calculation accuracy.

[0134] Reference Figure 5 As shown, in the case where the video frame group is determined to be a synchronous video frame group, the video processing method further includes the following steps S3051 and S3052:

[0135] S3051. When it is determined that the video frame group is a synchronous video frame group, preprocess the video frames in the synchronous video frame group based on multiple preprocessing modules to obtain a preprocessed video frame group.

[0136] In some embodiments, the preprocessing of the video frames in the synchronized video frame group based on multiple preprocessing modules can be divided into: an image signal processing (ISP) module, a region prediction (ROI) module, a depth estimation module and a foreground mask (Mask) prediction module.

[0137] Among them, the image signal processing module is used to perform some pre-processing on the images in the video frames in the synchronous video frame group, including: debayer, color correction, gamma correction, bad pixel removal, denoising, binocular correction and other processing; the region prediction module is used to obtain specific high-weight areas of the images in the video frames in the synchronous video frame group to focus on optimizing the feature areas and reduce the subsequent amount of calculations; the depth estimation module is used to perform depth estimation on the video frames in the synchronous video frame group processed by the image signal processing module and the region prediction module to obtain depth estimation results; the foreground mask prediction module is used to perform mask segmentation on the depth estimation results to generate the final pre-processed video frame group.

[0138] S3052: Send the pre-processed video frame group to a synchronization device.

[0139] For example, in combination with the above-mentioned step S3051, refer to Figure 6 As shown, it is a schematic diagram of the overall architecture of the preprocessing device 600, including: an image signal processing module 601, a region prediction module 602, a depth estimation module 603, a foreground mask prediction module 604, and a transmission module 605; wherein, the transmission module 605 is used to send the preprocessed video frame group to the synchronization device.

[0140] In an embodiment of the present application, the video frames in the synchronized video frame group are preprocessed based on multiple preprocessing modules to obtain a preprocessed video frame group, which is then sent to a synchronization device, and the synchronization device further processes the preprocessed video frame group.

[0141] In the above step S304, if the relative time difference of one or more video frames in the video frame group is greater than the first threshold time difference, the following step S306 is performed:

[0142] S306: Determine that the video frame group is not a synchronous video frame group consisting of video frames captured by the multiple video capturing devices based on the same trigger signal.

[0143] Exemplarily, when the first threshold time difference is set to 3ms, if the relative time difference of each video frame in the video frame group is greater than 3ms, it is determined that the video frame group is not a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0144] After the above step S306 is executed, when it is determined that the video frame group is not a synchronous video frame group composed of video frames captured by the multiple video capturing devices based on the same trigger signal, the video processing method further includes the following steps:

[0145] S307, replacing the first video frame in the video frame group with the second video frame, and reacquiring the relative time difference of each video frame in the video frame group, and determining whether the video frames in the video frame group are video frames shot by the multiple video shooting devices based on the same trigger signal according to the relative time difference of each video frame in the video frame group, until it is determined that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0146] The first video frame is a video frame with the smallest relative time difference in the video frame group, and the second video frame is a next video frame of the first video frame captured by a video capture device that captured the first video frame.

[0147] In some embodiments, after determining that the video frame group is not a synchronized video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal, it is necessary to further obtain a synchronized video frame group. In an embodiment of the present application, the video frame with the smallest relative time difference in the video frame group is replaced with the next video frame of the video frame with the smallest relative time difference in the video frame group to generate a new video frame group, and steps S201-S204 are repeated until the video frame group is determined, that is, the new video frame group is a synchronized video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0148] The embodiment of the present application also provides another video processing method, which is applied to a synchronization device in a video acquisition system. Figure 7 As shown, the video processing method includes the following steps S701 to S703:

[0149] S701: Obtain a video frame group set.

[0150] The video frame group set includes pre-processed video frame groups respectively sent by a plurality of pre-processing devices in the video acquisition system.

[0151] In some embodiments, the synchronization device is used to synchronously process a plurality of the pre-processed video frame groups, and after receiving the pre-processed video frame groups from the pre-processing device, the synchronization device generates the video frame group set.

[0152] S702: Obtain a relative time difference of each preprocessed video frame group according to a reference time stamp of each preprocessed video frame group in the video frame group set and a reference time stamp of a target preprocessed video frame group in the video frame group set.

[0153] The reference timestamp of any pre-processed video frame group is the reference timestamp of any video frame in the pre-processed video frame group.

[0154] Exemplarily, there are 5 video frame groups in the video frame group set, and each pre-processed video frame group has 3 video frames. According to the above step S702, in these 5 pre-processed video frame groups, the timestamp of any video frame is selected as the reference timestamp, then these 5 pre-processed video frame groups will have 5 reference timestamps, and then based on these 5 reference timestamps and the reference timestamp of the target pre-processed video frame group, the relative time differences corresponding to these 5 pre-processed video frame groups are obtained.

[0155] S703: Determine, according to the relative time difference of each pre-processed video frame group, whether each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal.

[0156] In some embodiments, based on the relative time difference of each preprocessed video frame group, it is determined whether each preprocessed video frame group is a preprocessed video frame group obtained based on the same trigger signal, so as to ensure that each preprocessed video frame group in the synchronous video group set is synchronized, so that in the subsequent three-dimensional reconstruction process, there will be no problems such as video content confusion due to video frame asynchrony, thereby improving the user experience.

[0157] As an extension and refinement of the above embodiment, the present application embodiment provides another video processing method, referring to Figure 8 As shown, the video processing method includes the following steps S801 to S806:

[0158] S801. Obtain a video frame group set.

[0159] The video frame group set includes pre-processed video frame groups respectively sent by a plurality of pre-processing devices in the video acquisition system.

[0160] S802: Obtain a relative time difference of each preprocessed video frame group according to a reference time stamp of each preprocessed video frame group in the video frame group set and a reference time stamp of a target preprocessed video frame group in the video frame group set.

[0161] The reference timestamp of any pre-processed video frame group is the reference timestamp of any video frame in the pre-processed video frame group.

[0162] S803: Determine whether the relative time difference of each pre-processed video frame group in the video frame group set is less than or equal to a second threshold time difference.

[0163] In some embodiments, the second threshold time difference may be set in a manner similar to step S404 described above, and this embodiment of the present application does not limit this.

[0164] Based on the above steps, after obtaining the relative time difference of each preprocessed video frame group in the video frame group set, by comparing the relative time difference with the second threshold time difference, it can be determined whether each preprocessed video frame group in the synchronous video group set is a preprocessed video frame group obtained based on the same trigger signal.

[0165] In the above step S803, if the relative time difference of each preprocessed video frame group in the video frame group set is less than or equal to the second threshold time difference, the following step S804 is executed: S804, determine that each preprocessed video frame group in the synchronous video group set is a preprocessed video frame group obtained based on the same trigger signal.

[0166] In the above step S803, if the relative time difference of one or more pre-processed video frame groups in the video frame groups is greater than the second threshold time difference, the following step S805 is performed:

[0167] S805: Determine that each pre-processed video frame group in the synchronous video group set is not a pre-processed video frame group obtained based on the same trigger signal.

[0168] In some embodiments, after the above step S805 (determining that each pre-processed video frame group in the synchronous video group set is not a pre-processed video frame group obtained based on the same trigger signal), the video processing method further includes the following step S806:

[0169] S806. When determining whether each preprocessed video frame group in the synchronized video group set is not a preprocessed video frame group obtained based on the same trigger signal, replace the first preprocessed video frame group in the video frame group with the second preprocessed video frame group, and reacquire the relative time difference of each preprocessed video frame group in the synchronized video group set, and determine whether each preprocessed video frame group in the synchronized video group set is a preprocessed video frame group obtained based on the same trigger signal according to the relative time difference of each preprocessed video frame group, until it is determined that each preprocessed video frame group in the synchronized video group set is a preprocessed video frame group obtained based on the same trigger signal.

[0170] The first preprocessed video frame group is the preprocessed video frame group with the smallest relative time difference in the video frame group set, and the second preprocessed video frame group is the next preprocessed video frame group sent by the preprocessing device that sends the first preprocessed video frame group.

[0171] In some embodiments, after determining whether each preprocessed video frame group in the synchronized video group set is not a preprocessed video frame group obtained based on the same trigger signal, it is necessary to further determine whether each preprocessed video frame group in the synchronized video group set is a preprocessed video frame group obtained based on the same trigger signal. In an embodiment of the present application, the preprocessed video frame group with the smallest relative time difference in the video frame group set is replaced with the next preprocessed video frame group with the smallest relative time difference in the video frame group set to generate a new synchronized video group set, and steps S701-S703 are repeated until it is determined that the synchronized video group set, that is, each preprocessed video frame group in the new synchronized video frame group set, is a preprocessed video frame group obtained based on the same trigger signal.

[0172] As an extension and refinement of the above embodiment, refer to Fig. 9 As shown, the video processing method further includes the following steps S901 to S903:

[0173] S901. Obtain the queue length of each pre-processing device.

[0174] The queue length of any preprocessing device is the length of the input queue of the preprocessing module that takes the longest time to process a single video frame among the preprocessing modules of the preprocessing device.

[0175] S902: Based on the queue length of each preprocessing device, the preset frame rate and the model prediction control algorithm, obtain the predicted frame rate of each preprocessing device in the next video frame acquisition cycle.

[0176] In an embodiment of the present application, for each of the preprocessing devices, the preprocessing modules in the preprocessing device are obtained, including: an image signal processing module, a region prediction module, a depth estimation module, a foreground mask prediction module, and a length of an input queue of a transmission module. In the above-mentioned multiple preprocessing modules, there may be a large amount of queue congestion in the input queue of the depth estimation module due to the fact that the depth estimation module takes a long time to process the synchronous video frame group. After the processing is completed, other preprocessing modules need to wait, resulting in an unsmooth process when outputting the preprocessed video frame group, which makes it very easy to have errors such as frame drops and no display when reconstructing the three-dimensional model later, and it also takes a lot of time.

[0177] Therefore, based on the length of the input queue of the preprocessing module with the longest time consumption for a single video frame in the preprocessing modules of each preprocessing device obtained in the above step S901, combined with the preset frame rate and the model prediction control algorithm, the predicted frame rate of each preprocessing device in the next video frame acquisition cycle is obtained. The predicted frame rate of each preprocessing device in the next video frame acquisition cycle is the optimal frame rate for the current preprocessing module. Video frames are acquired for each video shooting device based on the predicted frame rate, thereby avoiding the problem of too long input queue of any preprocessing module in the preprocessing device, resulting in unsmooth output of the preprocessed video frame group, thereby improving the effect of reconstructing the three-dimensional model.

[0178] In some embodiments, the implementation method of the above step S902 (obtaining the predicted frame rate of each preprocessing device in the next video frame acquisition cycle based on the queue length, preset frame rate and model prediction control algorithm of each preprocessing device) may be:

[0179] The target frame rate of each preprocessing device in the next video frame acquisition cycle is obtained based on the queue length of each preprocessing device, the preset frame rate, and the model predictive control algorithm and the following formula:

[0180]

[0181] Among them, k, α i , βi , γ i are preset constants, L t0+it is the queue length of the preprocessing device in the i-th video frame acquisition cycle, R t0+it is the predicted frame rate of the preprocessing device in the i-th video frame acquisition cycle, is the preset frame rate.

[0182] In some embodiments, the k, α i , β i , γ i It can be manually set by the developer according to actual needs. The core idea of ​​the Model Predictive Control (MPC) algorithm is to use the existing model, the current state of the system and the future control quantity to predict the future output of the system. Exemplarily, when k in the above calculation formula is 5, it means that multiple predicted frame rates are obtained by calculation based on the data of the current 5 cycles combined with the formula.

[0183] S903: Control the frame rate of each video shooting device for video frame acquisition based on the minimum value of the predicted frame rates of each pre-processing device.

[0184] In an embodiment of the present application, the queue length of each preprocessing device is first obtained, and then the predicted frame rate of each preprocessing device in the next video frame acquisition cycle is obtained based on the queue length of each preprocessing device, the preset frame rate and the model prediction control algorithm; finally, the frame rate of each video capture device for video frame acquisition is controlled based on the minimum value of the predicted frame rate of each preprocessing device; the frame rate of video frame acquisition of each video capture device can be controlled within a reasonable range, the processing process of the preprocessing device is optimized, and the problem of queue squeeze caused by the input queue length of a certain preprocessing device being too long is avoided.

[0185] As an extension and refinement of the above embodiment, refer to Fig.10 FIG. 1 is a flowchart of a video processing method provided in an embodiment of the present application. The video processing method includes the following steps S1001 to S805:

[0186] S1001. A preprocessing device obtains a video frame group.

[0187] The video frame group includes video frames respectively shot by a plurality of video shooting devices, and any video frame includes a shooting timestamp of the video frame.

[0188] S1002. The preprocessing device obtains a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and a reference time difference of each video shooting device, wherein the reference time difference of any video shooting device is a difference between a system time of the video shooting device and a system time of a target video shooting device among the multiple video shooting devices.

[0189] S1003: The preprocessing device obtains a relative time difference between each video frame according to a reference time stamp of each video frame and a reference time stamp of a target video frame in the video frame group.

[0190] S1004: The preprocessing device determines whether the relative time difference of each video frame in the video frame group is less than or equal to a first threshold time difference.

[0191] In the above step S1004, if the relative time difference of each video frame in the video frame group is less than or equal to the first threshold time difference, the following steps S1005 to S1010 are performed:

[0192] S1005: The preprocessing device determines that the video frame group is a synchronous video frame group consisting of video frames captured by the multiple video capturing devices based on the same trigger signal.

[0193] S1006: The preprocessing device preprocesses the video frames in the synchronous video frame group based on multiple preprocessing modules to obtain a preprocessed video frame group.

[0194] S1007: The preprocessing device sends the preprocessed video frame group to the synchronization device.

[0195] S1008. The synchronization device obtains a set of video frame groups.

[0196] The video frame group set includes pre-processed video frame groups respectively sent by a plurality of pre-processing devices in the video acquisition system.

[0197] S1009: The synchronization device obtains a relative time difference of each preprocessed video frame group according to a reference time stamp of each preprocessed video frame group in the video frame group set and a reference time stamp of a target preprocessed video frame group in the video frame group set.

[0198] S1010: The synchronization device determines whether a relative time difference between each pre-processed video frame group in the video frame group set is less than or equal to a second threshold time difference.

[0199] In the above step S1010, if the relative time difference of each pre-processed video frame group in the video frame group set is less than or equal to the second threshold time difference, the following step S1011 is performed:

[0200] S1011. The synchronization device determines that each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal;

[0201] In the above step S1010, if the relative time difference of one or more pre-processed video frame groups in the video frame groups is greater than the second threshold time difference, the following steps S1012 to S1013 are performed:

[0202] S1012: The synchronization device determines that each pre-processed video frame group in the synchronized video group set is not a pre-processed video frame group obtained based on the same trigger signal.

[0203] S1013: The synchronization device replaces the first pre-processed video frame group in the video frame group with the second pre-processed video frame group.

[0204] In the above step S1004, if the relative time difference of one or more video frames in the video frame group is greater than the first threshold time difference, the following steps S1014 to S1015 are performed:

[0205] S1014: The preprocessing device determines that the video frame group is not a synchronous video frame group consisting of video frames captured by the multiple video capturing devices based on the same trigger signal.

[0206] S1015: The preprocessing device replaces the first video frame in the video frame group with the second video frame.

[0207] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a video processing device, which corresponds to the above method embodiment. For ease of reading, this embodiment will no longer repeat the details of the above method embodiment one by one, but it should be clear that the video processing device in this embodiment can correspond to all the contents of the above method embodiment.

[0208] The embodiment of the present application provides a video processing device, wherein the video processing device belongs to a first client device participating in video communication, Fig.11 is a structural diagram of the video processing device, such as Fig.11 As shown, the synchronization server 1100 includes:

[0209] A first acquisition unit 1101 is used to acquire a video frame group, where the video frame group includes video frames respectively captured by a plurality of video capture devices, and any video frame includes a capture timestamp of the video frame;

[0210] A second acquisition unit 1102 is used to acquire a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and a reference time difference of each video shooting device, wherein the reference time difference of any video shooting device is a difference between a system time of the video shooting device and a system time of a target video shooting device among the multiple video shooting devices;

[0211] The third acquisition unit 1103 is used to acquire the relative time difference of each video frame according to the reference time stamp of each video frame and the reference time stamp of the target video frame in the video frame group; the target video frame is the video frame captured by the target video capture device;

[0212] The determination unit 1104 is configured to determine, according to the relative time difference of each video frame, whether the video frame group is a synchronous video frame group consisting of video frames captured by the multiple video capturing devices based on the same trigger signal.

[0213] As an optional implementation of the embodiment of the present application, the determination unit 1104 is specifically used to determine whether the relative time difference of each video frame in the video frame group is less than or equal to a first threshold time difference; if the relative time difference of each video frame in the video frame group is less than or equal to the first threshold time difference, then it is determined that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal; if the relative time difference of one or more video frames in the video frame group is greater than the first threshold time difference, then it is determined that the video frame group is not a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0214] As an optional implementation of the embodiment of the present application, the determination unit 1104 is also used to replace the first video frame in the video frame group with the second video frame when it is determined that the video frame group is not a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal, and re-acquire the relative time difference of each video frame in the video frame group, and determine whether the video frames in the video frame group are video frames shot by the multiple video shooting devices based on the same trigger signal according to the relative time difference of each video frame in the video frame group, until it is determined that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal; wherein, the first video frame is the video frame with the smallest relative time difference in the video frame group, and the second video frame is the next video frame of the first video frame shot by the video shooting device that shot the first video frame.

[0215] As an optional implementation of the embodiment of the present application, the determination unit 1104 is also used to update the reference time difference of each video shooting device according to the relative time difference of each video frame in the synchronous video frame group after determining that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

[0216] As an optional implementation of the embodiment of the present application, the first acquisition unit 1101 is also used to respectively receive the video frames shot by the multiple video shooting devices in response to the initial trigger signal; and calculate the reference time difference of each video shooting device based on the shooting timestamps of the video frames shot by the multiple video shooting devices in response to the initial trigger signal, and the shooting timestamps of the video frames shot by the target video shooting device in response to the trigger signal.

[0217] As an optional implementation of the embodiment of the present application, the determination unit 1104 is also used to, when determining that the video frame group is a synchronous video frame group, preprocess the video frames in the synchronous video frame group based on multiple preprocessing modules to obtain a preprocessed video frame group; and send the preprocessed video frame group to the synchronization device.

[0218] The embodiment of the present application provides a video processing device, wherein the video processing device belongs to a first client device participating in video communication, Fig.12 is a structural diagram of the video processing device, such as Fig.12 As shown, the video processing device 1200 includes:

[0219] A synchronization device used in a video acquisition system, the device comprising:

[0220] An acquiring unit 1201 is used to acquire a video frame group set, wherein the video frame group set includes a plurality of pre-processed video frame groups respectively sent by a plurality of pre-processing devices in the video acquisition system;

[0221] The processing unit 1202 is used to obtain the relative time difference of each preprocessed video frame group according to the reference timestamp of each preprocessed video frame group in the video frame group set and the reference timestamp of the target preprocessed video frame group in the video frame group set; the reference timestamp of any preprocessed video frame group is the reference timestamp of any video frame in the preprocessed video frame group;

[0222] The determination unit 1203 is configured to determine, according to the relative time difference of each pre-processed video frame group, whether each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal.

[0223] As an optional implementation of the embodiment of the present application, the determination unit 1203 is specifically used to determine whether the relative time difference of each preprocessed video frame group in the video frame group set is less than or equal to the second threshold time difference; if the relative time difference of each preprocessed video frame group in the video frame group set is less than or equal to the second threshold time difference, then it is determined that each preprocessed video frame group in the synchronized video group set is a preprocessed video frame group obtained based on the same trigger signal; if the relative time difference of one or more preprocessed video frame groups in the video frame group is greater than the second threshold time difference, then it is determined that each preprocessed video frame group in the synchronized video group set is not a preprocessed video frame group obtained based on the same trigger signal.

[0224] As an optional implementation of the embodiment of the present application, the determination unit 1203 is also used to replace the first preprocessed video frame group in the video frame group with the second preprocessed video frame group when determining whether the respective preprocessed video frame groups in the synchronized video group set are not preprocessed video frame groups obtained based on the same trigger signal, and reacquire the relative time difference of the respective preprocessed video frame groups in the synchronized video group set, and determine whether the respective preprocessed video frame groups in the synchronized video group set are preprocessed video frame groups obtained based on the same trigger signal according to the relative time difference of the respective preprocessed video frame groups, until it is determined that the respective preprocessed video frame groups in the synchronized video group set are preprocessed video frame groups obtained based on the same trigger signal; wherein, the first preprocessed video frame group is the preprocessed video frame group with the smallest relative time difference in the video frame group set, and the second preprocessed video frame group is the next group of preprocessed video frame groups sent by the preprocessing device that sends the first preprocessed video frame group.

[0225] As an optional implementation of the present application, refer to Fig.13 As shown, the video processing device also includes: a calculation unit 1204, which is specifically used to obtain the queue length of each preprocessing device, the queue length of any preprocessing device is the length of the input queue of the preprocessing module in which a single video frame takes the longest time in the preprocessing module of the preprocessing device; based on the queue length of each preprocessing device, a preset frame rate and a model prediction control algorithm, the predicted frame rate of each preprocessing device in the next video frame acquisition cycle is obtained; based on the minimum value of the predicted frame rate of each preprocessing device, the frame rate of each video shooting device for video frame acquisition is controlled.

[0226] As an optional implementation of the embodiment of the present application, the calculation unit 1204 is specifically used for:

[0227] The target frame rate of each preprocessing device in the next video frame acquisition cycle is obtained based on the queue length of each preprocessing device, the preset frame rate, and the model predictive control algorithm and the following formula:

[0228]

[0229] Among them, α i , β i , γ i are preset constants, L t0+it is the queue length of the preprocessing device in the i-th video frame acquisition cycle, R t0+it is the predicted frame rate of the preprocessing device in the i-th video frame acquisition cycle, is the preset frame rate.

[0230] The video processing device provided in this embodiment can execute the video processing method provided in the above method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0231] Based on the same inventive concept, the embodiment of the present application further provides a video processing system; comprising: a plurality of video shooting devices, a plurality of preprocessing devices, a synchronization device and a trigger; the plurality of video shooting devices are respectively connected to one of the preprocessing devices and are all connected to the trigger, the plurality of preprocessing devices are all connected to the synchronization device, and the synchronization device is connected to the trigger;

[0232] The multiple pre-processing devices are respectively used to execute the video processing methods described in claims 1-6 above;

[0233] The synchronization device is used to execute the video processing method described in claims 7-11 above.

[0234] Based on the same inventive concept, an embodiment of the present application also provides an electronic device. Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.14 As shown, the electronic device provided in this embodiment includes: a memory 141 and a processor 142, wherein the memory 141 is used to store a computer program, and the processor 142 is used to execute the video processing method provided in the above embodiment when executing the computer program.

[0235] Based on the same inventive concept, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements the video processing method provided in the above embodiment.

[0236] Based on the same inventive concept, an embodiment of the present application further provides a computer program product. When the computer program product is run on a computer, the computing device implements the video processing method provided in the above embodiment.

[0237] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0238] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0239] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0240] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0241] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein, and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a video frame group, where the video frame group includes video frames respectively captured by a plurality of video capturing devices, and any video frame includes a capturing timestamp of the video frame; Obtaining a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and a reference time difference of each video shooting device, wherein the reference time difference of any video shooting device is a difference between a system time of the video shooting device and a system time of a target video shooting device among the multiple video shooting devices; Obtaining a relative time difference of each video frame according to a reference time stamp of each video frame and a reference time stamp of a target video frame in the video frame group; the target video frame is a video frame captured by the target video capturing device; According to the relative time difference of each video frame, it is determined whether the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal.

2. The method according to claim 1, characterized in that The step of determining, according to the relative time difference of each video frame in the video frame group, whether the video frame group is a synchronous video frame group composed of video frames captured by the multiple video capturing devices based on the same trigger signal comprises: Determining whether a relative time difference between each video frame in the video frame group is less than or equal to a first threshold time difference; If the relative time difference of each video frame in the video frame group is less than or equal to the first threshold time difference, determining that the video frame group is a synchronous video frame group composed of video frames captured by the multiple video capture devices based on the same trigger signal; If the relative time difference of one or more video frames in the video frame group is greater than the first threshold time difference, it is determined that the video frame group is not a synchronous video frame group composed of video frames captured by the multiple video capturing devices based on the same trigger signal.

3. The method according to claim 2, characterized in that The method further comprises: In the case where it is determined that the video frame group is not a synchronous video frame group consisting of video frames shot by the multiple video shooting devices based on the same trigger signal, replacing the first video frame in the video frame group with the second video frame, and reacquiring the relative time difference of each video frame in the video frame group, and determining whether the video frames in the video frame group are video frames shot by the multiple video shooting devices based on the same trigger signal according to the relative time difference of each video frame in the video frame group, until it is determined that the video frame group is a synchronous video frame group consisting of video frames shot by the multiple video shooting devices based on the same trigger signal; The first video frame is a video frame with the smallest relative time difference in the video frame group, and the second video frame is a next video frame of the first video frame captured by a video capture device that captured the first video frame.

4. The method according to claim 2, characterized in that: The method further comprises: After determining that the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal, the reference time difference of each video shooting device is updated according to the relative time difference of each video frame in the synchronous video frame group.

5. The method according to claim 1, characterized in that Before acquiring the reference timestamp of each video frame in the video frame group according to the shooting timestamp of each video frame in the video frame group and the reference time difference of each video shooting device, the method further includes: Receiving respectively the video frames captured by the plurality of video capturing devices in response to the initial trigger signal; The reference time difference of each video capture device is calculated according to the capture timestamps of the video frames captured by the multiple video capture devices in response to the initial trigger signal and the capture timestamps of the video frames captured by the target video capture device in response to the trigger signal.

6. The method according to claims 1-5, characterized in that: The method further comprises: In the case where it is determined that the video frame group is a synchronous video frame group, preprocessing the video frames in the synchronous video frame group based on multiple preprocessing modules to obtain a preprocessed video frame group; The pre-processed video frame group is sent to a synchronization device.

7. A video processing method, characterized in that: The method comprises: Acquire a video frame group set, wherein the video frame group set includes a plurality of pre-processed video frame groups respectively sent by a plurality of pre-processing devices in the video acquisition system; According to the reference timestamps of the preprocessed video frame groups in the video frame group set and the reference timestamps of the target preprocessed video frame group in the video frame group set, the relative time difference of the preprocessed video frame groups is obtained; the reference timestamp of any preprocessed video frame group is the reference timestamp of any video frame in the preprocessed video frame group; According to the relative time difference of each pre-processed video frame group, it is determined whether each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal.

8. The method according to claim 7, characterized in that The step of determining, according to the relative time difference of each pre-processed video frame group, whether each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal comprises: Determining whether a relative time difference of each pre-processed video frame group in the video frame group set is less than or equal to a second threshold time difference; If the relative time difference of each pre-processed video frame group in the video frame group set is less than or equal to the second threshold time difference, determining that each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal; If the relative time difference of one or more preprocessed video frame groups in the video frame group is greater than the second threshold time difference, it is determined that the preprocessed video frame groups in the synchronous video group set are not preprocessed video frame groups obtained based on the same trigger signal.

9. The method according to claim 8, characterized in that The method further comprises: In the case of determining whether each preprocessed video frame group in the synchronous video group set is not a preprocessed video frame group obtained based on the same trigger signal, replacing the first preprocessed video frame group in the video frame group with the second preprocessed video frame group, and reacquiring the relative time difference of each preprocessed video frame group in the synchronous video group set, and determining whether each preprocessed video frame group in the synchronous video group set is a preprocessed video frame group obtained based on the same trigger signal according to the relative time difference of each preprocessed video frame group, until it is determined that each preprocessed video frame group in the synchronous video group set is a preprocessed video frame group obtained based on the same trigger signal; The first preprocessed video frame group is the preprocessed video frame group with the smallest relative time difference in the video frame group set, and the second preprocessed video frame group is the next preprocessed video frame group sent by the preprocessing device that sends the first preprocessed video frame group.

10. The method according to any one of claims 7 to 9, characterized in that: The method further comprises: Obtaining the queue length of each preprocessing device, the queue length of any preprocessing device being the length of the input queue of the preprocessing module that takes the longest time for a single video frame in the preprocessing module of the preprocessing device; Based on the queue length, preset frame rate and model prediction control algorithm of each pre-processing device, the predicted frame rate of each pre-processing device in the next video frame acquisition cycle is obtained; The frame rate of video frame acquisition performed by each video shooting device is controlled based on the minimum value among the predicted frame rates of each pre-processing device.

11. The method according to claim 10, characterized in that The method of obtaining a target frame rate of each preprocessing device in the next video frame acquisition cycle based on the queue length of each preprocessing device, a preset frame rate, and a model predictive control algorithm includes: The target frame rate of each preprocessing device in the next video frame acquisition cycle is obtained based on the queue length of each preprocessing device, the preset frame rate, and the model predictive control algorithm and the following formula: Among them, k, α i , β i , γ i are preset constants, L t0+it is the queue length of the preprocessing device in the i-th video frame acquisition cycle, R t0+it is the predicted frame rate of the preprocessing device in the i-th video frame acquisition cycle, is the preset frame rate.

12. A video processing device, characterized in that: The device comprises: A first acquisition unit, configured to acquire a video frame group, wherein the video frame group includes video frames respectively captured by a plurality of video capture devices, and any video frame includes a capture timestamp of the video frame; A second acquisition unit is used to acquire a reference timestamp of each video frame according to the shooting timestamp of each video frame in the video frame group and the reference time difference of each video shooting device, wherein the reference time difference of any video shooting device is the difference between the system time of the video shooting device and the system time of a target video shooting device among the multiple video shooting devices; A third acquisition unit is used to acquire a relative time difference of each video frame according to a reference time stamp of each video frame and a reference time stamp of a target video frame in the video frame group; the target video frame is a video frame captured by the target video capture device; The determination unit is used to determine whether the video frame group is a synchronous video frame group composed of video frames shot by the multiple video shooting devices based on the same trigger signal according to the relative time difference of each video frame.

13. A video processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a video frame group set, wherein the video frame group set includes a plurality of pre-processed video frame groups respectively sent by a plurality of pre-processing devices in the video acquisition system; A processing unit, configured to obtain a relative time difference between each preprocessed video frame group according to a reference time stamp of each preprocessed video frame group in the video frame group set and a reference time stamp of a target preprocessed video frame group in the video frame group set; the reference time stamp of any preprocessed video frame group is a reference time stamp of any video frame in the preprocessed video frame group; The determination unit is used to determine whether each pre-processed video frame group in the synchronous video group set is a pre-processed video frame group obtained based on the same trigger signal according to the relative time difference of each pre-processed video frame group.

14. A video processing system, characterized in that: include: A plurality of video capture devices, a plurality of preprocessing devices, a synchronization device and a trigger; the plurality of video capture devices are respectively connected to one of the preprocessing devices and are all connected to the trigger, the plurality of preprocessing devices are all connected to the synchronization device, and the synchronization device is connected to the trigger; The multiple pre-processing devices are respectively used to execute the video processing methods described in claims 1-6 above; The synchronization device is used to execute the video processing method described in claims 7-11 above.

15. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program and the processor is used to enable the electronic device to implement the video processing method according to any one of claims 1 to 11 when executing the computer program.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the video processing method according to any one of claims 1 to 11.

Images