Synchronous frame searching method of camera array video file and video shooting device

Abstract

The invention relates to the technical field of video synchronization, in particular to a synchronous frame searching method for a camera array video file and a video shooting device. The method comprises the following steps: firstly, controlling a camera array to shoot videos, controlling a flash lamp to flash in the shooting process, and leaving a flash mark frame in each video shot by the camera array; controlling the camera array to end video shooting, determining a flash mark frame in each video based on the Y channel brightness change of the video image, and determining a synchronization frame in each video through the flash mark frame in each video; and finally, synchronizing the shot video based on the synchronization frame. According to the method, flash marking of the flash lamp and image processing are effectively combined, the problem of low search efficiency of the video synchronization frame caused by traditional manual brightness identification is avoided, the search efficiency and accuracy of the video synchronization frame are greatly improved, and a high-quality synchronization video is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of video synchronization, and particularly relates to a method for searching synchronous frames of a video file of a camera array and a video shooting device. Background Art

[0002] Traditional videos are generally recorded using a single electronic device, such as a mobile phone, a digital camera, or a digital video camera. After the video recording function is started, the photographing component starts continuous exposure at a certain frequency, and then video encoding is performed through embedded software and stored as a video file. Although traditional videos can provide richer dynamic information than images in the time dimension, they can only provide a small two-dimensional image at any moment. To meet people's 360° all-round and non-blind dynamic observation of target people, animals, or objects, the development of holographic technology has advanced by leaps and bounds in recent years. The data of holographic technology comes from volumetric photography. Operators arrange hundreds of high-definition cameras (or video cameras) in advance to photograph the target from multiple angles, and pictures or videos containing depth information can be taken. These original pictures or videos are made into volumetric videos through a series of post-processing methods, and finally 360° stereoscopic imaging of the target is achieved. This technology is currently widely used in industries such as entertainment, archaeology, and sports. In addition, this technology also has broad application prospects in the inheritance of intangible cultural heritage.

[0003] Volumetric photography requires hundreds of high-definition cameras to record videos simultaneously, and the data used in holographic technology has high requirements for the synchronization of these videos. At present, many digital products with good imaging effects on the market do not have a video synchronous triggering function. Even if this type of high-definition camera supports off-camera cable triggering for video recording, the maximum difference in the first-frame exposure delay of different cameras is still dozens of milliseconds, and the first-frame exposure delay of each camera is not fixed, and it is impossible to ensure strict alignment of the first-frame exposure moments of the camera array through a fixed rule. If effective frame alignment processing is not performed on the videos recorded by different cameras, it will lead to the failure of holographic video production or a significant decline in the effect.

[0004] The maximum difference in the exposure moments of the first frames of different videos is dozens of milliseconds (0 - 3 frames). If an operator searches for aligned frames frame by frame for each video, it will undoubtedly be a huge workload. Moreover, in actual shooting, the difference between adjacent frames of each video is very small, while the difference between aligned frames of different videos is very large. And in some scenarios and shooting angles, the flash - marked frames are not obvious and are easily missed during the search process, ultimately resulting in a waste of time. Also, since the operator searches for flash - marked frames in more than a hundred videos for a long time, it is extremely easy to cause fatigue and prone to search errors. In some shooting occasions, the shutter cable cannot be used to start video recording, and at this time, the computer programming interface can only be used to start the video recording function of each camera in sequence. The execution time of the computer interface function for a single camera ranges from 5 to 40 ms, and it takes about 2 - 3 seconds for all cameras to fully start video recording. That is to say, after the first camera starts video recording and has exposed dozens or more than a hundred frames, the last camera has just completed the exposure of the first - frame image. In summary, in the existing methods, searching for the first aligned frame of all videos manually not only takes a long time but also has a very high error rate. Summary of the Invention

[0005] In view of this, the present invention aims to provide a method for searching synchronous frames of camera - array video files and a video - shooting device, which effectively combines the flash marks of the flash and image processing, greatly improves the search efficiency and accuracy of video synchronous frames, avoids the problem of low search efficiency of video synchronous frames caused by traditional manual brightness recognition, obtains high - quality synchronous videos, is more conducive to multi - angle shooting of a single object, and thus improves the quality of subsequent three - dimensional imaging.

[0006] To achieve the above - mentioned purpose, the technical solution of the present invention is realized as follows: A method for searching synchronous frames of camera - array video files includes the following steps: S1: Control the camera array to perform video shooting, and control a flash to flash during the shooting process, leaving flash - marked frames in each video captured by the camera array; S2: Control the camera array to end video shooting, determine the flash - marked frames in each video based on the brightness change of the Y channel of the video image, and determine the synchronous frames in each video through the flash - marked frames in each video.

[0007] Further, step S2 includes the following steps: S21: Decode each video to obtain the total number of frames N of each video; S22: Take the i - th frame video image and the (i + 1) - th frame video image in each video, ; S23: Calculate and compare the Y-channel brightness of the i-th video image and the (i + 1)-th video image. If the Y-channel brightness change between the two video images conforms to the preset brightness change, proceed to step S24; otherwise, execute step S25; S24: The video image with the larger Y-channel brightness among the two video images is the flash marker frame, and the frame immediately following the flash marker frame is the synchronization frame, thus ending the search for the synchronization frame; S25: Let i = i + 1, and repeat steps S22 and S23 until all frames in each video are traversed.

[0008] Further, in step S23: Convert the i-th video image and the (i + 1)-th video image into YUV images, extract the Y channels of the two video images, and correspondingly obtain Y-channel brightness images; Perform pixel-by-pixel subtraction on the two Y-channel brightness images to obtain a Y-channel brightness difference image; The preset brightness change is that the average value of consecutive W rows of the Y-channel brightness difference image is greater than the preset brightness threshold, where the value of W is less than the number of rows of the Y-channel brightness difference image.

[0009] A video shooting device includes: A camera array including multiple cameras for video shooting; A flash that flashes during shooting, leaving a flash marker frame in each video captured by the camera array; A triggering device connected to the flash and the camera array; A control core that triggers the flash and the camera array through the triggering device, and receives the videos captured by the camera array; synchronizes the videos based on the flash marker frames.

[0010] Further, in the control core, based on the Y-channel brightness change between adjacent two video images in the captured videos, determine the flash marker frames in each video, and determine the synchronization frames in each video through the flash marker frames in each video; synchronize the captured videos based on the synchronization frames.

[0011] Compared with the prior art, the present invention can achieve the following beneficial effects: In the synchronization frame search method for camera array video files and the video shooting device of the present invention, a flash marker is added. The flash marker frame is searched by using an image processing method to determine the synchronization frames in the videos obtained by the camera array, and then the videos are synchronized based on the synchronization frames, avoiding recognition errors caused by human fatigue and visual insensitivity, and greatly improving the search efficiency and accuracy of video synchronization frames. Description of the Drawings

[0012] The accompanying drawings, which form a part of the present invention, are used to provide a further understanding of the present invention. The schematic embodiments and descriptions thereof of the present invention are used to explain the present invention and do not constitute an improper limitation to the present invention. In the drawings: Figure 1 It is a schematic flowchart of the synchronous frame search method for the camera array video file according to the embodiment of the present invention; Figure 2 It is a schematic flowchart of step S2 according to the embodiment of the present invention; Figure 3 It is a schematic flowchart of step S23 according to the embodiment of the present invention Figure 4 It is a schematic structural diagram of the video shooting device according to the embodiment of the present invention.

[0013] Explanation of reference numerals: 1. Camera array; 2. Flashlight; 3. Trigger device; 4. Control core; 5. Camera. Detailed implementation manners

[0014] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation to the present invention.

[0015] It should be noted that, without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0016] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation to the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined with "first", "second", etc. may explicitly or implicitly include one or more of such features. In the description of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0017] In the description of the invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention can be understood according to specific circumstances.

[0018] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0019] like Figures 1 to 3 As shown, the synchronous frame search method of the camera array video file described in the embodiment of the present invention includes: S1: Control the camera array to shoot a video, and control a flash to flash during the shooting process, leaving a flash mark frame in each video shot by the camera array.

[0020] Since the flash start time and flash duration of the flash of each camera in the camera array cannot be guaranteed to be strictly consistent, a flash delay will occur during the shooting process, thereby affecting video synchronization. Therefore, it is necessary to control all cameras in the camera array to receive only the signal of one flash. Therefore, in an embodiment of the present invention, it is preferred to set a flash independent of the camera array so that the camera array can ensure that the flash is received at the same time and leave a flash mark in each video. In this way, the flash start time and flash duration of each camera are guaranteed to be consistent, thereby avoiding flash delays during the shooting process, thereby avoiding affecting video synchronization. There are two ways to shoot videos with existing camera arrays, using a shutter cable or a computer programming interface. The flash delay time of the flash can be calculated experimentally according to different startup methods. The flash delay time calculation adopts the existing calculation method, which will not be elaborated in the present invention.

[0021] S2: Control the camera array to end video shooting, determine the flash mark frame in each video based on the Y channel brightness change of the video image, and determine the synchronization frame in each video through the flash mark frame in each video. Step S2 includes the following steps: S21: Decode each video to obtain information such as video resolution and total number of frames N. In some embodiments, the video captured by the camera array is an RGB video, that is, each frame image in the video is an RGB image.

[0022] S22: Take the i-th frame video image and the i+1-th frame video image in each video, .

[0023] S23: Calculate and compare the Y-channel brightness of the i-th frame of video image and the (i + 1)-th frame of video image. If the Y-channel brightness change between the two frames of video images conforms to the preset brightness change, proceed to step S24; otherwise, execute step S25. In step S23: Convert the i-th frame of video image and the (i + 1)-th frame of video image into YUV images, extract the Y channels of the two frames of video images, and correspondingly obtain Y-channel brightness images. Perform subtraction on the corresponding pixels of the two Y-channel brightness images to obtain a Y-channel brightness difference image. Since the cameras in the camera array use the rolling shutter exposure method when taking pictures or videos, the frames marked by the flash are manifested as the brightness of a continuous number of rows being significantly higher than the previous frame. For the above reasons, the preset brightness change is: the average value of continuous W rows of the Y-channel brightness difference image is greater than the preset brightness threshold, where the value of W is less than the number of rows of the Y-channel brightness difference image. In some embodiments, W is set to 50 and the preset brightness threshold is 30, that is, if there are 50 consecutive rows in the Y-channel brightness difference image with an average value greater than 30, execute step S24; otherwise, execute step S25.

[0024] S24: The video image with the larger Y-channel brightness among the two frames of video images is the flash-marked frame, and the frame immediately following the flash-marked frame is the synchronization frame; S25: Let i = i + 1, and repeat step S22 and step S23 until all the frame video images in each video are traversed.

[0025] Through the method of image processing based on the Y-channel brightness change, the present invention automatically identifies the flash-marked frames by a computer, greatly shortening the search time for synchronization frames, while significantly improving the search efficiency and accuracy of video synchronization frames, obtaining high-quality synchronized videos, being more conducive to multi-angle shooting of a single object, thereby improving the quality of subsequent three-dimensional imaging, avoiding the problem of low search efficiency of video synchronization frames caused by traditional manual brightness recognition, avoiding recognition errors caused by human fatigue and visual insensitivity, and greatly shortening the search time.

[0026] A video shooting device, according to the synchronization frame search method for a camera array video file provided by an embodiment of the present invention, includes a camera array 1, a flash 2, a trigger device 3, and a control core 4. Among them: The camera array 1 includes a plurality of digital cameras 5 (or digital video cameras). All the cameras 5 in the camera array 1 preferably use cameras 5 of the same brand and model. If there is a real need to use multiple brands or models of cameras 5, it should also be ensured that all the cameras 5 can record videos at the same frame rate. The control interface of the camera array 1 can be an actual interface or a virtual interface.

[0027] The flash 2 is an independent flash that flashes during the shooting process. Its function is equivalent to making a mark in the video. Therefore, the flash 2 should be placed so that its light can shine into the entire camera 5, ensuring that the camera array 1 can receive the flash simultaneously and leave a flash-marked frame in each captured video.

[0028] The triggering device 3 is connected to the flash 2 and the camera array 1. It can send instructions to start and stop video recording to all the cameras 5 in the camera array 1 through the camera control interface, and delay and control the flash 2 to flash. The triggering device 3 can be a hardware device or a programming interface embedded in a computer. The triggering devices 3 for the camera array 1 and the flash 2 can be the same or different components or computer software.

[0029] The control core 4 has two main functions: controlling the operation of the triggering device 3 and video processing and synchronization frame search. After the video shooting of the camera array 1 is completed, all the videos can be imported into the control core 4 and processed by the control core 4. The processing includes: determining the flash-marked frames in each video based on the Y-channel brightness change between two adjacent video images in the captured video, and determining the synchronization frames in each video through the flash-marked frames in each video; synchronizing the captured videos based on the synchronization frames.

[0030] In the embodiment of the present invention, the control core 4 uses an Intel(R) Core(TM) i9-11900K @ 3.50GHz processor to search for the flash-marked frames of the 4K videos captured by each camera 5. The average time taken to find the flash-marked frames in each video is less than 10 seconds (the actual search time is related to the application scenario and the flash delay time of the flash). If 8-thread parallel processing is used with this CPU, the flash-marked frame search for 8 videos can be completed within 10 seconds. For the more than one hundred videos captured by the camera array 1 used in the embodiment of the present invention, it takes 3 to 5 minutes to complete the synchronization frame search for all the videos.

[0031] It should be understood that various forms of the processes shown above can be used, reordering, adding, or deleting steps. For example, the steps described in the disclosure of the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present invention can be achieved. There is no limitation herein.

[0032] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for synchronizing frame search of a camera array video file, characterized in that Including the following steps: S1: Control the camera array to perform video shooting, and control a flash to flash during the shooting process, leaving flash marker frames in each video captured by the camera array; S2: Control the camera array to end video shooting, determine the flash marker frames in each video based on the Y-channel brightness change of the video images, and determine the synchronization frames in each video through the flash marker frames in each video.

2. The method for synchronizing frame search of a camera array video file according to claim 1, wherein, Step S2 includes the following steps: S21: Decode each video to obtain the total number of frames N of each video; S22: Obtain the i-th frame video image and the (i + 1)-th frame video image in each video, ; S23: Calculate and compare the Y-channel brightness of the i-th video image and the (i + 1)-th video image. If the Y-channel brightness change between the two video images meets the preset brightness change, perform step S24, otherwise execute step S25; S24: The video image with the larger Y-channel brightness among the two video images is the flash marker frame, and the frame image after the flash marker frame is the synchronization frame. At this time, the search for the synchronization frame ends; S25: Let i = i + 1, and repeat steps S22 and S23 until all frames in each video are traversed.

3. The synchronous frame search method for the camera array video file according to claim 2, wherein In step S23: Convert the i-th video image and the (i + 1)-th video image into YUV images, extract the Y channels of the two video images, and correspondingly obtain Y-channel brightness images; Perform pixel-by-pixel subtraction on the two Y-channel brightness images to obtain a Y-channel brightness difference image; The preset brightness change is that the average value of consecutive W rows of the Y-channel brightness difference image is greater than the preset brightness threshold, where the value of W is less than the number of rows of the Y-channel brightness difference image.

4. A video shooting device, characterized in that, Including: A camera array, including multiple cameras for video shooting; A flash, which flashes during the shooting process, so that flash marker frames are left in each video captured by the camera array; A triggering device, connected to the flash and the camera array; A control core, which triggers the flash and the camera array through the triggering device, and receives the videos captured by the camera array; Perform synchronization processing on the videos based on the flash marker frames.

5. The video shooting device according to claim 4, wherein In the control core, determine the flash marker frames in each video based on the Y-channel brightness change between adjacent two video images in the captured videos, determine the synchronization frames in each video through the flash marker frames in each video; synchronize the captured videos based on the synchronization frames.

Images