Camera array video file synchronization frame search method and video shooting device

Abstract

The present invention relates to the field of video synchronization technology, and more particularly to a synchronization frame search method for camera array video files and a video capture device. First, the camera array is controlled to capture video, and a flash is controlled to flash during the capture process, leaving a flash mark frame in each video captured by the camera array. The camera array is controlled to terminate video capture, and the flash mark frame in each video is determined based on the brightness change of the Y channel of the video image. The synchronization frame in each video is determined using the flash mark frame in each video. Finally, the captured videos are synchronized based on the synchronization frame. The present invention effectively combines the flash mark of the flash with image processing, avoiding the low search efficiency of video synchronization frames caused by traditional manual brightness recognition, significantly improving the search efficiency and accuracy of video synchronization frames, and obtaining high-quality synchronized videos.

Description

Technical Field

[0001] The present invention belongs to the technical field of video synchronization, and in particular relates to a synchronous frame search method for camera array video files and a video shooting device. Background Art

[0002] Traditional video is typically recorded using a single electronic device, such as a mobile phone, digital camera, or camcorder. Once the recording function is activated, the camera begins continuous exposure at a certain frequency. Embedded software then encodes the video and stores it as a video file. While traditional video can provide richer dynamic information than images in terms of time, it only provides a small, two-dimensional image at any given moment. To meet the needs of 360°, all-around, dynamic observation of a target person, animal, or object, holographic technology has advanced rapidly in recent years. Holographic data is derived from volumetric photography. Operators pre-arrange hundreds of high-definition cameras (or camcorders) to capture the target from multiple angles, producing images or videos containing depth information. These raw images or videos are then converted into volumetric videos through a series of post-processing methods, ultimately achieving a 360° stereoscopic image of the target. This technology is currently widely used in industries such as entertainment, archaeology, and sports. It also has broad application prospects in the preservation of intangible cultural heritage.

[0003] Volumetric photography requires hundreds of high-definition cameras to record simultaneously, and the data used in holographic technology places high demands on the synchronization of these videos. Many digital products currently on the market with excellent imaging performance lack video synchronization triggering capabilities. Even if these high-definition cameras support external shutter release triggering for recording, the first-frame exposure delay between different cameras can still vary by tens of milliseconds. Furthermore, the first-frame exposure delay of each camera is not fixed, making it impossible to ensure strict alignment of the camera array's first-frame exposure moment through a fixed pattern. Without effective frame alignment of videos recorded by different cameras, holographic video production will fail or significantly degrade in quality.

[0004] The first exposure times of different videos can differ by as much as tens of milliseconds (0-3 frames). Searching for alignment frames frame by frame for each video is a tremendous amount of work for a human operator. Furthermore, in real-world shooting, the differences between adjacent frames within a video are very small, while the differences between aligned frames across different videos can be quite large. Furthermore, in some scenes and shooting angles, the flash marker frame is not readily apparent, making it easy to miss it during the search, ultimately wasting time. Furthermore, the operator can easily become fatigued by searching for the flash marker frame across more than a hundred videos, which can also lead to search errors. In some shooting scenarios, it's impossible to use a shutter release cable to start recording. In these cases, a computer programming interface can be used to sequentially activate the recording function of each camera. The execution time of the computer interface function for a single camera ranges from 5 to 40 milliseconds, and fully initiating recording for all cameras takes approximately 2-3 seconds. This means that by the time the first camera starts recording and has already exposed dozens or even hundreds of frames, the last camera has just finished exposing its first frame. Therefore, existing methods that manually search for the first aligned frame across all videos are not only time-consuming but also have a high error rate. Summary of the Invention

[0005] In view of this, the present invention aims to provide a synchronous frame search method for camera array video files and a video shooting device, which effectively combines the flash mark of the flash and image processing, greatly improving the search efficiency and accuracy of the video synchronization frame, avoiding the problem of low search efficiency of the video synchronization frame caused by traditional manual brightness recognition, obtaining high-quality synchronous video, and being more conducive to multi-angle shooting of a single object, thereby improving the quality of subsequent three-dimensional imaging.

[0006] To achieve the above object, the technical solution created by the present invention is implemented as follows:

[0007] A method for synchronous frame search of a camera array video file comprises the following steps:

[0008] S1: Control the camera array to shoot video, and control a flash to flash during the shooting process, leaving a flash mark frame in each video shot by the camera array;

[0009] S2: Control the camera array to end video shooting, determine the flash mark frame in each video based on the brightness change of the Y channel of the video image, and determine the synchronization frame in each video through the flash mark frame in each video.

[0010] Furthermore, step S2 includes the following steps:

[0011] S21: Decode each video to obtain the total number of frames N of each video;

[0012] S22: Take the i-th frame video image and the i+1-th frame video image in each video, ;

[0013] S23: Calculate and compare the Y channel brightness of the i-th video frame and the (i+1)-th video frame. If the Y channel brightness change between the two video frames meets the preset brightness change, proceed to step S24; otherwise, proceed to step S25.

[0014] S24: The video image with the larger Y channel brightness among the two video frames is the flash mark frame, and the image frame after the flash mark frame is the synchronization frame, and the search for the synchronization frame ends;

[0015] S25: Let i=i+1, and repeat steps S22 and S23 until all frames in each video are traversed.

[0016] Furthermore, in step S23:

[0017] Convert the i-th frame video image and the i+1-th frame video image into YUV images, extract the Y channel of the two frames of video images, and obtain the corresponding Y channel brightness image;

[0018] Subtract the corresponding pixels of the two Y channel brightness images to obtain the Y channel brightness difference image;

[0019] The preset brightness change is: an average value of W consecutive rows of the Y channel brightness difference image is greater than a preset brightness threshold, wherein the value of W is less than the number of rows of the Y channel brightness difference image.

[0020] A video shooting device, comprising:

[0021] a camera array, including multiple cameras for video capture;

[0022] A flash that flashes during the shooting process, so that a flash mark frame is left in each video captured by the camera array;

[0023] a trigger device connected to the flash and camera array;

[0024] The control core triggers the flash and camera array through the trigger device, and receives the video captured by the camera array; it synchronizes the video based on the flash marker frame.

[0025] Furthermore, in the control core, based on the brightness change of the Y channel of two adjacent frames of video images in the captured video, the flash mark frame in each video is determined, and the synchronization frame in each video is determined through the flash mark frame in each video; the captured video is synchronized based on the synchronization frame.

[0026] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0027] The present invention creates a method for searching synchronous frames of camera array video files and a video shooting device, adds a flash mark, searches for flash mark frames using an image processing method, determines the synchronous frames in the video obtained by the camera array, and then performs video synchronization based on the synchronous frames, thereby avoiding recognition errors caused by human fatigue and visual insensitivity, and greatly improving the search efficiency and accuracy of video synchronous frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute part of the present invention, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0029] Figure 1 A schematic flow chart of a method for synchronous frame search of a camera array video file according to an embodiment of the present invention;

[0030] Figure 2 This is a flow chart of step S2 according to an embodiment of the present invention;

[0031] Figure 3 Schematic diagram of the process of step S23 described in the embodiment of the present invention

[0032] Figure 4 A schematic structural diagram of a video shooting device according to an embodiment of the present invention.

[0033] Description of reference numerals:

[0034] 1. Camera array; 2. Flash; 3. Trigger device; 4. Control core; 5. Camera. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0036] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other.

[0037] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, features defined as "first", "second" and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0038] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "installed," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to internal connections between two components. Those skilled in the art can understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0039] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0040] like Figures 1 to 3 As shown, the synchronous frame search method of the camera array video file according to the embodiment of the present invention includes:

[0041] S1: Control the camera array to shoot video, and control a flash to flash during the shooting process, leaving a flash mark frame in each video shot by the camera array.

[0042] Since the flash start time and flash duration of each camera in the camera array cannot be guaranteed to be strictly consistent, a flash delay will occur during the shooting process, thereby affecting video synchronization. Therefore, it is necessary to control all cameras in the camera array to receive only the signal of one flash. Therefore, in an embodiment of the present invention, it is preferred to set a flash independent of the camera array so that the camera array can ensure that the flash is received at the same time and leave a flash mark in each video. In this way, the flash start time and flash duration of each camera are guaranteed to be consistent, thereby avoiding flash delays during the shooting process and thus avoiding affecting video synchronization. There are two ways to shoot videos with existing camera arrays, using a shutter cable or a computer programming interface. The flash delay time of the flash can be calculated through experiments based on different startup methods. The flash delay time calculation adopts the existing calculation method, which will not be elaborated in the present invention.

[0043] S2: Control the camera array to end video capture, determine the flash mark frame in each video based on the brightness change of the Y channel of the video image, and determine the synchronization frame in each video through the flash mark frame in each video. Step S2 includes the following steps:

[0044] S21: Decode each video to obtain information such as video resolution and total number of frames N. In some embodiments, the video captured by the camera array is an RGB video, that is, each frame image in the video is an RGB image.

[0045] S22: Take the i-th frame video image and the i+1-th frame video image in each video, .

[0046] S23: Calculate and compare the Y channel brightness of the i-th and i+1-th video frames. If the Y channel brightness change between the two frames meets a preset brightness change, proceed to step S24; otherwise, proceed to step S25. In step S23, the i-th and i+1-th video frames are converted to YUV images, and the Y channels of the two frames are extracted to obtain corresponding Y channel brightness images. The two Y channel brightness images are then pixel-wise subtracted to obtain a Y channel brightness difference image. Because the cameras in the camera array use a rolling shutter exposure method when taking or recording photos or videos, frames marked by the flash appear as consecutive rows with significantly higher brightness than the previous frame. For this reason, the preset brightness change is: the average value of W consecutive rows of the Y channel brightness difference image is greater than a preset brightness threshold, where W is less than the number of rows in the Y channel brightness difference image. In some embodiments, W is set to 50, and the preset brightness threshold is 30. That is, if the average value of 50 consecutive rows of the Y channel brightness difference image is greater than 30, then proceed to step S24; otherwise, proceed to step S25.

[0047] S24: The video image with the larger Y channel brightness among the two video frames is the flash mark frame, and the image frame after the flash mark frame is the synchronization frame;

[0048] S25: Let i=i+1, and repeat steps S22 and S23 until all the frame video images in each video are traversed.

[0049] The present invention uses an image processing method based on Y channel brightness changes to automatically identify flash mark frames by a computer, thereby significantly shortening the search time for synchronization frames and significantly improving the search efficiency and accuracy of video synchronization frames, obtaining high-quality synchronized videos, and being more conducive to multi-angle shooting of a single object, thereby improving the quality of subsequent three-dimensional imaging. The problem of low search efficiency of video synchronization frames caused by traditional manual brightness recognition is avoided, and recognition errors caused by human fatigue and visual insensitivity are avoided, and the search time is significantly shortened.

[0050] A video shooting device, according to an embodiment of the present invention, provides a method for searching synchronized frames of a camera array video file, comprising a camera array 1, a flash 2, a trigger device 3, and a control core 4. In particular:

[0051] The camera array 1 includes multiple digital cameras 5 (or digital video cameras). All cameras 5 in the camera array 1 should preferably be of the same brand and model. If multiple brands or models of cameras 5 are required, all cameras 5 should be able to record at the same frame rate. The control interface of the camera array 1 can be either a physical interface or a virtual interface.

[0052] Flash 2 is an independent flash that flashes during the shooting process. Its function is equivalent to marking in the video. Therefore, the placement of flash 2 should allow its light to illuminate all cameras 5, ensuring that the camera array 1 can receive the flash at the same time and leave a flash mark frame in each video captured.

[0053] The trigger device 3 is connected to the flash unit 2 and the camera array 1. It can send commands to all cameras 5 in the camera array 1 to start and stop recording, as well as delay control of the flash unit 2 flashing, via the camera control interface. The trigger device 3 can be a hardware device or a programming interface embedded in a computer. The trigger device 3 for the camera array 1 and the flash unit 2 can be the same or different components or computer software.

[0054] Control core 4 has two main functions: controlling the operation of trigger device 3, and processing video and searching for synchronization frames. Once camera array 1 completes video capture, all videos are imported into control core 4 for processing. This processing includes: determining the flash marker frame in each video based on the Y channel brightness change between two adjacent frames; determining the synchronization frame in each video based on the flash marker frame; and synchronizing the captured videos based on the synchronization frame.

[0055] In this embodiment of the present invention, control core 4 uses an Intel(R) Core(TM) i9-11900K @ 3.50GHz processor to search for flash marker frames in each 4K video captured by camera 5. Finding the flash marker frame for each video takes an average of less than 10 seconds (the actual search time depends on the application scenario and the flash delay time). If this CPU is used for eight parallel threads, the flash marker frame search for eight videos can be completed in 10 seconds. For a single shot of over 100 videos captured by camera array 1, the synchronized frame search for all videos takes 3-5 minutes.

[0056] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved. This is not limited herein.

[0057] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A synchronous frame search method for a camera array video file, characterized in that: For volumetric imaging, the following steps are involved: S1: Controlling a camera array to shoot a video, and controlling a flash to flash during the shooting process, leaving a flash mark frame in each video shot by the camera array; S2: Control the camera array to end video capture, determine a flash mark frame in each video based on a Y channel brightness change of the video image, and determine a synchronization frame in each video using the flash mark frame in each video; Step S2 includes the following steps: S21: Decode each video to obtain the total number of frames N of each video; S22: Take the i-th frame video image and the i+1-th frame video image in each video, ; S23: Calculate and compare the Y channel brightness of the i-th video frame and the (i+1)-th video frame. If the Y channel brightness change between the two video frames meets the preset brightness change, proceed to step S24; otherwise, proceed to step S25. In step S23: the i-th frame video image and the (i+1)-th frame video image are converted into YUV images, the Y channels of the two frames of video images are extracted, and corresponding Y channel brightness images are obtained; the two Y channel brightness images are subtracted from corresponding pixels to obtain a Y channel brightness difference image; the preset brightness change is: the average value of W consecutive rows of the Y channel brightness difference image is greater than a preset brightness threshold, where the value of W is less than the number of rows of the Y channel brightness difference image; S24: The video image with the larger Y channel brightness among the two video frames is the flash mark frame, and the image frame after the flash mark frame is the synchronization frame. At this time, the search for the synchronization frame ends; S25: Let i=i+1, and repeat steps S22 and S23 until all frames in each video are traversed.

2. A video shooting device, characterized in that: The method for synchronous frame search of a camera array video file according to claim 1 comprises: a camera array, including multiple cameras for video capture; a flash, which flashes during the shooting process, so that a flash mark frame is left in each video captured by the camera array; a trigger device connected to the flashlight and the camera array; A control core triggers the flashlight and the camera array through the trigger device and receives the video captured by the camera array; synchronizes the video based on the flash mark frame; in the control core, determines the flash mark frame in each video based on the Y channel brightness change of two adjacent frames of video images in the captured video, determines the synchronization frame in each video through the flash mark frame in each video; and synchronizes the captured video based on the synchronization frame.

Images