A video synchronization method and system

Abstract

A video synchronization method and system, belonging to the technical field of video synchronization, comprises the following steps: acquiring a logical frame serial number, calculating a corresponding screen frame serial number through a mapping relationship; converting the screen frame serial number into an encoded color block and displaying it to a video picture; acquiring the video picture, recording a shooting timestamp, and decoding the current screen frame serial number from the video picture; acquiring an average frame interval, a previous logical frame serial number, and a previous timestamp, and calculating an expected logical frame serial number; calculating a corresponding candidate logical frame serial number through the mapping relationship according to the current screen frame serial number; calculating the difference between the expected logical frame serial number and the candidate logical frame serial number; if the difference is greater than a threshold value, outputting the expected logical frame serial number; and if the difference is less than the threshold value, outputting the candidate logical frame serial number. The application can automatically detect and correct abnormal conditions such as frame loss and repeated frames by comparing the difference between the expected logical frame serial number and the candidate logical frame serial number.

Description

Technical Field

[0001] This invention belongs to the field of video synchronization technology, and particularly relates to a video synchronization method and system. Background Technology

[0002] In fields such as video processing, virtual production, display device testing, and camera evaluation, it is often necessary to synchronize displayed content with camera-captured content to ensure time consistency. Specifically, scenarios such as real-time compositing in video processing, virtual-real fusion and overlay in virtual production, image quality and response speed calibration in display device testing, and frame rate consistency and color reproduction verification in camera evaluation all require millisecond-level or even microsecond-level time alignment between the content output from the display and the content captured by the camera. This ensures strict consistency in the time dimension, thereby avoiding problems such as image misalignment, data distortion, and invalid test results caused by synchronization deviations.

[0003] Currently, the mainstream synchronization technologies in the industry are mainly divided into three categories: timecode-based synchronization technology, digital watermarking-based synchronization technology, and hardware-based synchronization technology. Details are as follows:

[0004] I. Timecode-based synchronization technology

[0005] Patent CN113260040B discloses a timecode synchronization method applied to a network system comprising multiple nodes. The method includes: determining a target master node and a target slave node among the multiple nodes; after determining the target master node and target slave node, the target master node periodically sends data packets to the target slave node, the data packets including a first timecode and a serial number; the target slave node compensates for the first timecode based on the serial number to obtain a second timecode, and synchronizes the second timecode; the first timecodes in the data packets sent by the target master node are the same, but the serial numbers are different. By having the master node periodically send timecode data packets with serial numbers, and the slave node compensate for delay time based on the serial numbers, this method solves the problem of timecode synchronization failure due to packet loss in wireless networks, achieving high-precision timecode synchronization.

[0006] In reality, timecode synchronization technology is susceptible to signal delays and packet loss due to line loss and electromagnetic interference during transmission. It also requires additional timecode generators and supporting transmission links, resulting in higher hardware procurement costs.

[0007] II. Synchronization Technology Based on Digital Watermarking

[0008] Patent CN102523535B discloses a method for real-time registration and synchronous transmission of video streams and GPS based on digital watermarking technology. Based on the analysis of compression attacks in video transmission, and considering that after encoding, the pixel values ​​of video frames fluctuate within a certain range but their positions remain unchanged, a quantization rule is established by mathematically transforming the video frames to correspond the frequency domain coefficients to the watermark information, embedding the watermark information into the video stream. Using this method, the sampling frequency of the GPS signal can be used as a reference during video acquisition, updating and embedding the watermark information in real time into the acquired video stream for transmission. A watermark extraction module is provided at the video receiving terminal to detect the watermark in the received video frames and extract the hidden GPS information in real time, simplifying and enhancing the accuracy and reliability of the real-time registration and synchronous transmission mechanism between the video stream and GPS signals.

[0009] However, this type of technology still has two major problems: First, the watermark embedding process requires modification of pixel values. Even with implicit watermarking algorithms, it may still have a slight impact on the brightness and color accuracy of the image. For scenarios with extremely high image quality requirements, such as display device testing and professional video production, this impact may lead to deviations in test results or a degradation in content quality. Second, the watermark has weak anti-interference capabilities. In practical applications, image compression encoding, partial occlusion in the shooting scene, color distortion caused by changes in light, and image noise pollution may all damage the integrity of the watermark information, leading to watermark extraction failure and ultimately causing synchronization interruption.

[0010] III. Synchronization Technology Based on Hardware Synchronization Signals

[0011] CN117998028B discloses a hardware-synchronized image acquisition controller structure, which includes a data synchronization module, a data buffer module, and a main control module. This controller is based on the AMBA system and is a peripheral connected to the AHB bus. The data synchronization module transmits pixel data from the external camera's pixel clock domain to the AHB bus clock domain, performs 32-bit data sorting, and then sends the data to the data buffer module. The data buffer module is an 8-depth, 32-bit FIFO used to buffer the data sent by the data synchronization module. The main control module mainly includes a state machine and two sub-modules, and the state machine transitions are completed by recognizing the rising and falling edges of the hardware synchronization signal. The data bit width and sorting method can be controlled through the AHB master device. The controller has a simple structure, reliable transmission, and is suitable for camera modules that support hardware synchronization mode.

[0012] Hardware synchronization technology provides a unified trigger signal to the display device and camera through a dedicated synchronization cable or triggering device, achieving frame synchronization between the two. Although this technology can achieve high synchronization accuracy in ideal environments, its deployment flexibility is extremely poor. For mobile scenarios, temporary test environments, or multi-device distributed deployment scenarios, hardware wiring and trigger installation are cumbersome, making rapid deployment and flexible adjustment impossible, thus limiting its adaptability.

[0013] Meanwhile, in addition to the aforementioned shortcomings, all three existing synchronization technologies suffer from insufficient robustness. In real-world applications, anomalies such as partial occlusion, screen tearing, frame jitter, and frame dropping occur frequently. Existing synchronization technologies lack effective adaptive compensation and fault-tolerance mechanisms, making them unable to cope with these complex interferences. This can easily lead to decreased synchronization accuracy, loss of synchronization signals, or even complete synchronization failure, severely impacting the normal operation of downstream services.

[0014] Therefore, there is an urgent need to develop a video synchronization method and system to solve the problems in the existing technology. Summary of the Invention

[0015] The purpose of this invention is to provide a video synchronization method and system that solves the problem mentioned in the background art of decreased synchronization accuracy or even complete synchronization failure due to abnormal situations such as frame loss and duplicate frames by comparing the difference between the expected logical frame number and the candidate logical frame number.

[0016] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0017] A video synchronization method includes the following steps:

[0018] Obtain the logical frame number; calculate the corresponding screen frame number based on the logical frame number and the mapping relationship.

[0019] Convert the screen frame number into encoded color blocks and display them on the video screen;

[0020] Acquire video footage, record the recording timestamp, and decode the current screen frame number from the video footage;

[0021] Obtain the average frame interval, the previous logical frame number, and the previous timestamp; calculate the expected logical frame number based on the shooting timestamp, average frame interval, previous logical frame number, and previous timestamp.

[0022] Based on the current screen frame number, calculate the corresponding candidate logical frame number through the mapping relationship;

[0023] Calculate the difference between the expected logical frame number and the candidate logical frame number; if the difference is greater than the threshold, output the expected logical frame number; if the difference is less than the threshold, output the candidate logical frame number.

[0024] Furthermore, the calculation of the corresponding screen frame number through the mapping relationship is performed using the following formula:

[0025] S = L mod M;

[0026] Where S is the screen frame number, L is the logical frame number, and M is the range of screen frame numbers.

[0027] Furthermore, the logical frame number is represented by the following formula:

[0028] L = Base + S + C * M;

[0029] Where L is the logical frame number; Base is the logical frame base value; C is the cycle count; and M is the screen frame number range.

[0030] Furthermore, the step of calculating the expected logical frame number based on the shooting timestamp, average frame interval, previous logical frame number, and previous timestamp includes the following steps:

[0031] According to the shooting timestamp T capture Compared to the previous timestamp T prev Calculate the time difference ΔT;

[0032] Based on the time difference ΔT and the average frame interval Δt avg Calculate the logical frame sequence number prediction increment ΔL pred ;

[0033] Predict the increment ΔL based on the logical frame number. pred With the previous logical frame number L prev Calculate the expected logical frame number L expected .

[0034] Furthermore, the step of calculating the corresponding candidate logical frame number based on the current screen frame number through a mapping relationship is achieved using the following formula:

[0035] L candidate =Base+S current +round((L expected -Base-S current ) / M)*M;

[0036] Among them, L candidate Here, S is the candidate logical frame sequence number, Base is the logical frame base value, and S is the logical frame base value. current The current screen frame number S current L expected M represents the expected logical frame number, and M represents the range of screen frame numbers.

[0037] Furthermore, the coded color blocks include red color blocks, blue color blocks, and green color blocks;

[0038] The red blocks indicate the index as follows:

[0039] R idx =floor(S / K) mod K;

[0040] The blue blocks indicate the index as follows:

[0041] B idx =floor(S / K²);

[0042] The green blocks indicate the index representation as follows:

[0043] G idx =S mod K.

[0044] Furthermore, the calculation steps for the average frame interval include:

[0045] The consecutive frame interval Δti is calculated by using the display timestamp Tdisplay or the capture timestamp Tcapture.

[0046] The average frame interval Δtavg is estimated using a filtering algorithm.

[0047] A video synchronization system includes a computer device, the computer device including a memory, a processor and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method.

[0048] Furthermore, it also includes:

[0049] A display, communicatively connected to the computer device, is used to display video images;

[0050] A camera, communicatively connected to the computer device, is used to capture video footage.

[0051] A computer program product includes a computer program that, when executed by a processor, implements the steps of the method.

[0052] The present invention has the following advantages:

[0053] (1) This application calculates the expected logical frame number by using the shooting timestamp, average frame interval, previous logical frame number, and previous timestamp, and can predict the expected logical frame number; and with the help of the current screen frame number, it can calculate the corresponding candidate logical frame number; thus, by comparing the expected logical frame number with the candidate logical frame number, it can automatically detect and correct abnormal situations such as dropped frames and duplicate frames. This achieves highly robust synchronization. This method does not perform frame interpolation generation, but ensures timing correctness through logical-level sequence number management, with low computational overhead and excellent real-time performance.

[0054] (2) This application embeds screen frame numbers encoded by color blocks in the display screen, which has strong resistance to occlusion, tearing and jitter, and can enhance the robustness of this solution.

[0055] Other features and advantages of the present invention will be disclosed in detail in the following detailed description and accompanying drawings. Attached Figure Description

[0056] Figure 1 This is the overall flowchart of this application.

[0057] Figure 2 The video image that displays the coded color blocks for this application. Detailed Implementation

[0058] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0059] A video synchronization method, such as Figure 1 As shown, it includes the following steps:

[0060] S1: Obtain the logical frame number; calculate the corresponding screen frame number based on the logical frame number and the mapping relationship;

[0061] S2: Convert the screen frame sequence number into encoded color blocks and display them on the video screen;

[0062] S3: Acquire video footage, record the shooting timestamp, and decode the current screen frame number from the video footage;

[0063] S4: Obtain the average frame interval, the previous logical frame number, and the previous timestamp; calculate the expected logical frame number based on the shooting timestamp, average frame interval, previous logical frame number, and previous timestamp.

[0064] S5: Calculate the corresponding candidate logical frame number based on the current screen frame number and the mapping relationship;

[0065] S6: Calculate the difference between the expected logical frame number and the candidate logical frame number; if the difference is greater than the threshold, output the expected logical frame number; if the difference is less than the threshold, output the candidate logical frame number.

[0066] Specifically, in step S1, the corresponding screen frame number is calculated through a mapping relationship, which is achieved using a forward mapping formula:

[0067] S = L mod M;

[0068] Where S is the screen frame number, L is the logical frame number, and M is the range of screen frame numbers.

[0069] In this embodiment, the screen frame number S is a value that cycles through 0 to M-1, and the logical frame number L is a continuous large integer, such as 64 bits.

[0070] Correspondingly, the logical frame number is represented by the following inverse mapping formula:

[0071] L = Base + S + C * M;

[0072] Where L is the logical frame number, representing the total number of frames currently decoded; Base is the logical frame base value, a dynamically adjusted offset that can be initially determined by the first successfully decoded frame; C is the cycle count, representing how many cycles the screen frame number S has cycled through; and M is the range of screen frame numbers.

[0073] Given the starting frame L0, then Base = L0 - S0. The method for obtaining the first successfully decoded frame is no different from that of subsequent frames. The screen frame sequence S is decoded independently for each frame, and there is no relationship between frames.

[0074] The step S2, which converts the screen frame sequence number into coded color blocks, includes the following steps:

[0075] Configure color block parameters K and M, and initialize Δt. avg

[0076] Where K is the number of color blocks in each group, and M is the range of screen frame numbers.

[0077] In this embodiment, as Figure 2 As shown, three sets of color blocks—red, green, and blue—are used for encoding. When K=8, the range of screen frame numbers is 0 to M-1, and M=K³=512. That is, when K=8, the total number of frame numbers that can be displayed on the screen is 512, from 0 to 511.

[0078] The red blocks indicate the index as follows:

[0079] R idx =floor(S / K) mod K.

[0080] The blue blocks indicate the index as follows:

[0081] B idx =floor(S / K²).

[0082] The green blocks indicate the index representation as follows:

[0083] G idx =S mod K.

[0084] Here, floor indicates rounding down.

[0085] Optionally, the above three-color index calculation method can be flexibly adjusted according to the characteristics of the display and imaging system. In actual display, each color block is a mixture of red, green, and blue, with each color component set according to the index value. For example, the color block corresponding to the index displays 255, and the others display 0. To enhance robustness, multiple sets of identical color blocks can be displayed on the screen for redundant encoding.

[0086] Specifically, displaying coded color blocks on a video screen can be achieved using existing technologies, which will not be elaborated upon in this application.

[0087] Optionally, S2 further includes: recording and displaying a timestamp T. display .

[0088] The step S3, which decodes the current screen frame number from the video image, includes the following steps:

[0089] Identify coded color blocks in video footage;

[0090] The current screen frame number is identified by coded color blocks.

[0091] Specifically, identifying coded color blocks in a video frame can be achieved using existing technologies, and identifying the current screen frame number through coded color blocks can be combined with S2 identification, which will not be elaborated further in this application.

[0092] In step S4, the average frame interval, the previous logical frame number, and the previous timestamp are obtained; based on the shooting timestamp, average frame interval, previous logical frame number, and previous timestamp, the expected logical frame number is calculated; specifically, this includes the following steps:

[0093] Based on the displayed timestamp T display Or shooting timestamp T capture Calculate the average frame interval Δt avg ;

[0094] Get the previous logical frame number L prev Compared to the previous timestamp T prev ;

[0095] According to the shooting timestamp T capture Compared to the previous timestamp T prev Calculate the time difference ΔT;

[0096] Based on the time difference ΔT and the average frame interval Δt avg Calculate the logical frame sequence number prediction increment ΔL pred ;

[0097] Predict the increment ΔL based on the logical frame number. pred With the previous logical frame number L prev Calculate the expected logical frame number L expected .

[0098] Under normal circumstances, according to the reverse mapping formula, the logical frame number L should increase linearly with the timestamp T. This is achieved using the average frame interval Δt obtained through tracking. avg This allows us to predict the expected growth of L between any two time points. Using the inverse mapping formula, we can calculate the relationship with S. current Corresponding, closest to L expected The logical frame number is used to find an L candidate , so that (L candidate -L expected The absolute value of ) is the smallest.

[0099] Among them, the average frame interval Δt is calculated. avg It includes the following steps:

[0100] By displaying the timestamp T display Or shooting timestamp T capture Calculate the consecutive frame interval Δt i ;

[0101] The average frame interval Δt is estimated using a filtering algorithm. avg .

[0102] Where, Δt i This can be expressed by the following formula:

[0103] Δt i =T i -T i-1 ;

[0104] Among them, T i Let T be the time of the i-th frame. i-1 This is the time at frame i-1. The timestamps T at frame i and i-1 are used to display the timestamps T. display Or shooting timestamp T capture get.

[0105] In this embodiment, the filtering algorithm is the exponential moving average method, and the calculation formula is as follows:

[0106] Δt avg =α*Δt i +(1-α)*Δt avglast ;

[0107] Where α is the smoothing factor, used to control the response speed to new data; Δt avglast This represents the average frame interval calculated in the previous iteration. The smoothing factor α can be adjusted according to the actual situation, generally between 0.05 and 0.2. Using a smaller α, the frame interval calculation result tends more towards the long-term mean, while a larger α reflects short-term changes more accurately. Since the frame interval of a typical video source is basically constant, a smaller value for α can be chosen.

[0108] In step S5, the corresponding candidate logical frame number is calculated based on the current screen frame number through a mapping relationship, using the following formula:

[0109] L candidate =Base+S current +round((L expected -Base-S current ) / M)*M;

[0110] Among them, L candidate Here, S is the candidate logical frame sequence number, Base is the logical frame base value, and S is the logical frame base value. current The current screen frame number S current L expected M represents the expected logical frame number, M represents the range of screen frame numbers, and round indicates rounding to an integer.

[0111] In this embodiment, through L expected Calculate the cycle count C, and use round-robin to avoid the expected logical frame number L. expected Affecting candidate logical frame number L candidate .

[0112] Specifically, S6 includes the following:

[0113] Calculate the expected logical frame number L expected With candidate logical frame number L candidate If the difference is less than the threshold, then the output correction logic frame number L is... decode Equal to candidate logical frame number L candidate If the difference is greater than the threshold, a serious sequence number jump is considered to have occurred, and the predicted value is forced to be used, correcting the logical frame sequence number L. decde Equal to the expected logical frame number L expected Meanwhile, based on the output correction logic frame number L decde Or expected logical frame number L expected Update the logical frame base value.

[0114] Specifically, the formula for calculating the difference is as follows:

[0115] Diff=|L candidate -L expected |

[0116] In this embodiment, the threshold is β*M. β is an empirical value, such as 0.25 or 0.3. It is selected based on the actual system's tolerance requirements for sequence number jumps. When β is small, jumps are easier to identify, which is suitable for scenarios with low error tolerance and where errors are not allowed. Conversely, when β is large, jumps are less likely to occur, which is suitable for scenarios requiring continuity and where errors are not a major concern.

[0117] If Diff > β*M, then a serious sequence number jump has occurred, which may be caused by continuous frame loss, decoding errors, or initialization problems.

[0118] Finally, output the corrected correction logic frame number L. decode and synchronized shooting timestamp T capture It is used for upper-layer applications.

[0119] A video synchronization system, comprising:

[0120] A computer device, the computer device including a memory, a processor and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method;

[0121] A display, communicatively connected to the computer device, is used to display video images;

[0122] A camera, communicatively connected to the computer device, is used to capture video footage;

[0123] A computer program product includes a computer program that, when executed by a processor, implements the steps of the method.

[0124] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method.

[0125] In summary, this application has the following beneficial effects:

[0126] 1. Extremely high robustness: The color block encoding method has strong resistance to occlusion, tearing and jitter; combined with frame order prediction by shooting timestamp, average frame interval, previous logical frame number and previous timestamp, it can automatically detect and correct abnormal situations such as dropped frames and duplicate frames.

[0127] 2. Precise timing synchronization: Through bidirectional mapping and frame rate tracking, precise synchronization is achieved not only in sequence number but also in time dimension, providing a reliable timestamp for subsequent processing.

[0128] 3. High resource efficiency: It does not involve computationally intensive frame interpolation operations at all. All synchronization is completed at the logical sequence number level, with low computational overhead and excellent real-time performance.

[0129] 4. Flexible and low-cost deployment: No special hardware is required; it can be implemented using ordinary monitors and cameras, greatly reducing the application threshold.

[0130] 5. High versatility: The core algorithm does not depend on specific content and is applicable to a wide range of scenarios, from professional virtual filmmaking to automated display / camera testing.

[0131] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

[0132] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A video synchronization method, characterized in that, Includes the following steps: Obtain the logical frame number; calculate the corresponding screen frame number based on the logical frame number and the mapping relationship. Convert the screen frame number into encoded color blocks and display them on the video screen; Acquire video footage, record the recording timestamp, and decode the current screen frame number from the video footage; Obtain the average frame interval, the previous logical frame number, and the previous timestamp; Calculate the expected logical frame number based on the shooting timestamp, average frame interval, previous logical frame number, and previous timestamp; Based on the current screen frame number, calculate the corresponding candidate logical frame number through the mapping relationship; Calculate the difference between the expected logical frame number and the candidate logical frame number; If the difference is greater than the threshold, the expected logical frame number is output. If the difference is less than the threshold, the candidate logical frame number is output. The calculation of the expected logical frame number based on the shooting timestamp, average frame interval, previous logical frame number, and previous timestamp includes the following steps: Calculate the time difference ΔT based on the capture timestamp Tcapture and the previous timestamp Tprev; The logical frame number prediction increment ΔLpred is calculated based on the time difference ΔT and the average frame interval Δtavg. The expected logical frame number Lexpected is calculated based on the predicted increment ΔLpred of the logical frame number and the previous logical frame number Lprev.

2. The video synchronization method according to claim 1, characterized in that, The calculation of the corresponding screen frame number through the mapping relationship is performed using the following formula: S = L mod M; Where S is the screen frame number, L is the logical frame number, and M is the range of screen frame numbers.

3. The video synchronization method according to claim 2, characterized in that, The logical frame number is represented by the following formula: L = Base + S + C * M; Where L is the logical frame number; Base is the logical frame base value; C is the cycle count; and M is the screen frame number range.

4. The video synchronization method according to claim 3, characterized in that, The calculation of the corresponding candidate logical frame number based on the current screen frame number through a mapping relationship is achieved using the following formula: Lcandidate=Base+Scurrent+round((Lexpected-Base-Scurrent) / M)*M; Where Lcandidate is the candidate logical frame number, Base is the logical frame base value, Scurrent is the current screen frame number Scurrent; Lexpected is the expected logical frame number, and M is the range of screen frame numbers.

5. The video synchronization method according to any one of claims 1-4, characterized in that, The coded color blocks include red, blue, and green color blocks; The red blocks indicate the index as follows: Ridx = floor(S / K) mod K; The blue blocks indicate the index as follows: Bidx = floor(S / K²); The green blocks indicate the index representation as follows: Gidx = S mod K.

6. The video synchronization method according to any one of claims 1-4, characterized in that, The calculation steps for the average frame interval include: The consecutive frame interval Δti is calculated by using the display timestamp Tdisplay or the capture timestamp Tcapture. The average frame interval Δtavg is estimated using a filtering algorithm.

7. A video synchronization system, comprising a computer device, the computer device including a memory, a processor and a computer program stored in the memory, the processor executing the computer program to implement the steps of the method according to any one of claims 1-6.

8. The video synchronization system according to claim 7, characterized in that, Also includes: A display, communicatively connected to the computer device, is used to display video images; A camera, communicatively connected to the computer device, is used to capture video footage.

9. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1-6.

Images