An end-to-end picture synchronization optimization and evaluation system for LED movie virtual shooting

Abstract

This invention discloses an end-to-end image synchronization optimization and evaluation system for LED cinema virtual shooting, specifically relating to the fields of digital video processing, display control, and virtual shooting technologies. The invention synchronously acquires images from the virtual engine output, LED display feedback, and camera capture, and performs unified time base calibration on these three images to establish a globally unified timing benchmark. Based on the calibrated data, it conducts multi-dimensional joint analysis of timing deviations, frame misalignments, brightness changes, and color shifts between rendering output, screen feedback, and camera imaging, quantitatively evaluating the synchronization status of the entire link. According to the analysis results, it coordinately adjusts the rendering queue depth, video transmission link latency, LED display refresh timing, and camera exposure timing to achieve end-to-end dynamic synchronization optimization. Finally, it outputs synchronization accuracy, frame stability, image consistency, and a comprehensive score, forming a quantitative evaluation report.

Description

Technical Field

[0001] This invention relates to the fields of digital video processing, display control, and virtual shooting technologies. More specifically, this invention relates to an end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films. Background Technology

[0002] LED virtual shooting technology, with its advantages of virtual-real fusion visualization, realistic lighting effects, and pre-production post-processing, has become one of the core technologies for film, television, and advertising production. Its core principle is to render a 3D scene in real time using a virtual engine, projecting the image onto an LED display screen to form a virtual background. The camera simultaneously captures the real foreground scene and the virtual background on the LED screen, directly obtaining a fused virtual-real image, significantly reducing the workload of post-production compositing. However, in practical applications, existing technologies still have many shortcomings that urgently need to be addressed:

[0003] First, the timing synchronization across the entire link is poor. Virtual shooting involves multiple key stages, including virtual engine rendering, video signal transmission, LED display refresh, and camera exposure. Each stage has inherent delays and is susceptible to interference from external factors, resulting in inaccurate alignment of the frame timing at each link node. For example, the virtual engine rendering delay dynamically changes with the complexity of the scene, network jitter during video transmission causes signal delay fluctuations, and the LED display refresh timing is out of phase with the camera exposure timing. These problems directly cause phenomena such as ghosting, tearing, flickering, and frame misalignment in the captured image, seriously affecting the quality of the final product.

[0004] Secondly, there is a lack of precise quantification methods for synchronization status. Existing technologies largely rely on the subjective visual judgment of operators, making it impossible to accurately detect and quantify key indicators such as timing deviations, frame synchronization stability, brightness consistency, and color shifts. Even if some solutions introduce simple delay measurement functions, they can only detect a single link and cannot cover the entire chain. This results in a lack of data support for optimization and adjustments, leading to blind optimization and low efficiency. Summary of the Invention

[0005] In order to overcome the above-mentioned defects of the prior art, embodiments of the present invention provide an end-to-end image synchronization optimization and evaluation system for virtual shooting of LED movies, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] The shooting data acquisition module is used to acquire images from the virtual engine output, LED display feedback, and camera capture.

[0008] The timestamp calibration module is connected to the shooting data acquisition module and the image synchronization analysis module respectively. It is used to perform unified time base calibration on the image data, control signals and rendering signals corresponding to the virtual engine output image, LED display feedback image and camera acquisition image.

[0009] The image synchronization analysis module is connected to the timestamp calibration module and the delay compensation optimization module respectively. It is used to analyze the timing deviation, frame misalignment, brightness change and color shift between the rendering output, screen display feedback and camera imaging based on the data after unified time base calibration.

[0010] The delay compensation optimization module is connected to the screen synchronization analysis module and the display control module respectively, and is used to coordinately adjust the rendering queue, video transmission link, LED refresh timing and camera exposure timing according to the analysis results of the screen synchronization analysis module.

[0011] The display control module, connected to the delay compensation optimization module, is used to control the display output of the LED display screen and the exposure synchronization of the camera according to the adjusted parameters.

[0012] The synchronization evaluation module is connected to the screen synchronization analysis module and the delay compensation optimization module, respectively, and is used to output the synchronization accuracy, frame stability, screen consistency and comprehensive score results based on the analysis results before and after optimization.

[0013] Preferably, in the shooting data acquisition module, the shooting data acquisition module is used to simultaneously acquire the image data of three key nodes in the entire virtual shooting chain, namely: the virtual engine output image, the LED display feedback image, and the camera acquisition image; specifically, the shooting data acquisition module includes a virtual engine acquisition unit, an LED screen feedback acquisition unit, and a camera acquisition unit;

[0014] The virtual engine acquisition unit is used to acquire the raw image data output by the virtual engine rendering pipeline. The virtual engine runs on a high-performance graphics workstation or rendering server cluster, and its output screen represents the ideal visual effect of the virtual scene. The virtual engine acquisition unit adopts frame buffer capture technology. After the rendering pipeline of the engine completes the rendering of each frame and before it is sent to the video output interface, the raw pixel data is read directly from the frame buffer object through the engine's built-in plugin or external frame capture tool. During the acquisition process, the virtual engine acquisition unit synchronously records the rendering completion time, frame number, and vertical synchronization signal status of the frame, and stores or forwards it in the original color space and uncompressed format.

[0015] The LED screen feedback acquisition unit is used to acquire the actual screen data displayed on the LED screen in real time; a high-precision optical sensor array, such as a colorimeter or photometer, is deployed on the surface of the LED screen to acquire the brightness, color and flicker characteristics of the screen surface in real time through non-contact measurement and reconstruct it into two-dimensional image data.

[0016] Preferably, the high-precision time base unit is used to generate a globally unified reference clock signal. The high-precision time base unit uses an external high-precision clock source as the master clock and a Global Positioning System or BeiDou Navigation Satellite System receiver with a timing accuracy better than ±1 microsecond as the external clock source to generate a high-precision second pulse signal and a high-frequency clock signal. The reference clock signal is distributed to each acquisition unit, display control device and camera synchronization interface through a dedicated synchronization signal distributor in a star topology or cascaded manner to ensure that the entire system operates in a unified clock domain.

[0017] The signal calibration unit is used to embed the reference clock signal generated by the high-precision time base unit into the vertical synchronization signal of the virtual engine, the refresh synchronization signal of the LED display screen, and the frame synchronization signal of the camera, respectively, to achieve unified time base calibration of image data, control signals, and rendering signals. Specifically, the signal calibration unit first calibrates the output screen of the virtual engine: at the moment when the vertical synchronization of the virtual engine rendering pipeline is interrupted, the signal calibration unit binds the current timestamp provided by the high-precision time base unit with the frame sequence number and rendering completion flag of the frame to form metadata containing precise time information, and embeds the metadata into the image data in the form of a watermark or transmits it in parallel with the image data through an independent data channel; for the LED display screen... For the feed image, the signal calibration unit captures the timestamp of the high-precision time base unit at the start of the LED display refresh cycle (i.e., the start of the first row of pixel scanning for each frame) and associates it with the display data fed back by the screen driver chip, thereby accurately marking the actual time origin of the frame. For the camera-captured image, the signal calibration unit injects the high-precision reference clock signal into the camera in a frame-synchronized manner through the camera synchronization input interface, so that the frame start time of the camera is precisely aligned with the global time base. At the same time, during the vertical blanking period of each frame image output by the camera, the signal calibration unit reads the exposure center time or frame start time generated internally by the camera, converts it into a timestamp under the global time base, and binds it to the image data of that frame.

[0018] Preferably, the timing deviation analysis unit is used to calculate the time offset between the rendering output time, the actual screen refresh time, and the camera exposure center time; specifically, the timing deviation analysis unit first obtains the timestamp information of each frame of the three data streams from the timestamp calibration module: for the virtual engine output screen, it obtains the rendering completion time. and the start time of vertical synchronization For the LED display screen's feedback image, obtain its actual refresh start time. For images captured by a camera, obtain the exposure center time. or frame start time The timing deviation analysis unit uses the exposure center time of the camera's captured image as the time reference (because it directly determines the final image capture time), and calculates the offset between this time and the rendering output time. and the offset between the display refresh time and the screen refresh time. To further characterize the end-to-end synchronization status of the entire link, the timing deviation analysis unit also calculates the offset between the rendering output and the screen refresh. The above offset calculation results are in microseconds, forming an end-to-end timing deviation map. The timing deviation analysis unit continuously monitors the dynamic change trend of the above offset, calculates its mean, standard deviation and drift rate. When the absolute value of the offset exceeds the preset threshold or the standard deviation of the offset exceeds the preset stability threshold, a synchronization anomaly warning is triggered.

[0019] The frame misalignment detection unit is used to detect the frame sequence number matching status between the three video feeds and identify whether there are any dropped, duplicate, or incorrect frames. The frame misalignment detection unit adopts a dual detection mechanism: firstly, hard matching detection based on frame sequence numbers; the virtual engine output screen, the LED display screen feedback screen, and the camera capture screen all maintain continuous frame sequence number counters in their respective links. The frame misalignment detection unit extracts the frame sequence numbers within the same timestamp window from the three data feeds and compares them one by one. When there are discontinuous or jumpy frame sequence numbers, or when the three frame sequence numbers cannot establish a corresponding relationship, it is determined that there are dropped or incorrect frames. When the same frame number appears repeatedly in a single data stream, it is determined that there is a duplicate frame phenomenon. Secondly, soft matching detection based on image content; when the frame number information is missing or unreliable, the frame misalignment detection unit uses an image feature matching algorithm to extract the perceptual hash features or scale-invariant feature transformation features of the three images, establishes the inter-frame correspondence through feature similarity comparison, and then identifies the frame misalignment state; the frame misalignment detection unit outputs the frame matching rate index (i.e., the proportion of correctly matched frames to the total number of frames) and the frame interval jitter value (i.e., the standard deviation of the time interval between adjacent frames) to quantitatively evaluate the stability of frame transmission.

[0020] Preferably, the transmission link compensation unit is used to measure the fixed delay in the video transmission link and apply corresponding advance or delay compensation; the video transmission link typically includes an optical fiber or SDI transmission path from the rendering server to the LED controller, as well as a processing buffer inside the LED controller; the compensation unit applies corresponding advance compensation at the video signal output end, that is, sends the signal a corresponding time ahead of the rendering output time, or applies delay compensation at the receiving end, so that the time when the signal arrives at the LED controller is precisely matched with the expected exposure window;

[0021] The LED timing synchronization unit is used to adjust the refresh start time and refresh cycle of the LED display screen to match the exposure window of the camera. The LED timing synchronization unit adjusts the refresh start time and refresh cycle of the LED display screen according to the output of the timing deviation analysis unit. The offset is adjusted by the LED display controller to set the refresh start phase of each frame. If the screen refresh is fixed ahead or behind the camera exposure, the LED timing synchronization unit dynamically adjusts the vertical synchronization start offset parameter of the LED display based on the camera frame synchronization signal, so that the effective display area of ​​the screen image completely covers the camera's exposure window. For LED displays using variable refresh rate technology, the LED timing synchronization unit can also adjust the LED refresh cycle in real time according to the camera frame rate to achieve dynamic synchronization between the refresh rate and the shooting frame rate.

[0022] The exposure timing coordination unit is used to adjust the camera's exposure start time and exposure duration to match the LED display's refresh cycle and bright / dark field distribution. Based on the timing deviation analysis results and the adjustment status of the LED timing synchronization unit, the exposure timing coordination unit sends precise frame synchronization and exposure control signals to the camera via the camera synchronization signal generator. When the LED display's bright field duration is short, the exposure timing coordination unit compresses the camera's exposure window and precisely aligns it with the bright field range to ensure that the camera completes exposure during the bright field period and acquires complete image information. When periodic brightness fluctuations are detected by the brightness change analysis unit, the exposure timing coordination unit fine-tunes the exposure start phase to ensure that the exposure window avoids the dark field transition area in the LED refresh cycle, thereby eliminating dark bands or flickering defects in the captured image.

[0023] Preferably, the LED display controller receives LED refresh timing adjustment parameters from the delay compensation optimization module and converts them into timing control instructions executable by the LED display driver chip. The LED display controller first establishes a communication link with the LED display's sending or receiving card and transmits the adjustment parameters via an industry-standard protocol. Specific adjustments include: a vertical synchronization start offset parameter, used to fine-tune the refresh start phase of each frame to ensure the screen's exposure window is precisely aligned with the camera's exposure center; a refresh cycle parameter, used to adjust the LED display's refresh rate to maintain an integer multiple relationship or dynamic match with the camera's frame rate; and a black frame insertion parameter, used to insert additional black frames when necessary to eliminate motion blur or improve visual persistence. The LED display controller performs parameter adjustments gradually, using a multi-step approach to progressively adjust to the target value, avoiding sudden parameter changes that could cause screen flicker or damage. The controller also monitors the LED display's feedback status in real time to confirm that the parameter adjustments have been successfully implemented and sends the execution status back to the delay compensation optimization module.

[0024] The camera synchronization signal generator receives exposure timing coordination parameters from the delay compensation optimization module and converts them into a synchronization signal recognizable by the camera. Based on the camera model and interface type, the generator generates frame synchronization signals and exposure control signals in corresponding formats. The frame synchronization signal locks the camera's frame start time, establishing a fixed phase relationship with the LED display's refresh timing. The exposure control signal precisely controls the camera's exposure start time and exposure duration, ensuring the exposure window is accurately aligned with the bright area of ​​the LED display. For advanced cameras supporting external exposure control, the camera synchronization signal generator can also output pulse width modulation signals to dynamically adjust the exposure duration, adapting to shooting requirements in different brightness scenarios. The camera synchronization signal generator uses a high-precision clock source synchronized with the time base reference provided by the delay compensation optimization module, ensuring the output signal's timing accuracy is better than the microsecond level.

[0025] Preferably, the synchronization evaluation module includes a synchronization accuracy scoring unit, a frame stability scoring unit, a picture consistency scoring unit, and a comprehensive scoring unit. The four scoring units calculate scores for each dimension based on the timing deviation map, frame synchronization status, brightness consistency index, and color consistency index provided by the picture synchronization analysis module. The comprehensive scoring unit then performs a weighted fusion output. The synchronization accuracy scoring unit calculates the end-to-end synchronization error value based on the timing deviation analysis results and converts it into a synchronization accuracy score. The synchronization accuracy scoring unit first obtains timing deviation data between the three signals from the picture synchronization analysis module, including the offset between the rendering output and the camera exposure. And the offset between the rendering output and the screen refresh rate. Synchronization accuracy scoring unit As a core evaluation indicator, it directly reflects the synchronization accuracy between the actual display of the LED screen and the imaging of the camera. The specific scoring method is as follows: The ideal synchronization error value is set to 0 microseconds, and the maximum tolerance error is set to... (Usually, 1 / 2 frame period is taken, that is, if the shooting frame rate is 24fps, then) (microseconds), synchronization accuracy score The calculation method is as follows:

[0026]

[0027] when Exceed At that time, the score drops to 0. The synchronization accuracy scoring unit also comprehensively considers the stability of timing deviations, through calculation. Standard deviation The score is adjusted accordingly. The better the stability, the higher the score is adjusted. The worse the stability, the lower the score is adjusted. Finally, a synchronization accuracy score between 0 and 100 is output.

[0028] The frame stability scoring unit calculates the frame matching rate and frame interval jitter value based on the frame misalignment detection results, and converts them into a frame stability score. The frame stability scoring unit obtains the frame matching rate from the image synchronization analysis module. (i.e., the proportion of correctly matched frames out of the total frames) and frame interval jitter value (i.e., the standard deviation of the time interval between adjacent frames, in microseconds). Frame stability score. Taking both indicators into account, the calculation method is as follows:

[0029]

[0030] in, This is the maximum tolerable value for frame interval jitter, typically set to 10% of the frame period. When the frame interval jitter exceeds... When the coefficient drops to 0, the frame stability scoring unit also records the location and frequency of dropped frames, duplicate frames, and erroneous frames, and deducts the corresponding score from the score, finally outputting a frame stability score between 0 and 100. This score reflects the stability of the video transmission link and frame synchronization control. The higher the score, the more stable and reliable the frame transmission across the entire link.

[0031] The technical effects and advantages of this invention are as follows:

[0032] 1. This invention performs unified time base calibration and joint analysis on the three channels of images: virtual engine output, LED screen display feedback, and camera acquisition. This enables precise location of timing deviations, frame misalignments, and image consistency issues in the entire virtual shooting chain, providing an accurate data foundation for synchronization optimization.

[0033] 2. This invention uses a delay compensation optimization module to coordinate the adjustment of the rendering queue, transmission link, LED refresh timing and camera exposure timing, achieving end-to-end dynamic synchronization optimization, effectively solving the problems of ghosting, tearing, flickering and image misalignment in the captured image;

[0034] 3. This invention outputs quantitative synchronization accuracy, frame stability, image consistency and comprehensive score results through the synchronization evaluation module, providing an objective and quantifiable evaluation basis for system debugging, equipment selection and shooting quality assessment. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Please see Figure 1 As shown, this invention provides an end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films, comprising:

[0038] The shooting data acquisition module is used to acquire images from the virtual engine output, LED display feedback, and camera capture.

[0039] The shooting data acquisition module is used to synchronously acquire image data from three key nodes in the entire virtual shooting chain: the virtual engine output image, the LED display feedback image, and the camera acquisition image. Specifically, the shooting data acquisition module includes a virtual engine acquisition unit, an LED screen feedback acquisition unit, and a camera acquisition unit.

[0040] The virtual engine acquisition unit is used to acquire the raw image data output by the virtual engine rendering pipeline. The virtual engine runs on a high-performance graphics workstation or rendering server cluster, and its output screen represents the ideal visual effect of the virtual scene. The virtual engine acquisition unit adopts frame buffer capture technology. After the rendering pipeline of the engine completes the rendering of each frame and before it is sent to the video output interface, the raw pixel data is read directly from the frame buffer object through the engine's built-in plugin or external frame capture tool. During the acquisition process, the virtual engine acquisition unit synchronously records the rendering completion time, frame number, and vertical synchronization signal status of the frame, and stores or forwards it in the original color space and uncompressed format.

[0041] The LED screen feedback acquisition unit is used to acquire the actual screen data displayed on the LED screen in real time; a high-precision optical sensor array, such as a colorimeter or photometer, is deployed on the surface of the LED screen to acquire the brightness, color and flicker characteristics of the screen surface in real time through non-contact measurement and reconstruct it into two-dimensional image data.

[0042] The camera acquisition unit is used to acquire the actual image captured by the camera. In the LED virtual shooting scene, the camera captures the virtual background and foreground of the LED display screen through its lens, and the imaging result directly determines the final image quality. The camera acquisition unit acquires the video stream data output by the camera in real time through the SDI or HDMI interface equipped on the camera body, using a video capture card or embedded acquisition board. During the acquisition process, the camera acquisition unit simultaneously parses the frame synchronization signal, timecode, and metadata information embedded in the video stream to ensure that each captured frame accurately corresponds to the camera's exposure sequence. For high-end digital cameras that support RAW format output, the camera acquisition unit prioritizes acquiring uncompressed and uncolored raw image data to retain complete imaging information for subsequent analysis.

[0043] The timestamp calibration module is connected to the shooting data acquisition module and the image synchronization analysis module respectively. It is used to perform unified time base calibration on the image data, control signals and rendering signals corresponding to the virtual engine output image, LED display feedback image and camera acquisition image.

[0044] The timestamp calibration module includes a high-precision time base unit and a signal calibration unit.

[0045] The high-precision time base unit is used to generate a globally unified reference clock signal. The high-precision time base unit uses an external high-precision clock source as the master clock and a Global Positioning System or BeiDou Satellite Navigation System receiver with a timing accuracy better than ±1 microsecond as the external clock source to generate a high-precision second pulse signal and a high-frequency clock signal. This reference clock signal is distributed to each acquisition unit, display control device and camera synchronization interface through a dedicated synchronization signal distributor in a star topology or cascaded manner to ensure that the entire system operates in a unified clock domain.

[0046] The signal calibration unit is used to embed the reference clock signal generated by the high-precision time base unit into the vertical synchronization signal of the virtual engine, the refresh synchronization signal of the LED display screen, and the frame synchronization signal of the camera, respectively, to achieve unified time base calibration of image data, control signals, and rendering signals. Specifically, the signal calibration unit first calibrates the output screen of the virtual engine: at the moment when the vertical synchronization of the virtual engine rendering pipeline is interrupted, the signal calibration unit binds the current timestamp provided by the high-precision time base unit with the frame sequence number and rendering completion flag of the frame to form metadata containing precise time information, and embeds the metadata into the image data in the form of a watermark or transmits it in parallel with the image data through an independent data channel; for the LED display screen... For the feed image, the signal calibration unit captures the timestamp of the high-precision time base unit at the start of the LED display refresh cycle (i.e., the start of the first row of pixel scanning for each frame) and associates it with the display data fed back by the screen driver chip, thereby accurately marking the actual time origin of the frame. For the camera-captured image, the signal calibration unit injects the high-precision reference clock signal into the camera in a frame-synchronized manner through the camera synchronization input interface, so that the frame start time of the camera is precisely aligned with the global time base. At the same time, during the vertical blanking period of each frame image output by the camera, the signal calibration unit reads the exposure center time or frame start time generated internally by the camera, converts it into a timestamp under the global time base, and binds it to the image data of that frame.

[0047] The timestamp calibration module assigns a precise timestamp under a globally unified time base to each frame of rendered image output by the virtual engine, each frame of screen display on the LED screen, and each frame of imaging recorded by the camera. This timestamp marks the key time points of each frame on its respective link node: for the virtual engine output, it marks the rendering completion time and the start time of vertical synchronization; for the LED screen feedback, it marks the actual refresh start time; and for the camera-captured image, it marks the exposure center time or the frame start time. These three data streams, carrying timestamps under a unified time base, are synchronously sent to the image synchronization analysis module. This allows the module to calculate the timing deviation between the rendered output, screen feedback, and camera imaging with microsecond-level precision, providing accurate data support for subsequent latency compensation optimization.

[0048] The image synchronization analysis module is connected to the timestamp calibration module and the delay compensation optimization module respectively. It is used to analyze the timing deviation, frame misalignment, brightness change and color shift between the rendering output, screen display feedback and camera imaging based on the data after unified time base calibration.

[0049] The timing deviation analysis unit is used to calculate the time offset between the rendering output time, the actual screen refresh time, and the camera exposure center time. Specifically, the timing deviation analysis unit first obtains the timestamp information of each frame in the three data streams from the timestamp calibration module: for the virtual engine output screen, it obtains the rendering completion time. and the start time of vertical synchronization For the LED display screen's feedback image, obtain its actual refresh start time. For images captured by a camera, obtain the exposure center time. or frame start time The timing deviation analysis unit uses the exposure center time of the camera's captured image as the time reference (because it directly determines the final image capture time), and calculates the offset between this time and the rendering output time. and the offset between the display refresh time and the screen refresh time. To further characterize the end-to-end synchronization status of the entire link, the timing deviation analysis unit also calculates the offset between the rendering output and the screen refresh. The above offset calculation results are in microseconds, forming an end-to-end timing deviation map. The timing deviation analysis unit continuously monitors the dynamic change trend of the above offset, calculates its mean, standard deviation and drift rate. When the absolute value of the offset exceeds the preset threshold or the standard deviation of the offset exceeds the preset stability threshold, a synchronization anomaly warning is triggered.

[0050] The frame misalignment detection unit is used to detect the frame sequence number matching status between the three video feeds and identify whether there are any dropped, duplicate, or incorrect frames. The frame misalignment detection unit adopts a dual detection mechanism: firstly, hard matching detection based on frame sequence numbers; the virtual engine output screen, the LED display screen feedback screen, and the camera capture screen all maintain continuous frame sequence number counters in their respective links. The frame misalignment detection unit extracts the frame sequence numbers within the same timestamp window from the three data feeds and compares them one by one. When there are discontinuous or jumpy frame sequence numbers, or when the three frame sequence numbers cannot establish a corresponding relationship, it is determined that there are dropped or incorrect frames. When the same frame number appears repeatedly in a single data stream, it is determined that there is a duplicate frame phenomenon. Secondly, soft matching detection based on image content; when the frame number information is missing or unreliable, the frame misalignment detection unit uses an image feature matching algorithm to extract the perceptual hash features or scale-invariant feature transformation features of the three images, establishes the inter-frame correspondence through feature similarity comparison, and then identifies the frame misalignment state; the frame misalignment detection unit outputs the frame matching rate index (i.e., the proportion of correctly matched frames to the total number of frames) and the frame interval jitter value (i.e., the standard deviation of the time interval between adjacent frames) to quantitatively evaluate the stability of frame transmission.

[0051] The brightness variation analysis unit is used to analyze the brightness consistency and fluctuation between the screen display and the camera image. The unit first performs spatial registration between the LED display feedback image and the camera image. This spatial registration process includes: first, using the Scale Invariant Feature Transform (SIFT) algorithm to extract local feature points from both the LED display feedback image and the camera image; second, using Fast Approximate Nearest Neighbor (FLANN) to match feature point pairs and employing the Random Sample Consensus (RANSAC) algorithm to eliminate mismatched points; and finally, calculating the homography matrix of the perspective transformation based on the purified matching point pairs. And using the homography matrix Geometric correction is performed on the camera image to eliminate geometric misalignment caused by lens distortion and differences in viewing angle. After spatial registration, the brightness change analysis unit calculates the brightness difference between the two images using pixel-by-pixel or region-by-region statistical methods. Specific calculation methods include, but are not limited to, calculating the average brightness difference value across the entire image. ,in, This refers to the brightness value of the screen display. The image brightness values ​​are determined by the camera. The brightness histogram similarity is calculated, quantifying the similarity of the brightness distributions of the two images using Bach distance or cross-entropy.

[0052]

[0053] in and The two images are at different brightness levels. The normalized histogram probability on, and ;

[0054] The peak signal-to-noise ratio is calculated, and the brightness fidelity of the two images is evaluated using mean square error. The brightness variation analysis unit also focuses on analyzing the brightness fluctuations in the camera's image. By calculating the rate of change of brightness between consecutive frames in the time domain, it identifies whether there are periodic brightness fluctuations (such as dark bands or flickering phenomena) caused by the mismatch between the LED display refresh sequence and the camera exposure sequence. When the absolute value of the brightness difference exceeds a preset threshold or the brightness fluctuation amplitude exceeds a preset stability threshold, a brightness synchronization anomaly indication is output.

[0055] The color shift analysis unit is used to analyze the chromaticity differences and color shifts between the displayed image and the camera image. Similar to the brightness change analysis unit, the color shift analysis unit first performs spatial registration between the displayed image and the camera image, and then performs quantitative analysis in the color space. The color shift analysis unit converts the displayed image and the camera image from the RGB color space to the CIELAB uniform color space and calculates the color difference of each pixel. And calculate the average color difference of the entire image. and maximum color difference Preferably, the CIE 2000 color difference formula ΔE00 is used for calculation to obtain a color difference assessment result that better matches human visual perception. The color shift analysis unit also analyzes the temporal characteristics of color drift, monitors the changing trend of color difference between consecutive frames, and identifies whether there is a slow color drift phenomenon caused by unstable LED display driving timing, temperature changes, or camera white balance drift. The color shift analysis unit outputs the average color difference, maximum color difference, and color difference standard deviation as quantitative indicators. When the average color difference exceeds a preset threshold (e.g., ΔE00 > 3) or the color difference standard deviation exceeds a preset stability threshold, a color synchronization anomaly indication is output.

[0056] The four analysis units in the picture synchronization analysis module work together to construct a comprehensive quantitative assessment of the synchronization status of the entire link. The timing deviation analysis unit provides the synchronization status in the time dimension, the frame misalignment detection unit provides the synchronization status in the frame structure dimension, and the brightness change analysis unit and color shift analysis unit provide the synchronization status in the picture content dimension. The analysis results of the four dimensions corroborate and complement each other. For example, periodic brightness fluctuations are often associated with the fixed phase shift detected by the timing deviation analysis unit, while color difference abnormalities may be related to the picture content mismatch caused by frame misalignment.

[0057] The image synchronization analysis module encapsulates the above analysis results into structured data packets and outputs them synchronously to the delay compensation optimization module and the synchronization evaluation module. The output data includes: timing deviation spectrum (containing the time offset between each pair of the three signals and its statistical characteristics), frame synchronization status (containing frame matching rate, frame interval jitter value and a list of dropped / repeated frame positions), brightness consistency index (containing average brightness difference, brightness histogram similarity and brightness fluctuation spectrum), and color consistency index (containing average color difference, maximum color difference and color difference distribution characteristics).

[0058] The delay compensation optimization module is connected to the screen synchronization analysis module and the display control module respectively, and is used to coordinately adjust the rendering queue, video transmission link, LED refresh timing and camera exposure timing according to the analysis results of the screen synchronization analysis module.

[0059] The rendering queue adjustment unit is used to dynamically adjust the rendering queue depth and pre-rendered frame count of the virtual engine based on the timing deviation analysis results. When a timing deviation is detected between the virtual engine's output image and the camera's exposure time (i.e.,...), the unit adjusts the rendering queue depth and pre-rendered frame count. Earlier When the lead time exceeds a threshold, the rendering queue adjustment unit pushes the rendering output backward to align with the camera exposure window by increasing the rendering queue depth or inserting additional pre-rendered delayed frames; conversely, when it detects that the rendering output lags behind the camera exposure, it shortens the delay from rendering to output by reducing the rendering queue depth or decreasing the number of pre-rendered frames.

[0060] The transmission link compensation unit is used to measure the fixed delay in the video transmission link and apply corresponding advance or delay compensation. The video transmission link typically includes an optical fiber or SDI transmission path from the rendering server to the LED controller, as well as a processing buffer inside the LED controller. The compensation unit applies corresponding advance compensation at the video signal output end, that is, sends the signal a corresponding time ahead of the rendering output time, or applies delay compensation at the receiving end, so that the time when the signal arrives at the LED controller is precisely matched with the expected exposure window.

[0061] The LED timing synchronization unit is used to adjust the refresh start time and refresh cycle of the LED display screen to match the exposure window of the camera. The LED timing synchronization unit adjusts the refresh start time and refresh cycle of the LED display screen according to the output of the timing deviation analysis unit. The offset is adjusted by the LED display controller to set the refresh start phase of each frame. If the screen refresh is fixed ahead or behind the camera exposure, the LED timing synchronization unit dynamically adjusts the vertical synchronization start offset parameter of the LED display based on the camera frame synchronization signal, so that the effective display area of ​​the screen image completely covers the camera's exposure window. For LED displays using variable refresh rate technology, the LED timing synchronization unit can also adjust the LED refresh cycle in real time according to the camera frame rate to achieve dynamic synchronization between the refresh rate and the shooting frame rate.

[0062] The exposure timing coordination unit is used to adjust the camera's exposure start time and exposure duration to match the LED display's refresh cycle and bright / dark field distribution. Based on the timing deviation analysis results and the adjustment status of the LED timing synchronization unit, the exposure timing coordination unit sends precise frame synchronization and exposure control signals to the camera via the camera synchronization signal generator. When the LED display's bright field duration is short, the exposure timing coordination unit compresses the camera's exposure window and precisely aligns it with the bright field range to ensure that the camera completes exposure during the bright field period and acquires complete image information. When periodic brightness fluctuations are detected by the brightness change analysis unit, the exposure timing coordination unit fine-tunes the exposure start phase to ensure that the exposure window avoids the dark field transition area in the LED refresh cycle, thereby eliminating dark bands or flickering defects in the captured image.

[0063] The four adjustment units follow a collaborative principle when performing compensation operations. For example, when both the rendering queue adjustment unit and the transmission link compensation unit correct the delay, the delay compensation optimization module first performs link transmission delay compensation to establish a stable transmission benchmark, and then performs rendering queue adjustment on this basis to avoid mutual interference between the two types of compensation. Similarly, the LED timing synchronization unit and the exposure timing coordination unit form a closed-loop linkage, adjust the LED refresh timing based on the camera exposure window, and then fine-tune the exposure parameters in reverse according to the adjusted LED refresh characteristics, achieving optimal synchronization matching through iterative approximation.

[0064] The display control module, connected to the delay compensation optimization module, is used to control the display output of the LED display screen and the exposure synchronization of the camera according to the adjusted parameters.

[0065] The LED display controller receives LED refresh timing adjustment parameters from the delay compensation optimization module and converts them into timing control instructions executable by the LED display driver chip. The controller first establishes a communication link with the LED display's sending or receiving card, transmitting the adjustment parameters via an industry-standard protocol. Specific adjustments include: a vertical synchronization start offset parameter, used to fine-tune the refresh start phase of each frame to ensure precise alignment between the screen's exposure window and the camera's exposure center; a refresh cycle parameter, used to adjust the LED display's refresh rate to maintain an integer multiple relationship or dynamic match with the camera's frame rate; and a black frame insertion parameter, used to insert additional black frames when necessary to eliminate motion blur or improve visual persistence. The LED display controller employs a gradual adjustment method, progressively adjusting to the target value through multiple steps to avoid sudden parameter changes that could cause screen flicker or damage. The controller also monitors the LED display's feedback status in real time, confirming that the parameter adjustments have been successfully implemented and sending the execution status back to the delay compensation optimization module.

[0066] The camera synchronization signal generator receives exposure timing coordination parameters from the delay compensation optimization module and converts them into a synchronization signal recognizable by the camera. Based on the camera model and interface type, the generator generates frame synchronization signals and exposure control signals in corresponding formats. The frame synchronization signal locks the camera's frame start time, establishing a fixed phase relationship with the LED display's refresh timing. The exposure control signal precisely controls the camera's exposure start time and exposure duration, ensuring the exposure window is accurately aligned with the bright area of ​​the LED display. For advanced cameras supporting external exposure control, the camera synchronization signal generator can also output pulse width modulation signals to dynamically adjust the exposure duration, adapting to shooting requirements in different brightness scenarios. The camera synchronization signal generator uses a high-precision clock source synchronized with the time base reference provided by the delay compensation optimization module, ensuring the output signal's timing accuracy is better than the microsecond level.

[0067] The synchronization scheduling unit, acting as the coordination hub of the display control module, is responsible for the unified management of the collaborative operation between the LED display controller and the camera synchronization signal generator. The synchronization scheduling unit receives the collaborative adjustment parameter package from the delay compensation optimization module, parses the execution parameters for both the LED display and camera sides, and ensures that the parameters on both sides are strictly synchronized in time. Specifically, the synchronization scheduling unit maintains a precise execution schedule, aligning the execution times of LED refresh timing adjustment and camera exposure timing adjustment to the same frame synchronization boundary, avoiding new timing deviations introduced by asynchronous adjustments on both sides. The synchronization scheduling unit also has an execution conflict detection function. When it detects a mutual constraint between LED-side and camera-side adjustments (such as LED refresh cycle adjustment and camera exposure duration adjustment exceeding hardware capabilities), the synchronization scheduling unit sends feedback information to the delay compensation optimization module, requesting a regeneration of the collaborative optimization scheme.

[0068] The synchronization evaluation module is connected to the screen synchronization analysis module and the delay compensation optimization module, respectively, and is used to output the synchronization accuracy, frame stability, screen consistency and comprehensive score results based on the analysis results before and after optimization.

[0069] The synchronization evaluation module includes a synchronization accuracy scoring unit, a frame stability scoring unit, a picture consistency scoring unit, and a comprehensive scoring unit. These four scoring units calculate scores for each dimension based on the timing deviation map, frame synchronization status, brightness consistency index, and color consistency index provided by the picture synchronization analysis module. The comprehensive scoring unit then performs a weighted fusion output. The synchronization accuracy scoring unit calculates the end-to-end synchronization error value based on the timing deviation analysis results and converts it into a synchronization accuracy score. The synchronization accuracy scoring unit first obtains timing deviation data between the three signals from the picture synchronization analysis module, including the offset between the rendered output and the camera exposure. And the offset between the rendering output and the screen refresh rate. Synchronization accuracy scoring unit As a core evaluation indicator, it directly reflects the synchronization accuracy between the actual display of the LED screen and the imaging of the camera. The specific scoring method is as follows: The ideal synchronization error value is set to 0 microseconds, and the maximum tolerance error is set to... (Usually, 1 / 2 frame period is taken, that is, if the shooting frame rate is 24fps, then) (microseconds), synchronization accuracy score The calculation method is as follows:

[0070]

[0071] when Exceed At that time, the score drops to 0. The synchronization accuracy scoring unit also comprehensively considers the stability of timing deviations, through calculation. Standard deviation The score is adjusted accordingly. The better the stability, the higher the score is adjusted. The worse the stability, the lower the score is adjusted. Finally, a synchronization accuracy score between 0 and 100 is output.

[0072] The frame stability scoring unit calculates the frame matching rate and frame interval jitter value based on the frame misalignment detection results, and converts them into a frame stability score. The frame stability scoring unit obtains the frame matching rate from the image synchronization analysis module. (i.e., the proportion of correctly matched frames out of the total frames) and frame interval jitter value (i.e., the standard deviation of the time interval between adjacent frames, in microseconds). Frame stability score. Taking both indicators into account, the calculation method is as follows:

[0073]

[0074] in, This is the maximum tolerable value for frame interval jitter, typically set to 10% of the frame period. When the frame interval jitter exceeds... When this occurs, the coefficient drops to 0; the frame stability scoring unit also records the location and frequency of dropped frames, duplicate frames, and erroneous frames, deducting corresponding points from the score, and finally outputting a frame stability score between 0 and 100; this score reflects the stability of the video transmission link and frame synchronization control, and the higher the score, the more stable and reliable the frame transmission across the entire link; the picture consistency scoring unit is used to calculate the peak signal-to-noise ratio, structural similarity index, and color difference index based on the brightness change analysis results and color shift analysis results, and convert them into a picture consistency score; the picture consistency scoring unit obtains the brightness consistency index (including average brightness difference) from the picture synchronization analysis module. Brightness histogram similarity ) and color consistency indicators (including average color difference) Maximum color difference Image consistency score Employing a multi-indicator fusion strategy:

[0075]

[0076] in, For brightness consistency scoring, Score the color consistency. Based on a comprehensive calculation using average brightness difference and brightness histogram similarity, when the absolute value of the average brightness difference is less than 5 viss per square meter and the histogram similarity is greater than 0.95, 100 points will be awarded; otherwise, points will be deducted proportionally. Based on average color difference Calculations are performed, setting the ideal color difference to 0, and the maximum tolerable color difference to be... (Set as 5), the specific calculation method for the color consistency score is as follows:

[0077]

[0078] The image consistency scoring unit also pays special attention to brightness fluctuations and color drift in the time domain. If periodic flickering or a continuous increase in color difference is detected, additional points are deducted from the initial score, ultimately outputting an image consistency score between 0 and 100. The comprehensive scoring unit is used to weight and fuse the synchronization accuracy score, frame stability score, and image consistency score to output a comprehensive synchronization performance score for the entire system. The comprehensive scoring unit can be configured with differentiated weight allocation strategies depending on the virtual shooting application scenario. Under the default configuration, the comprehensive score... The following weighting formula is used:

[0079]

[0080] in, To score the synchronization accuracy, Frame stability score, Rate the consistency of the image;

[0081] For shooting tasks primarily involving high-speed motion capture, the weight of frame stability score is increased; for shooting tasks requiring high precision in color reproduction and visual effects compositing, the weight of image consistency score is increased. The comprehensive scoring unit supports dynamic weight configuration, allowing users to adjust weight parameters via an external interface according to specific shooting needs.

[0082] The synchronous evaluation module not only outputs the final score after optimization, but also stores the initial score before optimization, forming a comparison report before and after optimization. The report is presented in the form of visual charts, which intuitively shows the changing trend and improvement of the scores in each dimension, providing objective basis for users to evaluate the synchronous optimization effect of the system.

[0083] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films, characterized in that, include: The shooting data acquisition module is used to acquire images from the virtual engine output, LED display feedback, and camera capture. The timestamp calibration module is connected to the shooting data acquisition module and the image synchronization analysis module respectively. It is used to perform unified time base calibration on the image data, control signals and rendering signals corresponding to the virtual engine output image, LED display feedback image and camera acquisition image. The image synchronization analysis module is connected to the timestamp calibration module and the delay compensation optimization module respectively. It is used to analyze the timing deviation, frame misalignment, brightness change and color shift between the rendering output, screen display feedback and camera imaging based on the data after unified time base calibration. The delay compensation optimization module is connected to the screen synchronization analysis module and the display control module respectively, and is used to coordinately adjust the rendering queue, video transmission link, LED refresh timing and camera exposure timing according to the analysis results of the screen synchronization analysis module. The display control module, connected to the delay compensation optimization module, is used to control the display output of the LED display screen and the exposure synchronization of the camera according to the adjusted parameters. The synchronization evaluation module is connected to the screen synchronization analysis module and the delay compensation optimization module, respectively, and is used to output the synchronization accuracy, frame stability, screen consistency and comprehensive score results based on the analysis results before and after optimization.

2. The end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films according to claim 1, characterized in that: The shooting data acquisition module includes a virtual engine acquisition unit, an LED screen feedback acquisition unit, and a camera acquisition unit. The virtual engine acquisition unit uses frame buffering technology to directly read the raw pixel data after the virtual engine rendering pipeline completes frame rendering and before sending it to the video output interface, and synchronously records the rendering completion time, frame number, and vertical synchronization signal status. The LED screen feedback acquisition unit uses a high-precision optical sensor array deployed on the screen surface to non-contactly acquire screen brightness, color, and flicker characteristics and reconstruct them into two-dimensional image data. The camera acquisition unit acquires the camera output video stream through the SDI / HDMI interface and capture card, parses the frame synchronization signal, timecode, and metadata, and prioritizes acquiring RAW format raw image data.

3. The end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films according to claim 1, characterized in that: The timestamp calibration module includes a high-precision time base unit and a signal calibration unit. The high-precision time base unit uses a GPS / BeiDou satellite navigation system receiver as an external clock source to generate a second pulse signal with a timing accuracy better than ±1 microsecond and a high-frequency clock signal, which are distributed to the entire system through a star topology or cascaded method. The signal calibration unit embeds the reference clock signal into the virtual engine vertical synchronization signal, the LED screen refresh synchronization signal, and the camera frame synchronization signal, binding precise timestamps under the global time base to the three types of image data respectively. The virtual engine image is marked with the rendering completion time and the vertical synchronization start time, the LED screen feedback image is marked with the actual refresh start time, and the camera acquisition image is marked with the exposure center time or the frame start time.

4. The end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films according to claim 1, characterized in that: The image synchronization analysis module includes a timing deviation analysis unit, a frame misalignment detection unit, a brightness change analysis unit, and a color shift analysis unit. The timing deviation analysis unit calculates the time offset between the rendering output and camera exposure, the screen refresh and camera exposure, and the rendering output and screen refresh, generating a timing deviation map. The frame misalignment detection unit uses a dual mechanism of hard matching of frame sequence numbers and soft matching of image features to identify dropped frames, duplicate frames, or misaligned frames, and outputs the frame matching rate and frame interval jitter value. The brightness change analysis unit, after spatial registration, quantifies brightness consistency through average brightness difference, histogram similarity, and peak signal-to-noise ratio, detecting periodic brightness fluctuations. The color shift analysis unit converts the image to CIELAB space, uses the CIE 2000 color difference formula to calculate pixel-level color difference, and statistically analyzes the average color difference, maximum color difference, and color difference standard deviation.

5. The end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films according to claim 1, characterized in that: The delay compensation optimization module includes a rendering queue adjustment unit, a transmission link compensation unit, an LED timing synchronization unit, and an exposure timing coordination unit. The rendering queue adjustment unit dynamically adjusts the virtual engine rendering queue depth and the number of pre-rendered frames based on timing deviations to align the rendering output with the camera's exposure window. The transmission link compensation unit measures the fixed delay in video transmission and corrects deviations through advance compensation at the sending end or delay compensation at the receiving end. The LED timing synchronization unit adjusts the screen refresh start phase and period to ensure that the effective display range covers the camera's exposure window. The exposure timing coordination unit controls the camera's exposure start time and duration through a synchronization signal generator.

6. The end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films according to claim 1, characterized in that: The display control module includes an LED display controller, a camera synchronization signal generator, and a synchronization scheduling unit. The LED display controller receives adjustment parameters and converts them into executable instructions for the driver chip, including vertical synchronization start offset, refresh cycle, and black frame insertion parameters. It performs adjustments in a gradual manner and provides feedback on the execution status. The camera synchronization signal generator generates frame synchronization and exposure control signals with timing accuracy better than microseconds and supports dynamic adjustment of exposure duration. The synchronization scheduling unit maintains a precise execution schedule to ensure that parameters are sent synchronously between the LED side and the camera side and has an execution conflict detection function.

7. The end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films according to claim 1, characterized in that: The synchronization evaluation module includes a synchronization accuracy scoring unit, a frame stability scoring unit, a picture consistency scoring unit, and a comprehensive scoring unit. The synchronization accuracy scoring unit takes the time offset between screen refresh and camera exposure as the core indicator, and combines timing deviation stability correction to output a score of 0-100. The frame stability scoring unit calculates a score based on the frame matching rate and frame interval jitter value, and deducts extra points for dropped frames and duplicate frames; the image consistency scoring unit adopts a weighted method of 0.4 for brightness consistency and 0.6 for color consistency. Brightness consistency is evaluated based on the average brightness difference and histogram similarity, and color consistency is quantified based on the average color difference.

8. The end-to-end image synchronization optimization and evaluation system for virtual shooting of LED films according to claim 1, characterized in that: The shooting data acquisition module has a built-in data preprocessing unit that performs YUV422 color space standardization and median filtering noise filtering on the acquired data; The timestamp calibration module adopts a dual identification format of UTC time and frame sequence number; the synchronous evaluation module supports the visualization output of evaluation results, including data reports, trend curves and non-conforming item markers, and stores comparison logs before and after optimization.

Images