Film and television production system based on deep learning reverse relighting algorithm
Through deep learning model combining Fourier and Hilbert transforms to evaluate the frequency and amplitude fidelity and phase retention capabilities of the illuminated signal, the problem of inaccurate illumination signal processing in the existing technology is solved, and efficient illumination capture and reconstruction are achieved , improves the effect and stability of film and television production.
Patent Information
- Application Number
- CN202411317904.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-09-20
AI Technical Summary
When existing deep learning models deal with periodic lighting signals in dynamic videos, they are prone to mistakenly mistakenly regarding noise or unexpected changes, resulting in the inability to capture the lighting mode correctly, resulting in incoherence of shadows and reflections, destroying the spatial reality and lighting reconstruction of the video. The effect of the algorithm.
Adopt the reverse lighting re-illumination algorithm based on deep learning, including data acquisition module, fidelity evaluation module, phase maintenance evaluation module, comprehensive analysis module, accuracy division module, model grading and application module and lighting sequence optimization module, through Fourier Transform and Hilbert transform evaluate the frequency and amplitude fidelity and phase retention capabilities of the illuminated signal, divide the model to capture the accuracy of the periodic characteristics of the illuminated light, and reduce mutations through periodic regular terms and illuminated sequence optimization.
It improves the overall accuracy and visual expression of light capture, enhances the effect and stability of light reconstruction in film and television production, ensures that the lighting effect is natural and realistic, and reduces mutations in shadows and reflections.
Smart Images
Figure CN118827894B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a film and television production system based on a deep learning reverse re-lighting algorithm. Background Art
[0002] In film and television production, reverse relighting technology extracts lighting information from images or videos and readjusts or generates new lighting effects to achieve background replacement or enhance visual effects. Although existing research has mostly focused on static images, reverse relighting of dynamic videos faces greater challenges, especially in maintaining temporal consistency and preventing inter-frame lighting inconsistencies. To solve this problem, researchers use deep learning models such as recurrent neural networks (RNNs) and long short-term memory networks (LSTMs) to process time series in videos to ensure smooth transitions of lighting over time.
[0003] However, in some video scenes, the lighting may show periodic changes, such as rotating light sources, flickering lights, or the day-night cycle in nature. When processing such periodic signals, RNN and LSTM models may mistake the periodicity of the lighting as noise or unexpected changes, and thus fail to correctly capture the periodic characteristics of the lighting pattern. At the same time, if the model cannot correctly capture the periodic changes of the light source, the dynamic performance of shadows and reflections may not match the actual lighting. Incoherent shadows and reflections will not only destroy the spatial realism of the video, but also clearly expose the defects of the lighting reconstruction algorithm. Summary of the invention
[0004] The purpose of the present invention is to provide a film and television production system based on a deep learning inverse re-lighting algorithm to address the shortcomings of the background technology.
[0005] In order to achieve the above-mentioned object, the present invention provides the following technical solutions: a film and television production system based on a deep learning reverse re-lighting algorithm, comprising a data acquisition module, a fidelity evaluation module, a phase preservation evaluation module, a comprehensive analysis module, an accuracy division module, a model classification and application module, and a lighting sequence optimization module;
[0006] Data acquisition module: collects video data containing periodic illumination features and splits it into frame sequences, extracts illumination features in each frame, inputs illumination features into a pre-built deep learning model, and outputs illumination sequences based on the model prediction results;
[0007] Fidelity evaluation module: Perform Fourier transform on the illumination sequence output by the deep learning model, and evaluate the frequency and amplitude fidelity of the model for the periodic illumination signal by comparing its frequency components;
[0008] Phase retention evaluation module: By comparing the periodic changes of the predicted illumination and the real illumination signals, the error accumulation phenomenon of the model in the fixed time series processing is detected, and by analyzing the phase drift, the phase retention ability of the model in the long time series is evaluated;
[0009] Comprehensive analysis module: Comprehensively analyze the frequency and amplitude fidelity of the model for periodic illumination signals and the model's phase retention ability in long time series to evaluate the accuracy of the deep learning model in capturing the periodic characteristics of illumination;
[0010] Accuracy classification module: Based on the evaluation results, the accuracy of the deep learning model in capturing the periodic characteristics of illumination is divided into different levels, namely high accuracy capture, general accuracy capture and low accuracy capture;
[0011] Model classification and application module: directly apply the deep learning model with high accuracy to the lighting reconstruction task in film and television production. For the deep learning model with low accuracy, periodic regularization terms are introduced to force the model to learn the pattern of periodic lighting signals.
[0012] Lighting sequence optimization module: For deep learning models with general accuracy capture, the degree of abnormal accuracy in capturing the periodic characteristics of illumination within a time period T is predicted. If the degree of abnormal accuracy is high, the lighting sequence is optimized to reduce mutations in shadows and reflections.
[0013] Preferably, in the fidelity evaluation module, after Fourier transforming the illumination sequence output by the deep learning model, the illumination sequence is converted from a time domain signal to a frequency domain signal, and the frequency component and amplitude information are extracted. The frequency and amplitude characteristics are compared with the real illumination signal to generate a main frequency offset anomaly index. The fidelity of the model in capturing periodic illumination characteristics is evaluated according to the main frequency offset anomaly index. The main frequency offset anomaly index is obtained as follows:
[0014] Extract the illumination sequence output by the deep learning model and the corresponding real illumination signal, convert them into frequency domain signals, perform Fourier transform on them respectively, convert the time domain signal into frequency domain signal, and obtain the model output spectrum and the real signal spectrum , respectively expressed as: ; ;in, represents the Fourier transform, and are the illumination intensities of the model output and the real illumination signal changing with time. In the Fourier transform results, the main frequency components of the model output spectrum and the real illumination signal spectrum are found, that is, the frequency corresponding to the maximum amplitude in the spectrum. The expression is: ; ;in, and are the main frequencies of the model output and the real signal respectively, and the offset between the main frequency of the model output illumination sequence and the main frequency of the real illumination signal is calculated. , the expression is: ; Calculate the main frequency deviation abnormality index, the expression is: ; In the formula, It is the main frequency deviation abnormality index.
[0015] Preferably, the acquired main frequency offset anomaly index is compared with a preset main frequency offset anomaly index reference threshold. If the main frequency offset anomaly index is greater than or equal to the preset main frequency offset anomaly index reference threshold, it means that the model has significant deviations in capturing periodic illumination characteristics, and a low-fidelity signal is generated at this time; if the main frequency offset anomaly index is less than the preset main frequency offset anomaly index reference threshold, it means that the model accurately captures the frequency of the periodic illumination signal, and a high-fidelity signal is generated at this time.
[0016] Preferably, in the phase retention evaluation module, phase drift refers to the relative shift in time of the illumination signal output by the model, that is, the phase of the illumination cycle is no longer synchronized with the real signal. After analyzing the phase synchronization deviation between the illumination signal and the real signal, a phase synchronization deviation index is generated to evaluate the phase retention ability of the model in a fixed time series. The method for obtaining the phase synchronization deviation index is:
[0017] Get the model output light signal in the S time period and the real light signal Take the illumination signal sequence, perform Hilbert transform on it, calculate its instantaneous phase, and calculate the analytical signal of the model output signal. The expression is: ;in, yes The Hilbert transform of the obtained analytical signal is in complex form. The analytical signal of the real illumination signal is calculated. ; Similarly, yes The Hilbert transform of the analytic signal , its instantaneous phase The calculation expression is: ;in, Represents the phase angle of the complex number, and calculates the instantaneous phase of the model output signal. The expression is: ; Calculate the instantaneous phase of the real illumination signal, the expression is: ; Calculate the phase shift between the model output signal and the real signal at different time points, that is, calculate the instantaneous phase difference , the calculation expression is: ; Calculate the phase synchronization deviation index , the expression is: .
[0018] Preferably, in the comprehensive analysis module, the frequency and amplitude fidelity of the model to the periodic illumination signal and the phase retention capability of the model in the long time series are comprehensively analyzed;
[0019] The main frequency offset anomaly index and the phase synchronization deviation index are normalized, and the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is calculated by the normalized main frequency offset anomaly index and the phase synchronization deviation index.
[0020] Preferably, in the accuracy classification module, the accuracy of the deep learning model in capturing the periodic characteristics of illumination is classified into different levels;
[0021] The accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is compared with the gradient standard threshold, the gradient standard threshold includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is compared with the first standard threshold and the second standard threshold respectively;
[0022] If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than the second standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is high, and a high-accuracy signal is generated at this time, which is classified as high-accuracy capture;
[0023] If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is at a medium level, and a medium accuracy signal is generated, which is classified as general accuracy capture;
[0024] If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is less than the first standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is low. At this time, a low-accuracy signal is generated and it is classified as low-accuracy capture.
[0025] Preferably, in the illumination sequence optimization module, for the deep learning model with general accuracy capture, the degree of abnormal accuracy of capturing the illumination periodicity characteristics within the T time period is predicted;
[0026] If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it is classified as general accuracy capture, and the accuracy coefficients of the deep learning model in capturing the periodic characteristics of illumination that are greater than or equal to the first standard threshold and less than or equal to the second standard threshold within the T time period are collected, and a corresponding data set is established; the deviation values of the accuracy coefficients of the deep learning model in the set in capturing the periodic characteristics of illumination and the first standard threshold are calculated respectively , the expression is: ; In the formula, The accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination, is the first standard threshold; calculate the cumulative accuracy abnormality in the entire time period T , the expression is: .
[0027] Preferably, if the cumulative accuracy is abnormal If the value is less than the predetermined abnormal threshold, the model captures the periodic characteristics of illumination in the T time period normally and does not need to be optimized. If L(t) is greater than or equal to the predetermined abnormal threshold, the model captures abnormalities in the illumination periodicity characteristics within the T time period and needs to be optimized: the illumination sequence is optimized by the smoothing function S(L(t)), and the expression is: ; L(t) is the original lighting sequence, S(L(t)) is the lighting smoothing function, is the optimized lighting sequence.
[0028] In the above technical solution, the technical effects and advantages provided by the present invention are:
[0029] 1. The present invention extracts periodic illumination feature data, uses a deep learning model to predict illumination sequences, combines Fourier transform to evaluate the fidelity of frequency and amplitude, and analyzes the stability of illumination capture of the model over a long period of time by analyzing phase drift. Through the accuracy classification module, the illumination capture capability of the model is divided into high, general, and low accuracy levels, so that it can be directly applied or further optimized according to the model performance.
[0030] 2. For high-accuracy capture models, the present invention can be directly used for lighting reconstruction tasks in film and television production to ensure natural and realistic lighting effects; for low-accuracy models, periodic regularization terms are used for enhanced training to improve their capture capabilities; and for general accuracy models, the system can reduce the sudden changes in shadows and reflections by optimizing the lighting sequence when abnormal lighting capture is detected. Through these technical means, the system not only improves the overall accuracy and visual expression of lighting capture, but also effectively enhances the effect and stability of lighting reconstruction in the process of film and television production. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0032] Figure 1 It is a system module diagram of the present invention. DETAILED DESCRIPTION
[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0034] For examples, see Figure 1 As shown, the film and television production system based on the deep learning reverse re-lighting algorithm described in this embodiment includes a data acquisition module, a fidelity evaluation module, a phase retention evaluation module, a comprehensive analysis module, an accuracy division module, a model classification and application module, and a lighting sequence optimization module;
[0035] Data acquisition module: collects video data containing periodic illumination features and splits it into frame sequences, extracts illumination features in each frame, inputs illumination features into a pre-built deep learning model, and outputs illumination sequences based on the model prediction results;
[0036] Fidelity evaluation module: Perform Fourier transform on the illumination sequence output by the deep learning model, and evaluate the frequency and amplitude fidelity of the model for the periodic illumination signal by comparing its frequency components;
[0037] Phase retention evaluation module: By comparing the periodic changes of the predicted illumination and the real illumination signals, the error accumulation phenomenon of the model in the fixed time series processing is detected, and by analyzing the phase drift, the phase retention ability of the model in the long time series is evaluated;
[0038] Comprehensive analysis module: Comprehensively analyze the frequency and amplitude fidelity of the model for periodic illumination signals and the model's phase retention ability in long time series to evaluate the accuracy of the deep learning model in capturing the periodic characteristics of illumination;
[0039] Accuracy classification module: Based on the evaluation results, the accuracy of the deep learning model in capturing the periodic characteristics of illumination is divided into different levels, namely high accuracy capture, general accuracy capture and low accuracy capture;
[0040] Model classification and application module: directly apply the deep learning model with high accuracy to the lighting reconstruction task in film and television production. For the deep learning model with low accuracy, periodic regularization terms are introduced to force the model to learn the pattern of periodic lighting signals.
[0041] Lighting sequence optimization module: For deep learning models with general accuracy capture, the degree of abnormal accuracy in capturing the periodic characteristics of illumination within a time period T is predicted. If the degree of abnormal accuracy is high, the lighting sequence is optimized to reduce mutations in shadows and reflections.
[0042] In the data acquisition module, video data containing periodic lighting features are collected from actual scenes, such as scenes with rotating light sources, flashing lights, or day-night cycles. These videos are captured using high-definition video equipment to ensure that the captured lighting changes are clear. High-definition video equipment can be: RED Digital Cinema Cameras, ARRI Alexa Mini LF, Sony FX6;
[0043] After the video is captured, it is sampled at a fixed frame rate (such as 30 FPS). Each frame of the video represents the lighting state at a point in time, forming a frame sequence for subsequent processing. The continuous video data is divided into separate frame sequences. Each frame corresponds to the scene information at a certain point in time in the video. Each frame is converted into a form suitable for input into the deep learning model. Usually, each frame is processed into a standard image format (such as an RGB image matrix), and the necessary size adjustment and format normalization are performed for subsequent feature extraction and input into the model.
[0044] For each frame, use image processing algorithms or pre-trained deep learning models to extract key lighting features. These features include the location, intensity, shadow shape, reflection area, etc. of the light source. These features can characterize the periodic changes in lighting in the video. The extracted lighting features are normalized to ensure that all data remain in the same numerical range. This normalization process helps improve the learning effect and prediction stability of the model.
[0045] Build a deep learning model based on a recurrent neural network (RNN) or a long short-term memory network (LSTM), or use a pre-trained model. The model is designed to capture lighting changes in a time series. Input the processed lighting feature sequence into the model. The model predicts the lighting conditions in the next frame based on the lighting features of the current frame. The model processes the frame sequence step by step to learn the rules of periodic changes. The deep learning model outputs a predicted lighting feature sequence frame by frame based on the input lighting features. This sequence reflects the periodic changes of the light source in the video, the dynamic performance of shadows, and the changes in reflections. Perform preliminary verification of the output lighting sequence to ensure that the predicted lighting features have a high degree of match with the actual lighting, especially the periodicity of the lighting, the performance of shadows and reflections.
[0046] In the fidelity assessment module, after Fourier transforming the illumination sequence output by the deep learning model, it is converted from a time domain signal to a frequency domain signal, and the main frequency components and amplitude information are extracted. These frequency and amplitude features are then compared with the real illumination signal to evaluate the fidelity of the model in capturing periodic illumination features. By analyzing the degree of match between the model output and the real signal in frequency, it can be determined whether the model accurately captures the periodic changes in illumination; at the same time, the comparison of amplitudes can measure whether the model correctly simulates the fluctuations in illumination intensity. If there is a significant deviation in frequency or amplitude, it means that there is a problem with the model's reconstruction of the periodic characteristics of illumination.
[0047] After Fourier transforming the illumination sequence output by the deep learning model, it is converted from a time domain signal to a frequency domain signal, and the main frequency components and amplitude information are extracted. The frequency and amplitude characteristics are compared with the real illumination signal to generate the main frequency offset anomaly index. The main frequency offset anomaly index is used to evaluate the fidelity of the model in capturing periodic illumination features. The main frequency offset anomaly index is obtained as follows:
[0048] Extract the illumination sequence output by the deep learning model and the corresponding real illumination signal, convert them into frequency domain signals, perform Fourier transform on the two illumination sequences (model output and real signal) respectively, convert the time domain signal into frequency domain signal, and obtain the model output spectrum and the real signal spectrum , respectively expressed as: ; ;in, represents the Fourier transform, and are the illumination intensities of the model output and the real illumination signal changing with time. In the Fourier transform results, the main frequency components of the model output spectrum and the real illumination signal spectrum are found, that is, the frequency corresponding to the maximum amplitude in the spectrum. The expression is: ; ;in, and are the main frequencies of the model output and the real signal respectively, and the offset between the main frequency of the model output illumination sequence and the main frequency of the real illumination signal is calculated. , the expression is: ; Calculate the main frequency deviation abnormality index, the expression is: ; In the formula, It is the main frequency deviation abnormality index.
[0049] Compare the obtained main frequency offset anomaly index with the preset main frequency offset anomaly index reference threshold. If the main frequency offset anomaly index is greater than or equal to the preset main frequency offset anomaly index reference threshold, it means that the model has significant deviations in capturing periodic illumination features and has low fidelity. At this time, a low-fidelity signal is generated, and the model has major problems in capturing the illumination cycle, and the main frequency offset is significant. Obvious illumination cycle disorder may appear in the video, resulting in discontinuous light and shadow and distorted reflections. If the main frequency offset anomaly index is less than the preset main frequency offset anomaly index reference threshold, it means that the model captures the frequency of the periodic illumination signal very accurately, indicating that the model output is almost consistent with the real signal and has extremely high fidelity. At this time, a high-fidelity signal is generated, and the model performs very well in capturing the main frequency of the illumination signal with very small errors. The illumination changes in the video will be very close to the real illumination changes, and the periodic performance (such as the dynamics of shadows and the periodicity of reflections) will appear natural and continuous.
[0050] It should be noted that the reference threshold of the main frequency deviation anomaly index can usually be set by experimental data or empirical values in practical applications. Different threshold ranges can reflect the accuracy of the model in capturing periodic illumination characteristics.
[0051] In the phase-holding evaluation module, the illumination sequence predicted by the model and the real illumination signal are Fourier transformed to extract their respective periodic frequency components. The periodic characteristics can be represented by the main frequency, the secondary frequency and their corresponding amplitudes. The periodic changes of the illumination signal output by the model and the real illumination signal are compared frame by frame to analyze the matching of the two in different time periods. By calculating the difference in periodic frequency and amplitude between the two, it is possible to detect whether there is a trend of long-term error accumulation.
[0052] Error accumulation means that as time goes by, the error of the model in predicting the illumination signal gradually increases, causing the periodic illumination features to gradually deviate from the real signal. In each time period, calculate the error between the illumination sequence output by the model and the real illumination signal, and detect whether the error continues to increase over time. If the error accumulates frame by frame, it means that the model has accumulated errors during the long-term processing, resulting in a decrease in the accuracy of capturing periodic illumination. Plot the error curve over time. If the curve shows a continuous upward trend, it can be judged that the model has error accumulation.
[0053] Phase drift refers to the relative shift in time of the illumination signal output by the model, that is, the phase of the illumination cycle is no longer synchronized with the real signal. After analyzing the phase synchronization deviation between the illumination signal and the real signal, a phase synchronization deviation index is generated to evaluate the phase retention ability of the model in a fixed time series. The method for obtaining the phase synchronization deviation index is as follows:
[0054] Get the model output light signal in the S time period and the real light signal The illumination signal sequence is used to perform Hilbert transform on the two illumination signals, calculate their instantaneous phases, and calculate the analytical signal of the model output signal. The expression is: ;in, yes The Hilbert transform of the obtained analytical signal is in complex form. The analytical signal of the real illumination signal is calculated. ; Similarly, yes The Hilbert transform of the analytic signal , its instantaneous phase The calculation expression is: ;in, Represents the phase angle of the complex number, and calculates the instantaneous phase of the model output signal. The expression is: ; Calculate the instantaneous phase of the real illumination signal, the expression is: ; Calculate the phase shift between the model output signal and the real signal at different time points, that is, calculate the instantaneous phase difference , the calculation expression is: ; Calculate the phase synchronization deviation index , the expression is: ; The smaller the phase synchronization deviation index is, the better the phase synchronization between the model output signal and the real signal is.
[0055] The larger the phase synchronization deviation index, the greater the phase difference between the illumination signal output by the model and the real signal, that is, the more significant the phase synchronization deviation. This means that the model has poor phase retention ability in a fixed time series and cannot accurately capture the periodic changes of the illumination signal, which may cause the illumination effect to be offset or distorted in a long time series.
[0056] The smaller the phase synchronization deviation index is, the smaller the phase difference between the illumination signal output by the model and the real signal is, and the model can better maintain phase synchronization. This means that the model has a strong phase retention ability in a fixed time series, can accurately capture and follow the periodic changes of the real illumination signal, and ensure the consistency and accuracy of the illumination effect in the time dimension.
[0057] Comprehensive analysis module: Comprehensively analyze the frequency and amplitude fidelity of the model for periodic light signals and the model's phase retention ability in long time series to evaluate the accuracy of the deep learning model in capturing the periodic characteristics of light.
[0058] The main frequency offset anomaly index and the phase synchronization deviation index are normalized, and the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is calculated by the normalized main frequency offset anomaly index and the phase synchronization deviation index.
[0059] For example, the present invention can use the following formula to calculate the accuracy coefficient of the deep learning model capturing the periodic characteristics of illumination, and the calculation expression is: ; In the formula, The accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination, is the main frequency deviation abnormality index, SA is the phase synchronization deviation index, is the proportional coefficient of the main frequency deviation anomaly index and the phase synchronization deviation index, and 0.
[0060] Accuracy division module: Based on the evaluation results, the accuracy of the deep learning model in capturing the periodic characteristics of illumination is divided into different levels, namely high accuracy capture, general accuracy capture, and low accuracy capture.
[0061] The accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is compared with the gradient standard threshold, the gradient standard threshold includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is compared with the first standard threshold and the second standard threshold respectively;
[0062] If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than the second standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is high, and a high-accuracy signal is generated at this time, which is classified as high-accuracy capture;
[0063] If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is at a medium level, and a medium accuracy signal is generated, which is classified as general accuracy capture;
[0064] If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is less than the first standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is low. At this time, a low-accuracy signal is generated and it is classified as low-accuracy capture.
[0065] Model classification and application module: The deep learning model with high accuracy is directly applied to the lighting reconstruction task in film and television production. For the deep learning model with low accuracy, a periodic regularization term is introduced to force the model to learn the pattern of periodic lighting signals.
[0066] In practical applications, deep learning models that capture with high accuracy can be directly applied to lighting reconstruction tasks in film and television production because they can accurately capture the periodic characteristics of lighting signals. This model performs well in reproducing complex lighting scenes, especially for dynamically changing details such as shadows and reflections. It can maintain high authenticity and coherence, so it is suitable for scene production with high lighting requirements without further optimization.
[0067] For deep learning models with low accuracy capture, the model shows obvious deviations when capturing periodic illumination signals, resulting in inaccurate and discontinuous illumination effects in the time series. In order to improve its accuracy, a periodic regularization term can be introduced. The periodic regularization term forces the model to learn the periodic variation pattern of illumination signals by adding constraints on periodic signals during model training. In this way, the model can better understand the dynamic features related to periodic illumination, such as light source position, intensity, shadows, and reflections, thereby improving its ability to capture periodic illumination signals and reducing deviations.
[0068] Lighting sequence optimization module: For deep learning models with general accuracy capture, the degree of abnormal accuracy in capturing the periodic characteristics of illumination within a time period T is predicted. If the degree of abnormal accuracy is high, the lighting sequence is optimized to reduce mutations in shadows and reflections.
[0069] If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it is classified as general accuracy capture, and the accuracy coefficients of the deep learning model in capturing the periodic characteristics of illumination that are greater than or equal to the first standard threshold and less than or equal to the second standard threshold in the T time period are collected, and the corresponding data set is established. Calculate the deviation values of the accuracy coefficients of the deep learning model in the set in capturing the periodic characteristics of illumination and the first standard threshold respectively , the expression is: ; In the formula, The accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination, is the first standard threshold; calculate the cumulative accuracy abnormality in the entire time period T , the expression is: ;
[0070] If the cumulative accuracy is abnormal If it is less than the preset abnormal threshold, it is considered that the model captures the periodic characteristics of illumination in the T time period normally and no optimization is required;
[0071] If the cumulative accuracy is abnormal If it is greater than or equal to the predetermined abnormal threshold, it is considered that the illumination periodicity feature of the model in the T time period captures abnormalities and needs to be optimized: the illumination sequence is optimized by the smoothing function S(L(t)), and the expression is: ; L(t) is the original lighting sequence, S(L(t)) is the lighting smoothing function, which is used to reduce mutations in the sequence and ensure a more natural transition between shadows and reflections. The optimized lighting sequence reduces mutations and ensures smooth and coherent visual effects.
[0072] It should be noted here that the deviation values of the accuracy coefficients of the deep learning model in the calculation set in capturing the periodic characteristics of illumination and the first standard threshold are respectively calculated because when the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is less than the first standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is low, which means that the deep learning model is capturing with low accuracy, and the greater the possibility of accuracy abnormality.
[0073] In this embodiment, the video data containing periodic illumination features is obtained by the data acquisition module and split into frame sequences, the illumination features are extracted and input into the deep learning model for prediction, and the illumination sequence is output. The fidelity evaluation module compares the frequency and amplitude of the model output with the real signal through Fourier transform to evaluate its fidelity. The phase retention evaluation module detects the error accumulation and phase drift of the model in a fixed time series and evaluates its ability to capture the illumination cycle for a long time. The comprehensive analysis module combines the frequency, amplitude and phase retention capabilities to comprehensively evaluate the accuracy of the model in capturing periodic illumination features. According to the evaluation results, the accuracy classification module divides the model into three categories: high, medium and low, among which the high-accuracy model can be directly applied to film and television illumination reconstruction, and the low-accuracy model is strengthened by introducing a periodic regularization term. The illumination sequence optimization module monitors the capture anomalies of the general accuracy model within the time period. If the degree of anomaly is high, the illumination sequence is optimized to reduce shadow and reflection mutations.
[0074] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.
[0075] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.
[0076] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A film and television production system based on a deep learning reverse relighting algorithm, characterized by: It includes data acquisition module, fidelity assessment module, phase retention assessment module, comprehensive analysis module, accuracy classification module, model classification and application module and illumination sequence optimization module; Data acquisition module: collects video data containing periodic illumination features and splits it into frame sequences, extracts illumination features in each frame, inputs illumination features into a pre-built deep learning model, and outputs illumination sequences based on the model prediction results; Fidelity evaluation module: Perform Fourier transform on the illumination sequence output by the deep learning model, and evaluate the frequency and amplitude fidelity of the model for the periodic illumination signal by comparing its frequency components; The fidelity evaluation module specifically includes: after Fourier transforming the illumination sequence output by the deep learning model, converting it from a time domain signal to a frequency domain signal, extracting the frequency component and amplitude information, and generating a main frequency offset anomaly index after comparing the frequency and amplitude characteristics with the real illumination signal. The fidelity of the model in capturing periodic illumination characteristics is evaluated according to the main frequency offset anomaly index. The main frequency offset anomaly index is obtained as follows: Extract the illumination sequence output by the deep learning model and the corresponding real illumination signal, convert them into frequency domain signals, perform Fourier transform on them respectively, convert the time domain signal into frequency domain signal, and obtain the model output spectrum and the real signal spectrum , respectively expressed as: ; ;in, represents the Fourier transform, and are the illumination intensities of the model output and the real illumination signal changing with time. In the Fourier transform results, the main frequency components of the model output spectrum and the real illumination signal spectrum are found, that is, the frequency corresponding to the maximum amplitude in the spectrum. The expression is: ; ;in, and are the main frequencies of the model output and the real signal respectively, and the offset between the main frequency of the model output illumination sequence and the main frequency of the real illumination signal is calculated. , the expression is: ; Calculate the main frequency deviation abnormality index, the expression is: ; In the formula, is the main frequency deviation abnormality index; Phase retention evaluation module: By comparing the periodic changes of the predicted illumination and the real illumination signals, the error accumulation phenomenon of the model in the fixed time series processing is detected, and by analyzing the phase drift, the phase retention ability of the model in the long time series is evaluated; The phase retention evaluation module specifically includes: phase drift refers to the relative shift in time of the illumination signal output by the model, that is, the phase of the illumination cycle is no longer synchronized with the real signal, and the phase synchronization deviation between the illumination signal and the real signal is analyzed to generate a phase synchronization deviation index, and the phase retention ability of the model in a fixed time series is evaluated. The method for obtaining the phase synchronization deviation index is: Get the model output light signal in the S time period and the real light signal Take the illumination signal sequence, perform Hilbert transform on it, calculate its instantaneous phase, and calculate the analytical signal of the model output signal. The expression is: ;in, yes The Hilbert transform of the obtained analytical signal is in complex form. The analytical signal of the real illumination signal is calculated. ; Similarly, yes The Hilbert transform of the analytic signal , its instantaneous phase The calculation expression is: ;in, Represents the phase angle of the complex number, and calculates the instantaneous phase of the model output signal. The expression is: ; Calculate the instantaneous phase of the real illumination signal, the expression is: ; Calculate the phase shift between the model output signal and the real signal at different time points, that is, calculate the instantaneous phase difference , the calculation expression is: ; Calculate the phase synchronization deviation index , the expression is: ; Comprehensive analysis module: Comprehensively analyze the frequency and amplitude fidelity of the model for periodic illumination signals and the model's phase retention ability in long time series to evaluate the accuracy of the deep learning model in capturing the periodic characteristics of illumination; Accuracy classification module: Based on the evaluation results, the accuracy of the deep learning model in capturing the periodic characteristics of illumination is divided into different levels, namely high accuracy capture, general accuracy capture and low accuracy capture; Model classification and application module: directly apply the deep learning model with high accuracy to the lighting reconstruction task in film and television production. For the deep learning model with low accuracy, periodic regularization terms are introduced to force the model to learn the pattern of periodic lighting signals. Lighting sequence optimization module: For deep learning models with general accuracy capture, the degree of abnormal accuracy in capturing the periodic characteristics of illumination within a time period T is predicted. If the degree of abnormal accuracy is high, the lighting sequence is optimized to reduce mutations in shadows and reflections.
2. The film and television production system based on the deep learning reverse relighting algorithm according to claim 1, characterized in that: The obtained main frequency offset anomaly index is compared with the preset main frequency offset anomaly index reference threshold. If the main frequency offset anomaly index is greater than or equal to the preset main frequency offset anomaly index reference threshold, it means that the model has significant deviations in capturing periodic illumination characteristics, and a low-fidelity signal is generated at this time; if the main frequency offset anomaly index is less than the preset main frequency offset anomaly index reference threshold, it means that the model accurately captures the frequency of the periodic illumination signal, and a high-fidelity signal is generated at this time.
3. The film and television production system based on the deep learning reverse relighting algorithm according to claim 1, characterized in that: In the comprehensive analysis module, the frequency and amplitude fidelity of the model to the periodic illumination signal and the phase retention ability of the model in the long time series are comprehensively analyzed; The main frequency offset anomaly index and the phase synchronization deviation index are normalized, and the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is calculated by the normalized main frequency offset anomaly index and the phase synchronization deviation index.
4. The film and television production system based on the deep learning reverse relighting algorithm according to claim 3 is characterized in that: In the accuracy classification module, the accuracy of the deep learning model in capturing the periodic characteristics of illumination is divided into different levels; The accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is compared with the gradient standard threshold, the gradient standard threshold includes a first standard threshold and a second standard threshold, and the first standard threshold is less than the second standard threshold, and the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is compared with the first standard threshold and the second standard threshold respectively; If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than the second standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is high, and a high-accuracy signal is generated at this time, which is classified as high-accuracy capture; If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is at a medium level, and a medium accuracy signal is generated, which is classified as general accuracy capture; If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is less than the first standard threshold, it means that the accuracy of the deep learning model in capturing the periodic characteristics of illumination is low. At this time, a low-accuracy signal is generated and it is classified as low-accuracy capture.
5. The film and television production system based on the deep learning reverse relighting algorithm according to claim 1, characterized in that: In the illumination sequence optimization module, for the deep learning model with general accuracy capture, the degree of abnormal accuracy in capturing the periodic characteristics of illumination within the T time period is predicted; If the accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination is greater than or equal to the first standard threshold and less than or equal to the second standard threshold, it is classified as general accuracy capture, and the accuracy coefficients of the deep learning model in capturing the periodic characteristics of illumination that are greater than or equal to the first standard threshold and less than or equal to the second standard threshold within the T time period are collected, and a corresponding data set is established; the deviation values of the accuracy coefficients of the deep learning model in the set in capturing the periodic characteristics of illumination and the first standard threshold are calculated respectively , the expression is: ; In the formula, The accuracy coefficient of the deep learning model in capturing the periodic characteristics of illumination, is the first standard threshold; Calculate the cumulative accuracy anomaly over the entire time period T , the expression is: .
6. The film and television production system based on the deep learning reverse relighting algorithm according to claim 5, characterized in that: If the cumulative accuracy is abnormal If the value is less than the predetermined abnormal threshold, the model captures the periodic characteristics of illumination in the T time period normally and does not need to be optimized. If L(t) is greater than or equal to the predetermined abnormal threshold, the model captures abnormalities in the illumination periodicity characteristics within the T time period and needs to be optimized: the illumination sequence is optimized by the smoothing function S(L(t)), and the expression is: ; L(t) is the original lighting sequence, S(L(t)) is the lighting smoothing function, is the optimized lighting sequence.