Method and system for generating visual effects video

Abstract

This invention discloses a method and system for generating visual special effects videos, relating to the field of video generation technology. The invention includes: multimodal data acquisition and preprocessing, synchronously acquiring RGB video, depth maps, and ambient lighting data to generate a registered 3D point cloud; spatiotemporal joint motion compensation keying to extract alpha masks of moving objects; adaptive illumination separation edge cleanup to eliminate light pollution and edge color bleeding in green screen scenes; a multi-scale transparency estimation network to output a transparency map; dynamic special effects synthesis and output to ensure video smoothness; and intelligent recommendation of special effects videos to implement an adaptive output strategy. This invention improves keying and edge processing accuracy, enhances transparency estimation and special effects synthesis, implements intelligent recommendation and resource optimization, and addresses the shortcomings of existing technologies in dynamic scene processing.

Description

Technical Field

[0001] The present invention relates to the technical field of video generation, and in particular to a method and system for generating visual special effects videos. Background Art

[0002] With the development of digital media technology, visual effects videos are increasingly used in film production, advertising media, live broadcasting and other fields. Such technologies usually require the efficient integration of virtual elements with real scenes to create realistic visual effects. However, existing technologies face multiple challenges when processing dynamic scenes: for example, traditional matting methods have difficulty in accurately extracting the edges of fast-moving objects, resulting in edge jitter or jagged edges in the synthesized video; the synchronous processing and spatial registration accuracy of multimodal data (such as RGB video, depth map, and ambient lighting) is insufficient, affecting the spatial consistency of special effects synthesis; in addition, light pollution in green screen shooting (such as edge color bleeding) and the simulation of the optical properties of translucent objects (such as glass and smoke) also make it difficult for the realism of existing special effects videos to meet high-quality image requirements.

[0003] In terms of keying and edge processing, traditional algorithms rely on single-modal data (such as RGB images only) for target segmentation, lack dynamic compensation for motion trajectories, and are unable to adapt to the sub-pixel precision required in complex motion scenes. In the lighting processing stage, the edge color bleeding caused by green screen reflections has not been effectively solved. Existing methods have difficulty combining material properties (such as diffuse reflection and metal surfaces) for differentiated purification, resulting in significant color distortion at the synthesized edges. Regarding transparency estimation, traditional models lack physical parameter constraints and cannot accurately simulate the light absorption and scattering characteristics of translucent objects. This is especially true when rendering dynamic transparent objects such as smoke and liquids, where optical distortion often occurs.

[0004] When synthesizing dynamic special effects, existing technologies fail to adequately optimize temporal consistency. Inter-frame lighting changes and motion blur can easily cause video freezes or flickering, impacting the viewing experience. Furthermore, when it comes to outputting special effects videos, traditional systems lack the ability to assess real-time performance, image quality, and resource consumption in multiple dimensions. They are unable to adaptively adjust output parameters based on scenario requirements (such as live streaming or film production), resulting in wasted resources or insufficient performance. For example, live streaming scenarios are sensitive to latency, while film production prioritizes image quality. However, existing solutions struggle to achieve the optimal balance across different scenarios. There is an urgent need for a special effects generation system that integrates multi-dimensional optimization and intelligent recommendation.

[0005] In order to solve the above-mentioned defects, a technical solution is now provided. Summary of the Invention

[0006] The purpose of the present invention is to solve the existing problems and to provide a method and system for generating visual effects videos.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A method for generating a visual effects video comprises the following steps:

[0009] S1. Multimodal data acquisition and preprocessing: Synchronously collect RGB video, depth map, and ambient lighting data, and generate a registered 3D point cloud through temporal alignment and IMU motion compensation, providing a spatial basis for special effects synthesis;

[0010] S2, spatiotemporal joint motion compensation matting: Alpha mask extraction of moving objects based on target contour displacement prediction and bidirectional mask optimization;

[0011] S3, Edge Cleanup with Adaptive Illumination Separation: This integrates polarization data with colorimetric analysis to build a material-aware pollution weight model, eliminating light pollution and edge color bleeding in green screen scenes.

[0012] S4, Multi-scale Transparency Estimation Network: Through U-Net, it fuses depth, motion mask and material features, combines physical parameter constraints to output a transparency map, and simulates the optical properties of translucent objects;

[0013] S5, dynamic special effects synthesis and output: performs virtual lighting matching, foreground-background fusion and bilateral filtering noise reduction, and applies a temporal consistency loss function to ensure video smoothness;

[0014] S6, intelligent recommendation of special effects videos, based on the three-dimensional evaluation matrix of real-time performance, image quality, and resource consumption, combined with the weighted calculation of scene templates to achieve an adaptive output strategy.

[0015] Furthermore, the specific process of S1 is as follows:

[0016] Perform time domain alignment on RGB video streams and achieve inter-frame synchronization by matching feature points of adjacent frames;

[0017] The IMU angular velocity data is integrated to calculate the rotation compensation matrix, and the improved optical flow algorithm is used to perform motion compensation.

[0018] The depth map and RGB pixels are aligned through the calibration matrix to generate a pixel-by-pixel world coordinate point cloud.

[0019] Furthermore, the specific operation steps of S2 are as follows:

[0020] Based on the instance segmentation results of the target contour, the displacement vector field between consecutive frames is extracted;

[0021] Generate the prediction mask of the current frame through forward prediction, and optimize the mask accuracy by combining backward correction;

[0022] Iteratively adjust the displacement vector and calculate the mask residual to dynamically optimize the motion compensation effect;

[0023] Perform sub-pixel upsampling and edge detection on the contour edge area and output a high-resolution alpha mask.

[0024] Furthermore, the specific operation steps of S3 are as follows:

[0025] Build a light pollution model based on ambient light data to analyze light interference in green screen scenes;

[0026] Convert the RGB image to LAB color space, separate and quantify the pollution degree of the green component;

[0027] Combined with polarization data, material differences are calculated and a material weight map is constructed to distinguish different surface characteristics.

[0028] Perform dual-channel purification on the edge area of ​​the Alpha mask, using differentiated purification strategies for the diffuse reflection area and the metal or glass area respectively, and output the optimized edge mask.

[0029] Furthermore, the specific operation steps of S4 are as follows:

[0030] Build a transparency estimation network based on U-Net, which takes as input RGB image patches, depth differences, motion masks, and material classification maps;

[0031] In the decoding stage, multi-scale features are integrated, including edge details at the original resolution, texture structure at 1 / 2 resolution, and semantic context at 1 / 4 resolution.

[0032] Introducing a material-aware attention mechanism to enhance transparency prediction in feature regions;

[0033] The physical parameters are queried through the differentiable material library and the final transparency map of the physical constraints is calculated in combination with the depth information.

[0034] Furthermore, the specific operation steps of S5 are as follows:

[0035] Generate a virtual background based on the depth map and match the ambient lighting, and calculate the lighting compensation value of the foreground and virtual light source;

[0036] The foreground and background are fused according to the final transparency map, and bilateral filtering is performed on the edge transition area;

[0037] Apply inter-frame consistency loss function and illumination flow constraint to optimize video temporal coherence;

[0038] Output special effects video with lighting correction and time domain optimization.

[0039] Furthermore, the specific operation steps of S6 are as follows:

[0040] S601, generating multiple optimized versions of special effects videos according to the input scene, including a high-speed version, a quality version, and a balanced version, each version having a different resolution, frame rate, and optimized parameter configuration;

[0041] S602: Establish a multi-dimensional evaluation matrix including real-time performance, image quality, resource consumption, and special effect realism, and obtain quantitative indicators of each dimension through system timers, OpenCV quality assessment tools, and hardware monitoring APIs;

[0042] S603: Matching user demand characteristics based on a preset scene template library. The scene templates include four types: real-time live broadcast, movie production, mobile terminal publishing, and advertising special effects. Each type corresponds to a different weight vector.

[0043] S604: After standardizing the evaluation indicators, the comprehensive scores of each version are calculated based on the scenario weights, and the recommendation probability distribution is generated through exponential transformation;

[0044] S605: Finally, the special effects video version with the highest recommendation probability is output, and the user selection result is recorded for weight optimization.

[0045] A system for generating a visual effects video, comprising:

[0046] The data acquisition and processing module synchronously collects RGB video streams, depth map streams, and ambient lighting data, and generates a registered 3D point cloud through IMU motion compensation and temporal alignment, providing accurate spatial basic data for special effects synthesis.

[0047] The dynamic keying and edge optimization module extracts target contours based on Mask R-CNN, generates sub-pixel alpha masks through bidirectional motion compensation and iterative displacement vector optimization, and combines polarization data to achieve material-aware edge cleanup and eliminate light pollution.

[0048] The transparency physical modeling module uses a multi-scale U-Net to fuse depth maps, motion masks, and material features, queries physical parameters through a differentiable material library, and outputs a physically constrained transparency map to simulate the optical properties of translucent objects.

[0049] The dynamic synthesis and temporal optimization module performs virtual illumination matching, foreground-background fusion, and bilateral filtering noise reduction. It applies inter-frame consistency loss and light flow constraints to ensure video smoothness and suppress edge haloing.

[0050] The recommendation and output module generates multiple versions of special effects videos. Based on the four-dimensional evaluation matrix of real-time performance, image quality, resource consumption, and realism, it combines scene templates to weightedly calculate the optimal version and implement an adaptive output strategy.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] (1) The present invention, through the spatiotemporal domain joint motion compensation matting technology, based on target contour displacement prediction and bidirectional mask optimization, can achieve high-precision extraction of the alpha mask of the moving object, solve the problems of edge jitter and aliasing in dynamic scenes caused by traditional matting methods, and effectively improve the quality and realism of the synthesized video; at the same time, the edge purification step of adaptive illumination separation, the integration of polarization data and chromaticity analysis to construct a material-aware pollution weight model, can specifically eliminate the light pollution and edge color bleeding of the green screen scene, further optimize the edge processing effect, make the synthesized picture more natural and realistic, and meet the needs of film production, advertising media, etc. for high-quality visual effects video;

[0053] (2) The present invention uses a multi-scale transparency estimation network and outputs a transparency map with physical parameter constraints. This can simulate the optical properties of translucent objects and solve the problem of optical property distortion in traditional models when rendering dynamic transparent objects such as smoke and liquid, making the special effects synthesis more realistic and delicate. During the dynamic special effects synthesis and output process, virtual lighting matching, foreground-background fusion, and bilateral filtering noise reduction are performed, and a temporal consistency loss function is applied to ensure video smoothness, effectively suppressing the freeze or flicker caused by inter-frame lighting changes and motion blur, and improving the viewing experience. It can be widely used in various scenarios such as real-time live broadcast and film production;

[0054] (3) In the present invention, the intelligent recommendation system for special effects videos calculates the optimal version based on the three-dimensional evaluation matrix of real-time performance, image quality, and resource consumption, combined with the weighted calculation of scene templates, to achieve an adaptive output strategy; for different scene requirements such as real-time live broadcast, film production, mobile terminal publishing, and advertising special effects, the system automatically generates and recommends the most suitable special effects video version, avoiding resource waste or insufficient performance, improving the overall efficiency and practicality of the system, and providing users with valuable and personalized visual special effects video generation solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0056] Figure 1 This is the overall system block diagram of the present invention. DETAILED DESCRIPTION

[0057] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0058] It should be understood that the terms “include” and “comprising” used in the specification and claims of the present disclosure indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0059] It should also be understood that the terminology used in this disclosure is for the purpose of describing specific embodiments only and is not intended to limit the disclosure. As used in this disclosure and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should be further understood that the term "and / or" as used in this disclosure and the claims refers to any and all possible combinations of one or more of the associated listed items, including and including these combinations.

[0060] like Figure 1 As shown, a method for generating a visual effects video includes the following steps:

[0061] Step 1: Multimodal data acquisition and preprocessing: synchronously collect RGB video, depth map, and ambient lighting data. Generate a registered 3D point cloud through temporal alignment and IMU motion compensation, providing a spatial foundation for special effects synthesis.

[0062] Synchronously acquire the following data streams: RGB video stream (main camera), depth map stream (TOF depth camera), and high dynamic range illumination data (ambient light sensor); perform temporal alignment on the RGB streams: calculate SIFT feature point matches between adjacent frames;

[0063] Motion compensation with IMU data: real-time reading of IMU angular velocity , respectively, represent the angular velocity components of the IMU on the x, y, and z axes; integral calculation of the rotation component: , i is the coordinate axis, ( ),in is the angular velocity time domain signal of the IMU (inertial measurement unit); construct the rotation compensation matrix: ; are the rotation components around the x, y, and z axes respectively;

[0064] Perform IMU-enhanced optical flow compensation: using the improved Farneback dense optical flow algorithm: ,in is the image gradient (vector), v is the optical flow vector, I is the image brightness, is the change of the image over time, x is the homogeneous coordinate, and Id is the unit matrix;

[0065] Construct a 3D spatial mapping: align the depth map with the RGB pixels through a calibration matrix; generate a pixel-by-pixel world coordinate point cloud.

[0066] Step 2: Joint spatiotemporal motion compensation keying: Based on target contour displacement prediction and bidirectional mask optimization, sub-pixel precision alpha mask extraction of moving objects is achieved, solving the problem of edge jitter in dynamic keying.

[0067] Motion trajectory prediction: Extract the target body contour (based on Mask R-CNN instance segmentation) and calculate the displacement vector field of the contour points within three consecutive frames;

[0068] Bidirectional motion compensation: Forward prediction: Generate the prediction mask of frame t based on the displacement vector of frame t-1;

[0069] Backward correction: use the actual edge of frame t to correct the mask of frame t-1;

[0070] Iterative optimization: Calculate the mask residual: , where E is the mask residual, is the XOR operation, are the predicted mask and actual mask of the t-th frame respectively;

[0071] Dynamically adjust the displacement vector: ,in is the adjustment coefficient (controlling the dynamic response amplitude), is the gain coefficient (amplifying the residual effect), Respectively, the displacement vector after adjustment and the displacement vector before adjustment; re-run forward prediction: use Regenerate optimized mask ;

[0072] Sub-pixel edge optimization: perform 4× upsampling on the edge area of ​​the contour, apply the Laplacian operator to detect the sub-pixel edge gradient; input the optimized mask , output motion compensated alpha mask (resolution increased to 200% of the original image).

[0073] Step 3: Adaptive illumination separation and edge cleanup: By integrating polarization data with colorimetric analysis, a material-aware pollution weight model is constructed to specifically eliminate light pollution and edge color bleeding in green screen scenes.

[0074] Light pollution modeling: Constructing a light function based on ambient light sensor data: ,in is the light intensity at the (x,y) position, is the ambient light intensity, For the green screen reflected light, is the weighting coefficient of ambient light and reflected light;

[0075] Color space decomposition: Convert RGB to LAB color space and separate the green component pollution value in the B channel: ,in, is the degree of green pollution (the larger the value, the more serious the pollution), B is the B channel value in the LAB color space, L is the L channel value (brightness) in the LAB color space, a and b are the linear coefficients of the green screen reference color;

[0076] Material channel separation: Calculate polarization difference (reuse polarization data from step 1): ,in The polarization angle is The light intensity at that time, The polarization direction is The difference in time, ;

[0077] Build the material weight map: ; is the material weight vector (probability distribution of each polarization direction);

[0078] Edge cleaning: 5px width area at the edge of the mask: calculate the pollution weight , where W is the comprehensive pollution weight (used for edge purification), is the sigmoid function (mapping the input to ((0,1)), : are the pollution value amplification coefficient and material weight influence coefficient respectively;

[0079] Dual-channel purification operation: diffuse reflection area: ;Metal / Glass Area: ; Fusion output: Where is the original alpha value, is the predicted alpha value, is the depth-based transparency estimation.

[0080] Step 4: Multi-scale transparency estimation network, which fuses depth, motion mask and material features through U-Net and outputs a transparency map based on physical parameter constraints to accurately simulate the optical properties of translucent objects;

[0081] Construct a transparency estimation model: Network structure: U-Net with Attention Gates; Input data: RGB image block (64×64px); Depth map difference (distance between current point and background plane); Alpha mask after motion compensation; Material classification map (in step 3) Data); Multi-scale feature fusion: fuse three scale features at the decoder stage: original resolution: edge details; 1 / 2 resolution: texture structure; 1 / 4 resolution: semantic context;

[0082] At the same time, add material-aware attention: inject material information into AttentionGates: , where Attention is the attention weight map (enhancing key features), is the learnable weight matrix, are encoder features and decoder features respectively, M is the material feature map, For the sigmoid function, generate material weighted features: , is the material enhancement coefficient, smoke is 0.8, glass is 0.5; is element-wise multiplication;

[0083] Physical constraint output:

[0084] Differentiable material library query:

[0085] Construct the physical parameter matrix: The physical parameter matrix K is defined as: , where K is the physical parameter matrix (storing the optical parameters of each material), is the absorption coefficient of material i (the larger the value, the stronger the light absorption), is the scattering index of material i (characterizing the light scattering properties), and n is the total number of preset material categories; the initial value is based on physical measurements;

[0086] Material-related parameter prediction: predict the physical parameters of the current pixel through the material classification map M: ,in is the predicted absorption coefficient and scattering index of the current pixel, is the material feature vector centered at pixel p; MLP is a three-layer connection network;

[0087] Final transparency map calculation: ;Physical constraint function: Physical constraint =1-exp(-β×d tr ),in is the final transparency map, β is the query value of the material absorption coefficient library, d is the thickness of the object in the depth map, and ty is the material scattering index;

[0088] The gradient propagation mechanism is the back propagation derivative formula: .

[0089] Step 5: Dynamic special effects synthesis and output, performing virtual lighting matching, foreground-background fusion, and bilateral filtering noise reduction, and applying a temporal consistency loss function to ensure video smoothness;

[0090] Background virtualization rendering: Use depth maps to generate parallax scrolling backgrounds and adjust virtual light intensity based on ambient light data; deep lighting matching, including:

[0091] Extracting surface lighting of foreground objects: ;in is the surface illumination intensity of the foreground object at position (x, y), The intensity of the c channel (R, G, B) at the (x, y) position is summed for the three color channels (R, G, B), and then the average is taken (divided by 3); the virtual lighting compensation is calculated by the difference between the ambient light and the virtual light: ; Generate lighting compensation map: ,in : attenuation coefficient, is the distance to the foreground edge, is the final illumination compensation value, is an exponential term, as the distance from the foreground edge increases ( increases), the compensation amount decays exponentially;

[0092] Foreground-background fusion:

[0093] Lighting correction fusion formula: ;in is the final synthesized output image (RGB pixel value), is the color value of the foreground image (RGB vector), is the color value of the virtual background image (RGB vector);

[0094] Edge halo suppression:

[0095] Detect edge transitions: ;in is the pixel set of the edge transition area, is the transparency value at position (x, y);

[0096] Apply bilateral filtering: ,in is the output color after filtering (at pixel p), is the normalization factor, ensuring that the weights sum to 1, is the color value of the input image at pixel q, p and q are the coordinates of the pixel currently being processed and the pixel coordinates in the neighborhood Ω, respectively. is the Gaussian kernel function in the spatial domain, is the range Gaussian kernel function, is the Euclidean distance between pixels p and q, is the absolute difference in transparency between pixels p and q;

[0097] Temporal consistency enhancement: Applying an inter-frame consistency loss function to the synthesized video: in, is the synthetic video frame at time point t, is the synthetic video frame at time point t-1, To use the optical flow field to transform the previous frame The image is warped to the coordinate system of the current frame t. represents the square of the Euclidean norm;

[0098] Light flow constraints:

[0099] Calculate the illumination change field: ;in is the illumination map at time point t, To use the same optical flow field, the illumination map of the previous frame Warped to the coordinate system of the current frame t, is the illumination variation field;

[0100] Extended loss function: ,in is the total loss function, is the L1 norm (sum of absolute values); , is the lighting smoothing weight.

[0101] Step 6: Intelligently recommend special effects videos based on a three-dimensional evaluation matrix of real-time performance, image quality, and resource consumption, and weighted calculation of the optimal version in combination with scene templates to implement an adaptive output strategy.

[0102] Multi-version video generation:

[0103] Generate three optimized versions for the same input scene:

[0104] Version A (High-speed): Resolution: 1280×720; Disable translucent object optimization; Frame rate: 60fps;

[0105] Version B (quality): resolution: 3840×2160; all edge optimizations enabled; color depth: 10-bit;

[0106] Version C (Balanced): Resolution: 1920×1080; Core optimization enabled; Bit rate: 25Mbps;

[0107] Multidimensional parameter analysis:

[0108] The quantitative evaluation matrix is ​​established as shown in Table 1:

[0109] ;

[0110] In the formula in Table 1: is the structural similarity index, is the peak signal-to-noise ratio;

[0111] Output evaluation matrix: ;in They are respectively the real-time performance, image quality, resource consumption and realism of special effects of version i;

[0112] User demand modeling: The preset scenario template is shown in Table 2:

[0113] ;

[0114] Priority calculation: Normalization: ;in To standardize the evaluation matrix, For Min-Max normalization, the index is normalized to the range of [0,1];

[0115] Weighted score calculation: ;in is the weighted score of version i, i.e., the comprehensive evaluation value; is the scene weight coefficient;

[0116] Final priority: ;in is the recommendation probability of version i, is the exponential transformation of the score, is the sum of the index scores; output probability distribution: ,in ; Recommend the version with the highest probability and record the user's final choice for weight optimization.

[0117] A visual effects video generation system includes a data acquisition and processing module, a dynamic keying and edge optimization module, a transparency physical modeling module, a dynamic synthesis and time domain optimization module, and a recommendation and output module;

[0118] The data acquisition and processing module synchronously collects RGB video streams, depth map streams, and ambient lighting data, and generates a registered 3D point cloud through IMU motion compensation and temporal alignment, providing accurate spatial basic data for special effects synthesis.

[0119] The rotoscoping and edge optimization module extracts target contours based on Mask R-CNN, generates sub-pixel alpha masks through bidirectional motion compensation and iterative displacement vector optimization, and combines polarization data to achieve material-aware edge cleanup and eliminate light contamination.

[0120] The transparency physical modeling module uses a multi-scale U-Net to fuse depth maps, motion masks, and material features, queries physical parameters (absorption / scattering coefficients) through a differentiable material library, and outputs a physically constrained transparency map to accurately simulate the optical properties of translucent objects.

[0121] The dynamic synthesis and temporal optimization module performs virtual illumination matching (ambient light compensation), foreground-background fusion, and bilateral filtering noise reduction. It applies inter-frame consistency loss and light flow constraints to ensure video smoothness and suppress edge haloing.

[0122] The recommendation and output module generates multiple versions of special effects videos (high speed, image quality, and balanced). Based on the four-dimensional evaluation matrix of real-time performance, image quality, resource consumption, and realism, it calculates the optimal version with weighted calculations based on scene templates to implement an adaptive output strategy.

[0123] The preferred embodiments of the present invention disclosed above are intended only to help illustrate the present invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the present invention to specific embodiments. Obviously, many modifications and variations are possible based on the contents of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the present invention, thereby enabling those skilled in the art to better understand and utilize the present invention. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for generating a visual effects video, characterized in that: The following steps are involved: S1. Multimodal data acquisition and preprocessing: Synchronously collect RGB video, depth map, and ambient lighting data, and generate a registered 3D point cloud through temporal alignment and IMU motion compensation, providing a spatial basis for special effects synthesis. The specific process of S1 is as follows: Synchronously obtain RGB video stream, depth map stream and ambient lighting data; Perform time domain alignment on RGB video streams and achieve inter-frame synchronization by matching feature points of adjacent frames; The IMU angular velocity data is integrated to calculate the rotation compensation matrix, and the improved optical flow algorithm is used to perform motion compensation. The depth map and RGB pixels are aligned through the calibration matrix to generate a pixel-by-pixel world coordinate point cloud; S2, spatiotemporal joint motion compensation matting: Alpha mask extraction of moving objects based on target contour displacement prediction and bidirectional mask optimization; S3, Edge Cleanup with Adaptive Illumination Separation: This integrates polarization data with colorimetric analysis to build a material-aware pollution weight model, eliminating light pollution and edge color bleeding in green screen scenes. The specific operation steps of S3 are as follows: Build a light pollution model based on ambient light data to analyze light interference in green screen scenes; Convert the RGB image to LAB color space, separate and quantify the pollution degree of the green component; Combined with polarization data, material differences are calculated and a material weight map is constructed to distinguish different surface characteristics. Perform dual-channel purification on the edge area of ​​the Alpha mask, using differentiated purification strategies for the diffuse reflection area and the metal or glass area, and output an optimized edge mask; S4, Multi-scale Transparency Estimation Network: Through U-Net, it fuses depth, motion mask and material features, combines physical parameter constraints to output a transparency map, and simulates the optical properties of translucent objects; The specific operation steps of S4 are as follows: Build a transparency estimation network based on U-Net, inputting RGB image patches, depth differences, motion masks, and material classification maps; In the decoding stage, multi-scale features are integrated, including edge details at the original resolution, texture structure at 1 / 2 resolution, and semantic context at 1 / 4 resolution. Introducing a material-aware attention mechanism to enhance transparency prediction in feature regions; Query physical parameters through the differentiable material library and calculate the final transparency map of physical constraints in combination with depth information; S5, dynamic special effects synthesis and output: performs virtual lighting matching, foreground-background fusion and bilateral filtering noise reduction, and applies a temporal consistency loss function to ensure video smoothness; S6, intelligent recommendation of special effects videos, based on the three-dimensional evaluation matrix of real-time performance, image quality, and resource consumption, combined with the weighted calculation of scene templates to achieve an adaptive output strategy.

2. The method for generating a visual effects video according to claim 1, wherein: The specific operation steps of S2 are as follows: Based on the instance segmentation results of the target contour, the displacement vector field between consecutive frames is extracted; Generate the prediction mask of the current frame through forward prediction, and optimize the mask accuracy by combining backward correction; Iteratively adjust the displacement vector and calculate the mask residual to dynamically optimize the motion compensation effect; Perform sub-pixel upsampling and edge detection on the contour edge area and output a high-resolution alpha mask.

3. The method for generating a visual effects video according to claim 1, wherein: The specific operation steps of S5 are as follows: Generate a virtual background based on the depth map and match the ambient lighting, and calculate the lighting compensation value of the foreground and virtual light source; The foreground and background are fused according to the final transparency map, and bilateral filtering is performed on the edge transition area; Apply inter-frame consistency loss function and illumination flow constraint to optimize video temporal coherence; Output special effects video with lighting correction and time domain optimization.

4. The method for generating a visual effects video according to claim 1, wherein: The specific operation steps of S6 are as follows: S601, generating multiple optimized versions of special effects videos according to the input scene, including a high-speed version, a quality version, and a balanced version, each version having a different resolution, frame rate, and optimized parameter configuration; S602: Establish a multi-dimensional evaluation matrix including real-time performance, image quality, resource consumption, and special effect realism, and obtain quantitative indicators of each dimension through system timers, OpenCV quality assessment tools, and hardware monitoring APIs; S603: Matching user demand characteristics based on a preset scene template library. The scene templates include four types: real-time live broadcast, movie production, mobile terminal publishing, and advertising special effects. Each type corresponds to a different weight vector. S604: After standardizing the evaluation indicators, the comprehensive scores of each version are calculated based on the scenario weights, and the recommendation probability distribution is generated through exponential transformation; S605: Finally, the special effects video version with the highest recommendation probability is output, and the user selection result is recorded for weight optimization.

5. A system for generating a visual effects video according to any one of claims 1 to 4, characterized in that: include: The data acquisition and processing module synchronously collects RGB video streams, depth map streams, and ambient lighting data, and generates a registered 3D point cloud through IMU motion compensation and temporal alignment, providing accurate spatial basic data for special effects synthesis. The dynamic keying and edge optimization module extracts target contours based on Mask R-CNN, generates sub-pixel alpha masks through bidirectional motion compensation and iterative displacement vector optimization, and combines polarization data to achieve material-aware edge cleanup and eliminate light pollution. The transparency physical modeling module uses a multi-scale U-Net to fuse depth maps, motion masks, and material features, queries physical parameters through a differentiable material library, and outputs a physically constrained transparency map to simulate the optical properties of translucent objects. The dynamic synthesis and temporal optimization module performs virtual illumination matching, foreground-background fusion, and bilateral filtering noise reduction. It applies inter-frame consistency loss and light flow constraints to ensure video smoothness and suppress edge haloing. The recommendation and output module generates multiple versions of special effects videos. Based on the four-dimensional evaluation matrix of real-time performance, image quality, resource consumption, and realism, it combines scene templates to weightedly calculate the optimal version and implement an adaptive output strategy.

Images