Dynamic 3D special effects rendering synthesis method and system based on cloud native architecture

Abstract

A dynamic 3D special effects rendering and synthesis method and system based on a cloud native architecture relate to the field of image processing technology, including: S1, slicing the original video and dynamically allocating the slices according to the real-time load status of the cloud rendering node, and then the multi-cloud rendering node uses a dynamic three-dimensional special effects generation module to parallel render and splice the slices to output a synthesized video file; S2, decoding the video stream and audio stream of the original video respectively to obtain video frames and audio blocks, aligning the special effects frames with the video frames and audio blocks through a two-way matching algorithm, and then synthesizing the special effects frames with the video frames through a video and special effects synthesis module to generate a final display frame, and updating the final display frame through a Laplacian pyramid and sub-pixel convolution, and writing the final display frame and audio blocks into the synthesized video file in timestamp order. This solves the problems of high energy consumption, unnatural fusion boundaries between special effects and video, and insufficient audio and video synchronization accuracy in traditional special effects rendering, and significantly improves the quality of the generated video.

Description

Technical Field

[0001] The present invention relates to the fields of computer graphics, distributed computing and green computing technology, and in particular to a dynamic 3D special effects rendering synthesis method and system based on a cloud native architecture. Background Art

[0002] Traditional dynamic 3D special effects rendering technology has long faced three core issues in film, television, and gaming: high energy consumption limits scalable application, unnatural special effects fusion boundaries, and insufficient audio and video synchronization accuracy. In high-performance standalone rendering mode, complex special effects generation relies on high-power GPU clusters. Actual data shows that single-frame rendering consumes over 300 watts, resulting in a surge in hardware costs and carbon footprint. Furthermore, the fusion of special effects and video often produces visual artifacts such as aliasing and ghosting due to projection parameter mismatches or sudden changes in transparency. This is particularly true in the transition areas between flames and clouds, where PSNR values ​​are low, requiring manual frame-by-frame correction. Furthermore, the synchronization of audio and video streams with the special effects timeline relies on fixed timestamps. In distributed rendering scenarios, network latency or accumulated fragmentation errors can easily lead to audio and video desynchronization, resulting in significant errors that severely impact the user experience.

[0003] Therefore, in the existing technology, the rendering of dynamic 3D special effects often relies on single-machine high-performance hardware, resulting in high energy consumption and difficulty in scalability; the fusion of special effects and video is prone to boundary jaggedness or transparency mutations, which require manual intervention and correction; audio and video synchronization relies on fixed timestamp matching, which is prone to cumulative errors during jumps or segmentation processing.

[0004] To address the above issues, there is an urgent need for a method and system that focuses on "dynamic 3D special effects low-power rendering and synthesis in cloud-native architecture" to build an integrated process from special effects generation, hybrid synthesis, distributed rendering to energy efficiency optimization. Summary of the Invention

[0005] To address these technical challenges, this paper proposes a dynamic 3D special effects rendering and synthesis method and system based on a cloud-native architecture. This approach aims to address core issues inherent in traditional special effects rendering, including high energy consumption, unnatural effects-video fusion boundaries, and insufficient audio and video synchronization. By integrating a cloud-native architecture with algorithm optimization, this approach achieves efficient generation, low-power synthesis, and cross-platform compatibility of dynamic 3D special effects, providing an intelligent rendering solution for complex scenes in film, television, gaming, virtual reality, and other fields.

[0006] In a first aspect, the present invention provides a dynamic 3D special effects rendering and synthesis system based on a cloud-native architecture, comprising the following modules:

[0007] A dynamic 3D special effects generation module is used to dynamically generate 3D special effects based on a multi-scale noise model according to the timestamp of the current rendering frame, pixel screen coordinates, and basic special effects parameters, and dynamically adjust the shape and color of the 3D special effects through parameterized control;

[0008] The video and special effects synthesis module is used to eliminate synthesis boundary defects by using a hybrid projection matrix switching strategy combined with a color mixing equation and an edge ablation algorithm for 3D special effects and 2D video frames, and perform depth control to obtain the final display frame;

[0009] The rendering control module is used to slice the original video and dynamically allocate the slices according to the real-time load status of the cloud rendering node. Then, the multi-cloud rendering node uses the dynamic 3D special effects generation module to render the slices in parallel to obtain the slice rendering results, and then splice the slice rendering results to output a composite video file. ; The fragment rendering result includes a special effects frame sequence;

[0010] The audio and video synchronization module is used to decode the video stream and audio stream of the original video respectively to obtain video frames and audio blocks, align the special effect frames with the video frames and audio blocks through a two-way matching algorithm, and then synthesize the special effect frames with the video frames through the video and special effect synthesis module to generate the final display frame, reconstruct the multi-scale features through the Laplacian pyramid, and use sub-pixel convolution to enhance high-frequency details to update the final display frame. Finally, the updated final display frame and audio block are written into the synthesized video file in timestamp order through the audio and video encapsulator. .

[0011] Specifically, the rendering control module includes:

[0012] Video segmentation submodule, used to divide the video into segments according to the number of cloud rendering nodes Use the fragmentation formula to split the original video into shards and serialize the metadata of each shard into a JSON format file The segments contain continuous video frames with nearly equal duration. The metadata of the segments include the start timestamp, end timestamp and special effect parameter subset of the segments. The segmentation formula is shown as follows:

[0013] ,

[0014] in, is the total number of video frames, ; is the total length of the video; is the video frame rate; N ≥1;

[0015] The dynamic load balancing scheduling submodule is used to allocate the slices to the cloud rendering node with the lowest real-time load through a greedy strategy according to the real-time load status of the cloud rendering node; the cloud rendering node calls the dynamic 3D special effects generation module to render according to the start timestamp, end timestamp and special effects parameter subset in the slice metadata, and generates the slice rendering result ;

[0016] The rendering result splicing submodule is used to render the fragments in chronological order. Perform stitching and eliminate the frame gaps at the boundaries of the tile rendering results by aligning the timestamps to output a composite video file .

[0017] Specifically, the audio and video synchronization module includes:

[0018] The audio and video decoding submodule is used to decode the original audio and video files through the FFmpeg library. Video stream and audio streams Perform independent decoding to obtain video frame sequence and audio block sequence ;

[0019] Video decoding uses jump decoding technology to convert video frames Quickly locate the target time point based on the key frame index. The formula is:

[0020] ,

[0021] in, For video frames The start time of the video, FPS is the video frame rate, in frames per second; is the start time in seconds; is the frame number; is the total number of frames of the original video;

[0022] Audio decoding maps audio blocks to timestamps The timestamps are aligned to the continuous time axis, and the formula is:

[0023] ,

[0024] in For audio blocks The starting time on the continuous time axis, The number of samples contained in each audio block, is the sampling rate, in units of , is the total number of audio blocks;

[0025] The time deviation calculation submodule is used to calculate the time deviation of each rendered frame. With video frames Video time deviation , and calculate each rendered frame With audio block Audio time deviation ;

[0026] ,

[0027] in, Rendering frame timestamp; For video frames timestamp, For audio blocks timestamp;

[0028] Synchronous frame insertion submodule is used for and , then determine and 、 Synchronize and jump to the final synthesis submodule; otherwise, start the interpolation compensation mechanism:

[0029] like , then by the video frame and adjacent video frames Linear interpolation to generate intermediate frames ; ; The linear interpolation formula is as follows:

[0030] ,

[0031] in, adjacent video frames The start time of

[0032] when When the audio block and Compress the duration and delete redundant segments through the WSOLA algorithm to speed up audio matching ;

[0033] when When the audio block and Stretch the duration and insert similar waveform segments through the WSOLA algorithm to slow down the audio to match ;

[0034] Display frame synthesis submodule is used to render the frame through off-screen rendering technology Overlay onto video frame or intermediate frames The final display frame is generated by eliminating the synthesis boundary through the video and special effects mixing module ;

[0035] Detail enhancement submodule for decomposing the final display frame via a Laplacian pyramid It is a multi-scale feature layer to separate high-frequency details and low-frequency infrastructure. After parallel reconstruction on distributed rendering nodes, the features are fused through pyramid inverse transform. At the same time, the learnable convolution kernel of sub-pixel convolution and sub-pixel spatial reconstruction operation are used to enhance the high-frequency details of the synthesized frame, and finally updated to the final display frame. ;

[0036] Encapsulate the output submodule and remap the audio blocks by timestamp and the final display frame Synchronize, and then use the audio and video encapsulator to synchronize the audio blocks and the final display frame Encapsulate and output the final output video file in timestamp order .

[0037] Specifically, the dynamic three-dimensional special effects generation module includes:

[0038] Initialize the submodule to define input parameters and convert pixel screen coordinates Perform normalization to obtain normalized coordinates The input parameters include the timestamp t of the current rendered frame, the pixel screen coordinates And special effect basic parameters; the normalization process is based on the formula and Conduct; among them, and Represents the width and height of the screen resolution respectively; ;

[0039] Synthetic noise generation submodule is used to generate the noise according to the normalized coordinates and the timestamp of the currently rendered frame The improved Simplex noise algorithm is used to generate basic noise values, and natural textures are generated by superimposing multiple layers of basic noise values ​​through fractal Brownian motion. , and then introduce the perturbation vector Get the synthetic noise value ;

[0040] The special effect color generation submodule is used to use different noise processing strategies to synthesize noise values ​​according to the different types of 3D special effects. Perform morphological and color processing to generate corresponding three-dimensional special effects and output special effect color values ;

[0041] 3D special effects generation submodule, used to convert special effects color values Convert to sRGB space through Gamma correction to get the correction effect color value , and then correct the special effect color value With background color value By transparency Mix according to the mixing formula to get the final color value of the pixel , complete the generation of three-dimensional special effects;

[0042] The mixing formula is shown below:

[0043] ,

[0044] in is a user-defined transparency parameter, .

[0045] Specifically, the video and special effects synthesis module includes:

[0046] 2D rendering submodule, used to set the projection matrix of the rendering pipeline to an orthographic projection matrix , 2D video frames are rendered into the color buffer through the rendering pipeline ; The color buffer Stores the original pixel color values ​​of the two-dimensional video frame in ; The color buffer The size is ,in is the width of the screen resolution; is the height of the screen resolution;

[0047] The orthographic projection matrix As shown below:

[0048] ,

[0049] in, is the width of the resolution of the 2D video frame; is the height of the resolution of the 2D video frame;

[0050] 3D special effects color calculation submodule, used to set the projection matrix of the rendering pipeline to the perspective projection matrix , 3D effects through the model view matrix and the perspective projection matrix Transform to screen space, generate fragments containing depth information, and calculate special effect color values ​​based on the fragment shader ; The perspective projection matrix As shown below:

[0051] ,

[0052] in, ; FOV is the camera field of view; aspect Camera viewing aspect ratio; near is the near clipping plane; far is the far clipping plane;

[0053] 2D video and special effects mixing submodule, used to convert special effects color values ​​into The original pixel color value of the two-dimensional video frame is superimposed to obtain the mixed color value ; The color mixing equation is as follows:

[0054] ,

[0055] Among them, the source factor By the transparency parameter of the special effect Dynamic control, Target factor Fixed to ;

[0056] The edge correction submodule is used to correct the mixed color values ​​through the edge ablation algorithm Correction is performed to eliminate boundary defects and obtain the final corrected mixed color .

[0057] Specifically, the rendering result splicing submodule further includes:

[0058] For fragment rendering results The overlapping intervals between the two are smoothly transitioned using a weighted hybrid algorithm; the weighted hybrid algorithm is shown in the following formula:

[0059] ,

[0060] in , The timestamp of the current frame; is the end timestamp of the i-th shard; The start timestamp of the i+1th shard.

[0061] Specifically, the improved Simplex noise algorithm in the synthetic noise generation submodule is implemented based on the following formula:

[0062] ,

[0063] in is the noise frequency parameter, which is used to control the density of the noise texture; It is the noise dynamic speed parameter, which determines how fast the noise changes over time.

[0064] Specifically, the noise processing strategy in the special effect color generation submodule includes:

[0065] For the flame effect, the formula Calculate the flame intensity and use the formula Adjust the base color of the flame to get the flame color , set the special effect color value Set to ;in 、 、 The base colors of the flames The red, green, and blue color components of the image; the clamp function is used to limit the value to between 0 and 1;

[0066] For cloud effects, according to the flow direction of the clouds Adjust the noise coordinates, the formula is , and combined with the density of the clouds pass Generate cloud color , set the special effect color value Set to ; Among them, the smoothstep function is used to smoothly interpolate within the threshold range;

[0067] For rain effects, based on the density of rain Rain color Perform binarization processing: When When the rain color Set to 1.0, otherwise the rain color Set to 0.0 to generate raindrop shapes based on the noise value distribution, and then set the special effect color value Set to .

[0068] Specifically, the edge ablation algorithm in the edge correction submodule is shown in the following formula:

[0069] ,

[0070] in, is the ablation factor, based on the normalized coordinates of the current pixel calculate:

[0071]

[0072] in, and are normalized coordinates.

[0073] In a second aspect, the present invention provides a dynamic 3D special effects rendering and synthesis method based on a cloud native architecture, based on the dynamic 3D special effects rendering and synthesis system based on a cloud native architecture described in the first aspect above, comprising the following steps:

[0074] S1. Split the original video into slices and dynamically allocate the slices according to the real-time load status of the cloud rendering node. Then, the multi-cloud rendering node renders the slices in parallel through the dynamic 3D special effects generation module to obtain the slice rendering results, and splices the slice rendering results to output the composite video file. ; The fragment rendering result includes a special effects frame sequence;

[0075] S2. Decode the video stream and audio stream of the original video separately to obtain video frames and audio blocks. Align the special effect frames with the video frames and audio blocks through a two-way matching algorithm. Then, synthesize the special effect frames with the video frames through a video and special effect synthesis module to generate the final display frame. Reconstruct multi-scale features through the Laplacian pyramid, and use sub-pixel convolution to enhance high-frequency details to update the final display frame. Finally, write the updated final display frame and audio blocks into the synthesized video file in timestamp order through the audio and video encapsulator. .

[0076] Focusing on the goal of "dynamic 3D special effects and low-power rendering synthesis in a cloud-native architecture", the present invention has constructed an integrated process from special effects generation, hybrid synthesis, distributed rendering to energy efficiency optimization. First, dynamic 3D special effects are generated in real time based on a multi-scale noise model. The morphological evolution of natural phenomena such as flames and clouds is controlled through parameterized interfaces (such as density, flow rate, and color parameters), and real physical motion is simulated in combination with a timestamp-driven dynamic perturbation vector. In the hybrid synthesis stage, an orthogonal projection matrix is ​​used to render a two-dimensional background video. At the same time, a perspective projection matrix is ​​used to superimpose three-dimensional special effects. An edge ablation factor is introduced to eliminate jagged edges at the synthesis boundary. In combination with a depth-based bottoming strategy, the background layer is always at the bottom of the rendering layer to avoid visual occlusion conflicts.

[0077] To meet the needs of large-scale video processing, a distributed, decode-free sharding strategy is proposed: the video is divided into multiple tasks along the timeline. These tasks are then dynamically allocated to a cloud node cluster for parallel rendering, combining real-time node computing power assessment (such as GPU load rate and memory availability) with task complexity weights (number of frames and special effects). Each node synchronously generates special effect frames based on the global timeline, using off-screen rendering technology to save intermediate results and compress transmission to reduce network bandwidth usage. Regarding energy efficiency optimization, a 7nm process chip and an immersion liquid cooling system are integrated, combined with low-power shader algorithms such as pre-calculated noise textures and Taylor expansion truncation to reduce real-time computing overhead. At the same time, optical flow consistency constraints and joint confidence assessment modules are used to correct motion distortion in distributed rendering and improve the visual fidelity of the synthesized results.

[0078] A bidirectional timestamp alignment mechanism is designed for audio and video synchronization, enabling rapid keyframe location through jump decoding. Bilinear interpolation is used on the video stream to compensate for interframe gaps between slice boundaries, and the audio stream duration is adjusted using the WSOLA algorithm to match the special effects timing. In the final output stage, multi-scale features are reconstructed using a Laplacian pyramid, and sub-pixel convolution is used to enhance high-frequency details. High-resolution, low-bitrate composite video is simultaneously generated to meet the real-time interactive needs of film and television industrialization and cloud gaming. The entire process achieves elastic scaling of computing power through a cloud-edge-end collaborative architecture, ensuring visual quality while reducing overall energy consumption to less than 10% of traditional stand-alone models. This provides an efficient and green technical foundation for the large-scale application of dynamic 3D special effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0080] Figure 1 A schematic diagram of a dynamic 3D special effects rendering and synthesis system based on a cloud-native architecture provided by an embodiment of the present invention;

[0081] Figure 2 A schematic diagram of a dynamic 3D special effects rendering and synthesis method based on a cloud-native architecture provided by an embodiment of the present invention;

[0082] Figure 3 Schematic diagram of a dynamic 3D special effects rendering and synthesis device based on a cloud-native architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The present invention can be explained in detail through the following examples. The purpose of providing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0084] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0085] Example 1

[0086] refer to Figure 1This embodiment provides a dynamic 3D special effects rendering and synthesis system based on a cloud-native architecture, including the following modules:

[0087] A dynamic 3D special effects generation module is used to dynamically generate 3D special effects based on a multi-scale noise model according to the timestamp of the current rendering frame, pixel screen coordinates, and basic special effects parameters, and dynamically adjust the shape and color of the 3D special effects through parameterized control;

[0088] The types of the three-dimensional special effects include flame special effects, cloud special effects and rain special effects; the basic parameters of the special effects include the basic color of the flame , the direction of cloud flow , cloud density and the density of rainwater ;

[0089] The dynamic 3D special effects generation module specifically includes the following submodules:

[0090] Initialize the submodule to define input parameters and convert pixel screen coordinates Perform normalization to obtain normalized coordinates The input parameters include the timestamp of the current rendered frame, the pixel screen coordinates and the basic parameters of the special effects;

[0091] Specifically, pixel screen coordinates By formula and Map it to the standardized space to get the normalized coordinates ;

[0092] The implementation process of dynamic 3D special effects generation starts with the definition and preprocessing of input parameters. Input parameters include the timestamp of the current rendered frame. (in milliseconds), pixel screen coordinates (in and Respectively represent the horizontal and vertical positions of pixels on the screen, with a value range of and , and Represents the width and height of the screen resolution respectively), as well as user-defined basic parameters of special effects, such as the basic color of the flame ( It consists of three components: red, green and blue 、 、 composition), cloud flow direction (two-dimensional vector parameter) and the density of rainwater (scalar parameter). The preprocessing stage first converts the pixel screen coordinates Normalize it and use the formula and Map it to the standardized space to get the normalized coordinates (The value range is ), providing a unified coordinate reference for subsequent noise calculations.

[0093] Synthetic noise generation submodule is used to generate the noise according to the normalized coordinates and the timestamp of the currently rendered frame The improved Simplex noise algorithm is used to generate basic noise values, and natural textures are generated by superimposing multiple layers of basic noise values ​​through fractal Brownian motion. , and then introduce the perturbation vector Get the synthetic noise value ;

[0094] Next, we enter the multi-scale noise model building and dynamic perturbation stage. Based on the normalized coordinates and the timestamp of the currently rendered frame , using the improved Simplex noise algorithm to generate basic noise values , the improved Simplex noise algorithm is implemented based on the following formula:

[0095] ,

[0096] in It is the noise frequency parameter, the default value is 5.0, which is used to control the density of the noise texture; It is the noise dynamic speed parameter, with a default value of 0.1, which determines how fast the noise changes over time;

[0097] The Simplex noise algorithm is an improved version of the original Perlin noise algorithm developed by Ken Perlin, designed to address some of the issues with the original Perlin noise, particularly computational efficiency and its ability to scale to high dimensions. It remains a gradient noise algorithm and is widely used in computer graphics, game development, procedural content generation, and other fields to create natural, organic textures, terrains, clouds, and other effects.

[0098] Subsequently, the base noise value is superimposed via Fractal Brownian Motion (FBM). Generate more complex natural textures , the formula is:

[0099] ,

[0100] in The number of fractal overlay layers, the default value is 4; The Hurst index has a default value of 0.5 and is used to control the roughness of the noise. Set to ;

[0101] The core of the multi-scale noise model is to superimpose basic noise values ​​through fractal Brownian motion (FBM) Generate more complex natural textures The mathematical modeling of Directly defines the generation process of natural textures;

[0102] In addition, the dynamic perturbation vector is introduced ,

[0103] in The perturbation amplitude is 0.1 by default. The perturbation offset is generated by the time-related sine and cosine functions, and the synthetic noise value is finally obtained. , providing basic data for the morphological control of different special effects.

[0104] Fractal Brownian Motion (FBM) is a mathematical model proposed by Mandelbrot and Ness in 1968. It is mainly used to describe irregular shapes such as mountains, clouds, terrain, and simulated planetary surfaces in nature. This is an existing technology and will not be described in detail here.

[0105] The special effect color generation submodule is used to use different noise processing strategies to synthesize noise values ​​according to the different types of 3D special effects. Perform morphological and color processing to generate corresponding three-dimensional special effects and output special effect color values ;

[0106] The noise processing strategy includes:

[0107] For the flame effect, the formula Calculate the flame intensity and use the formula Adjust the base color of the flame to get the flame color , set the special effect color value Set to ;in 、 、 The base colors of the flames The red, green, and blue color components of

[0108] For cloud effects, according to the flow direction of the clouds Adjust the noise coordinates, the formula is , and combined with the density of the clouds pass Generate cloud color , set the special effect color value Set to ; Among them, the smoothstep function is used to smoothly interpolate within the threshold range;

[0109] For rain effects, based on the density of rain Rain color Perform binarization processing: When When the rain color Set to 1.0, otherwise the rain color Set to 0.0 to generate raindrop shapes based on the noise value distribution, and then set the special effect color value Set to .

[0110] In the parameter-based special effect shape and color control stage, the synthetic noise value is adjusted according to the user's input parameters. Differentiated processing to produce specific effects.

[0111] For the flame effect, the formula Calculate the flame intensity (the clamp function is used to clamp the value between 0 and 1) and adjust the color component: flame color , which simulates the visual effect of flames by enhancing the red channel and suppressing the blue and green channels. 、 、 The base colors of the flames The red, green, and blue color components of

[0112] The cloud effects are based on the flow direction parameters (two-dimensional vector) adjusts the noise coordinates, the formula is , and combined with the density of the clouds ,pass Generates smooth transitions in cloud color , and the smoothstep function is used to smoothly interpolate within the threshold interval).

[0113] Rain effects based on the density of rain Rain color Perform binarization processing: When When the rain color Set to 1.0 (white) and 0.0 (transparent) otherwise to generate raindrop shapes based on the noise value distribution.

[0114] The essence of the formula is to pass the noise value and density threshold By comparing the noise value N(P) of the pixel point with the preset density threshold , simulating the physical process of random condensation of raindrops in the atmosphere. When the local noise value exceeds the density threshold, it is determined that raindrops are generated at that location.

[0115] Generating raindrop shapes based on noise value distribution specifically includes: using the second improved Simplex noise (optimizing gradient and interpolation) to make raindrop edges sharper, and adding spatiotemporal continuity parameters to eliminate inter-frame flickering, rather than the raindrop generation mechanism itself.

[0116] The second improved Simplex noise specifically includes:

[0117] 1) Generate a spatiotemporally varying noise field using anisotropic Simplex noise (y-axis stretching simulates falling raindrops);

[0118] 2) Compare the noise value with the dynamic density threshold and output the rainwater transparency in binary form;

[0119] 3) Remove noise through morphological corrosion and superimpose motion blur to enhance realism;

[0120] 4) Layer and mix noises of different frequencies according to depth of field to finally synthesize raindrop texture.

[0121] It is understandable that the special effect color value The color value of the pixel.

[0122] 3D special effects generation submodule, used to convert special effects color values Convert to sRGB space through Gamma correction to get the correction effect color value , and then correct the special effect color value With background color value By transparency Mix according to the mixing formula to get the final color value of the pixel , complete the generation of three-dimensional special effects.

[0123] Finally, perform color space conversion and output. (like 、 or ) is converted to sRGB space through Gamma correction, the formula is: ,but ;otherwise This step ensures that the color on the display device is consistent with the brightness distribution perceived by the human eye. With background color By transparency Mix according to the mixing formula to get the final color value ; The mixed formula is: ,in For user-defined transparency parameters ( ). Output As the final color value of the current pixel, the generation process of dynamic 3D special effects (natural special effects) is completed.

[0124] Background color The rain effect itself is derived directly from the rendering engine's frame buffer, which contains the rendered pixel values ​​of the current scene (such as buildings and the sky). The blending operation is standard alpha blending, with innovations focused on raindrop generation and color correction.

[0125] It is the pre-existing pixel color in the rendering pipeline, which comes from the rendering results of scene geometry, lighting, environment and other elements. The blending step is essentially the transparency synthesis (Alpha Blending) in general graphics. Its technical value lies in the generation and color correction of noise raindrops, rather than the background acquisition mechanism. In actual implementation, the engine directly samples the current pixel value from the frame buffer as .

[0126] The symbols involved in the above process are all derived from system input or intermediate calculations: timestamp Global clock from the rendering engine; pixel screen coordinates Passed in by the graphics rendering pipeline; normalized coordinates Calculated by resolution parameter; noise parameter ( 、 etc.) are set by the user or take the default value; the color components ( etc.) are defined by the user-configured base color; the corrected color With background The final color value is obtained by mixing the formula according to the transparency The whole process realizes realistic dynamic simulation of natural phenomena through mathematical modeling and parameterized control.

[0127] The rendering pipeline, also known as the rendering pipeline, pixel pipeline, or pixel pipeline, is a series of independent, parallel processing units within the display chip that handle graphics signals. To some extent, the rendering pipeline can be likened to the various production lines commonly found in factories. While production lines in factories are designed to improve production capacity and efficiency, the rendering pipeline improves the performance and efficiency of graphics cards.

[0128] The "Graphics Rendering Pipeline" is the industry-wide "rendering pipeline," which is responsible for converting 3D data into screen pixels and delivering coordinates (x, y) and timestamp t. This is existing technology and will not be discussed in detail here.

[0129] The video and special effects synthesis module is used to eliminate synthesis boundary defects by using a hybrid projection matrix switching strategy combined with a color mixing equation and an edge ablation algorithm for 3D special effects and 2D video frames, and perform depth control to obtain the final display frame;

[0130] The video and special effects mixing module specifically includes the following sub-modules:

[0131] 2D rendering submodule, used to set the projection matrix of the rendering pipeline to an orthographic projection matrix , 2D video frames are rendered into the color buffer through the rendering pipeline ; The color buffer Stores the original pixel color values ​​of the two-dimensional video frame in ; The color buffer The size is ,in is the width of the screen resolution; is the height of the screen resolution;

[0132] The orthographic projection matrix As shown below:

[0133] ,

[0134] in, is the width of the resolution of the 2D video frame; is the height of the resolution of the 2D video frame;

[0135] The implementation process of mixing 3D special effects with 2D video frames starts with the configuration of the projection matrix and the switching of the rendering mode. The input parameters include the resolution (width) of the 2D video frame. and height ), model view matrix of 3D special effects and the projection matrix (both 4×4 matrices), and screen resolution ( and ).

[0136] It is understandable that the screen resolution of the input parameters and and the above W and H They are completely equivalent, representing the width and height of the screen resolution respectively. In order to distinguish them from other height and width variables, the screen subscript is added here;

[0137] In the orthographic projection mode, the orthographic projection matrix is ​​defined by the formula , whose parameters are determined by the size of the 2D video frame:

[0138] ,

[0139] This matrix ensures that the 2D video frame maintains the exact dimensions in the screen coordinate system ( ), to avoid stretching or compression caused by projection deformation. Then, set the projection matrix of the rendering pipeline to , and render the 2D video frame into the color buffer through the rendering pipeline (resolution is ), which stores the original pixel color values ​​of the two-dimensional video frame , which is the background color mentioned above .

[0140] This step configures an orthogonal projection matrix based on the two-dimensional video frame size , and switch the rendering pipeline mode to render the 2D video frame to the color buffer of the screen resolution without distortion As a synthetic background;

[0141] 3D special effects color calculation submodule, used to set the projection matrix of the rendering pipeline to the perspective projection matrix , 3D effects through the model view matrix and the perspective projection matrix Transform to screen space, generate fragments containing depth information, and calculate special effect color values ​​based on the fragment shader ;

[0142] After the 2D video rendering is completed, the 3D special effects projection and mixing stage begins. Switch the projection matrix to the perspective projection matrix. , whose parameters are determined by the camera field of view ( ), aspect ratio ( ) and the near and far clipping planes ( , )definition:

[0143] ,

[0144] in In this mode, 3D effects are applied via the model view matrix. and the projection matrix Transform to screen space, generate fragments containing depth information, and calculate special effect color values ​​based on the fragment shader (such as fire, cloud or rain colors).

[0145] Specifically, the model view matrix It is a 4×4 matrix that transforms the 3D special effects from the model coordinate system to the camera observation coordinate system. It is composed of the model matrix Model (which controls the transformation of the special effects themselves) and the view matrix View (which controls the camera viewing angle). = View × Model), which serves as the standard input to the vertex shader of the rendering pipeline;

[0146] Vertex shader utilization The vertices are transformed into the clipping space using the projection matrix, and after primitive assembly and rasterization, fragments with depth values ​​are output; then, colors are generated through the shader program based on the fragment attributes (such as texture coordinates and lighting); this is existing technology and will not be described here.

[0147] 2D video and special effects mixing submodule, used to convert special effects color values ​​into The original pixel color value of the two-dimensional video frame is superimposed to obtain the mixed color value ;

[0148] The color mixing stage uses the color mixing equation of the graphics API to achieve the superposition of special effects and background. The color mixing equation is defined as:

[0149] ,

[0150] Among them, the source factor By the transparency parameter of the special effect Dynamic Control ( ), target factor Fixed to This configuration makes the special effect color value When the transparency is low ( Approaching 0) gradually blending into the background , but completely covers the background to achieve a depth-independent synthesis effect, which is an innovation of the present invention;

[0151] For example, the flame effect Values ​​can change over time Change, through the formula Achieve dynamic flashing effect.

[0152] The edge correction submodule is used to correct the mixed color values ​​through the edge ablation algorithm Correction is performed to eliminate boundary defects and obtain the final corrected mixed color ;

[0153] In order to achieve a smooth transition of the special effect edge, the boundary area needs to be ablated to eliminate boundary defects. According to the normalized coordinates of the current pixel calculate:

[0154]

[0155] In the above formula, and is the normalized coordinate . For the horizontal center area ( ) retains high weights, while marginal areas ( Approaching 0 or 1) rapidly decaying; For the vertical top area ( Close to 1.0, corresponding to the top of the screen) to attenuate to avoid the abrupt boundary of the special effect rectangle. The final mixed color is corrected to the final corrected mixed color :

[0156] The edge ablation algorithm is shown in the following formula:

[0157] ,

[0158] The depth control submodule is used to set the homogeneous coordinate z component output by the vertex shader to The value ensures that the 2D video frame is always used as the background layer after perspective division;

[0159] To ensure that the 2D video frame is always the background layer, it is necessary to force it to be at the bottom layer of the rendered scene through depth testing. When rendering the 2D video frame, set the homogeneous coordinate z component of the vertex shader output to Value (i.e. ), so that the z value of the normalized device coordinate after perspective division is 1.0 (the maximum value in the NDC space range [-1, 1]). The specific formula is:

[0160] Vertex coordinates:

[0161] After perspective division:

[0162] This step is the core process of rendering the 2D video frame into the color buffer through the rendering pipeline. Specifically, the background layer is forced to the bottom through depth control:

[0163] Vertex coordinates of a 2D video frame (typically the 3D coordinates of the four corners of a rectangle, with z initially set to 0) are input into the vertex shader. The vertex shader outputs homogeneous coordinates (x, y, z, and w), with the z component fixed at 1.0 and the w component set to 1.0. After rasterization, each fragment calls a fragment shader to calculate its color, which is ultimately written to the color buffer. This process is standard in the rendering pipeline and is currently used.

[0164] w is the fourth component of the homogeneous coordinates, used for perspective correction (e.g., objects appear larger when closer and smaller when farther away). It is not a depth value. The depth value is determined by the z value of the normalized device coordinates. Here, w is fixed to 1.0.

[0165] Depth control is an existing technology, and its principle is as follows:

[0166] Normalized Device Coordinates (NDC): Vertex coordinates are normalized after perspective division (NDC_xyz = (x / w, y / w, z / w)). z = 1.0 represents the farthest depth (with a depth range of 0 to 1, 1.0 is the maximum value).

[0167] Purpose: To convert the NDC_z of all vertices of the two-dimensional video frame (that is, the ) is set to 1.0 so that it is always judged as the "furthest" in the depth test, thus forcing it to be the background layer.

[0168] Since depth testing is retained by default The fragment with smaller value (i.e., the nearby object) makes the two-dimensional video frame The value is always greater than the 3D special effect fragment, so it is blocked, ensuring that it is only displayed in areas without 3D objects.

[0169] This step achieves efficient synthesis of two-dimensional video and three-dimensional special effects through projection matrix switching, color mixing equations, edge ablation algorithm and depth control, while avoiding visual boundary defects and rendering level conflicts.

[0170] The rendering control module is used to slice the original video and dynamically allocate the slices according to the real-time load status of the cloud rendering node. Then, the multi-cloud rendering node uses the dynamic 3D special effects generation module to render the slices in parallel to obtain the slice rendering results, and then splice the slice rendering results to output a composite video file. ; The fragment rendering result includes a special effects frame sequence;

[0171] The rendering control module includes the following submodules:

[0172] Video segmentation submodule, used to divide the video into segments according to the number of cloud rendering nodes Use the fragmentation formula to split the original video into shards and serialize the metadata of each shard into a JSON format file The segments contain continuous video frames with nearly equal duration. The metadata of the segments include the start timestamp, end timestamp and special effect parameter subset of the segments. The segmentation formula is shown as follows:

[0173] ,

[0174] in, is the total number of video frames, ; is the total length of the video; is the video frame rate; N ≥1;

[0175] The implementation process of distributed rendering and task splitting begins with the preprocessing of video files and the formulation of sharding strategies. Input parameters include the original video file (Format is , the total duration is seconds, and the frame rate is , total number of frames ), number of cloud rendering nodes , and user-defined sets of effect parameters (Including the type, time range and dynamic parameters of special effects such as flames, clouds, rain, etc.) First, without decoding the video content, according to the number of nodes Split the video into shards ( ), ensuring that each segment contains consecutive video frames and has approximately equal duration. The segment time range formula is as follows:

[0176]

[0177] in, and Respectively The start and end timestamps of each shard. The metadata associated with each shard (including the time range and , associated effect parameter subset ) is serialized into a JSON format file , By formula Generate, where For time range A subset of special effect parameters that are effective within.

[0178] The dynamic load balancing scheduling submodule is used to allocate the slices to the cloud rendering node with the lowest real-time load through a greedy strategy according to the real-time load status of the cloud rendering node; the cloud rendering node calls the dynamic 3D special effects generation module to render according to the start timestamp, end timestamp and special effects parameter subset in the slice metadata, and generates the slice rendering result ;

[0179] The cloud rendering node is a server that performs rendering tasks.

[0180] The task scheduling phase dynamically allocates shards based on the real-time load status of the cloud rendering node. Define Node The available computing power is , Value range ,Depend on The calculation is based on weighted indicators such as utilization, remaining memory, and task queue pressure. The formula is as follows:

[0181] ,

[0182] in, is the CPU utilization (e.g. 70% → 0.7); is the GPU utilization (e.g. 70% → 0.7); The ratio of available memory (e.g., remaining memory / total memory = 0.3); is the task queue pressure (e.g. number of pending shards / queue capacity = 0.6); w1, w2, w3, and w4 correspond to , , , The weight of . , , , It can be obtained according to actual conditions. This is an existing technology and will not be described in detail here. The values ​​of w1, w2, w3, and w4 can be adjusted as needed.

[0183] It's understood that the expression "70% → 0.7" above refers to converting a percentage value to a decimal. In this context, it specifically refers to the CPU or GPU utilization of a compute node. When these utilizations are expressed as percentages, such as 70%, they need to be converted to a decimal, i.e., 0.7, when calculating the node's available computing power. This conversion serves to unify the numerical representations used in calculations, enabling comparison and weighting of various metrics on a consistent basis.

[0184] is the shard resource requirement. The larger the value, the heavier the task. Estimate, For sharding The number of frames, is the number of special effects in the slice, and is the weight coefficient, the default , The scheduling algorithm allocates shards to the node with the lowest load through a greedy strategy, with the goal of minimizing the maximum node load:

[0185] Specifically, use Indicates node load pressure, greedy strategy selection The smallest node.

[0186] pass ≤ Prevent overload, Indicates the maximum load pressure of the node;

[0187] ,

[0188] Indicates the real-time load status of node j. The above formula represents the JSON format file The corresponding shard is assigned to the corresponding node j;

[0189] in is a tolerance factor (default 0.1) used to avoid load imbalance.

[0190] After each node receives the sharding task, it starts an independent rendering process, Time point jump decoding, through key frame positioning and inter-frame prediction (such as of Frame and Frame) decodes the video frame in the slice and uses the effect parameter subset Generate fragmented rendering results including 3D special effects (stored as image sequences or video blocks), the fragment rendering results This is the intermediate rendering result.

[0191] Through key frame positioning and inter-frame prediction (such as of Frame and Decoding video frames within a slice (frame) is a mature technology. Video codec standards (such as H.264 / HEVC) have clearly defined the collaboration mechanism between key frames (I frames) and predicted frames (P / B frames):

[0192] Key frame positioning: The decoder jumps to the key frame near the target time point as the decoding starting point;

[0193] Inter-frame prediction: Based on motion vectors and residual data, it gradually reconstructs subsequent predicted frames (P / B frames) from a key frame, ultimately outputting a continuous frame sequence. This process is fundamental to all modern video decoders and will not be detailed here.

[0194] It is worth noting that the basic special effect parameters are the core attributes that define the global 3D scene (such as model mesh, basic material, and physical engine configuration), and are usually the common parameters of the complete rendering task; while the special effect parameter subset It is a local parameter dynamically extracted from the basic parameters of special effects, which only contains the incremental configuration required by the current slice (such as light source displacement and particle density change within the slice time period); special effect parameter subset It is a subset of the time / space dimensions of the basic parameters of special effects, generated through parameter filtering (such as time range clipping and object ID screening), avoiding the transmission of full parameters to improve distributed rendering efficiency.

[0195] The dynamic load balancing scheduling submodule also includes: during the parallel rendering process, each cloud rendering node uses a unified global timeline for frame synchronization;

[0196] During the parallel rendering process, each cloud rendering node uses a unified global timeline for frame synchronization. The concept of global timeline frame synchronization is well known in the art and will not be further described here.

[0197] For the timestamp of any video frame in slice i , and its corresponding special effect parameters are dynamically calculated through linear interpolation:

[0198] ,

[0199] in and The time interval during which the special effect takes effect. Normalized interpolation coefficient, used to dynamically adjust the intensity of special effects (such as the disturbance amplitude of the flame ). Rendering result Each frame of the image is saved to the buffer through off-screen rendering technology to avoid conflicts with real-time display, and is compressed by compression algorithms (such as Encoding) to reduce the amount of transmitted data.

[0200] Offscreen rendering is a common performance optimization strategy used to reduce the time and complexity of graphics rendering. In this mode, all rendering operations are performed in a memory area outside the current screen view. Only after all rendering operations are completed is the final image transferred to the screen.

[0201] An important benefit of this approach is that it can avoid screen flickering or unnecessary redrawing, because all rendering work is done in the background and the user will not see half-finished rendering results. This is existing technology and will not be repeated here.

[0202] The rendering result splicing submodule is used to render the fragments in chronological order. Perform stitching and eliminate the frame gaps at the boundaries of the tile rendering results by aligning the timestamps to output a composite video file ;

[0203] The rendering result splicing submodule also includes: for the fragment rendering results The overlapping intervals between them are smoothly transitioned using a weighted hybrid algorithm;

[0204] The rendering result splicing submodule also includes: verifying the synthesized video file by checking and authenticating The data integrity of Hash value;

[0205] The cloud rendering node also includes an error handling mechanism, specifically: if the cloud rendering node does not reach the timeout threshold The fragment rendering result is returned in The corresponding shards are reallocated to other idle cloud rendering nodes, and the number of failures is recorded. If the same shard fails for more than times, it is marked as unavailable and the user is notified to intervene; the timeout threshold ; M=3; total length of the original video file;

[0206] Timestamp alignment refers to synchronizing data frames with different date and time stamps to facilitate data analysis and processing. This is an existing technology and will not be described in detail here.

[0207] After all nodes have completed the fragment rendering, they enter the result merging stage. The master node merges the fragment results in chronological order. Splicing is performed to eliminate the frame gaps at the fragment boundaries by aligning the timestamps. of With sharding of There may be overlap), and a weighted blending algorithm is used for smooth transition:

[0208] ,

[0209] in , The timestamp of the current frame. The final output is a complete composite video file and by checking the sum (such as Hash value) to verify data integrity.

[0210] The master node is statically designated by the management platform or dynamically elected by the Raft consensus algorithm, and is responsible for the timeline sorting of the sharding results, weighted mixing of the overlapping areas, and final video packaging.

[0211] Error handling mechanism for node failure or timeout problems, defining timeout thresholds If the node Not present The system will return the result within The corresponding shards are reallocated to other idle nodes and the number of failures is recorded. times (default ), it is marked as unavailable and the user is notified for intervention.

[0212] The audio and video synchronization module is used to decode the video stream and audio stream of the original video respectively to obtain video frames and audio blocks, align the special effect frames with the video frames and audio blocks through a two-way matching algorithm, and then synthesize the special effect frames with the video frames through the video and special effect synthesis module to generate the final display frame, reconstruct the multi-scale features through the Laplacian pyramid, and use sub-pixel convolution to enhance high-frequency details to update the final display frame. Finally, the updated final display frame and audio block are written into the synthesized video file in timestamp order through the audio and video encapsulator. .

[0213] The audio and video decoding submodule is used to decode the original audio and video files through the FFmpeg library. Video stream and audio streams Perform independent decoding to obtain video frame sequence and audio block sequence ;

[0214] The core goal of audio and video frame synchronization and output is to ensure that the rendered special effect frames are strictly aligned with the original audio and video streams in the time dimension, and output the final file through efficient encoding. The process begins with the audio and video decoding and timestamp extraction stage. The input parameters include the original audio and video files. (including video streaming and audio streams ), special effects rendering results (a sequence of images with timestamps generated by distributed nodes), as well as user-defined output parameters (such as target format, encoder type, bit rate, etc.). First, the FFmpeg library is used to and Perform independent decoding to obtain video frame sequence (Time stamp of each frame is ) and audio block sequence (The timestamp of each sample block (audio chunk) is ).

[0215] It is understandable that the video frame sequence The video frame Composed sequence, audio block sequence The audio block A sequence consisting of is the frame number, j is the audio block number;

[0216] Video Frame The decoding adopts jump decoding technology to quickly locate the target time point based on the key frame index. The formula is:

[0217] ,

[0218] in, For video frames The start time of the video, FPS is the video frame rate, in frames per second; The starting time is in seconds, and the default value is 0; is the frame number; is the total number of frames of the original video;

[0219] Audio decoding maps audio blocks to timestamps The timestamps are aligned to the continuous time axis, and the formula is:

[0220] ,

[0221] in For audio blocks The starting time on the continuous time axis, The number of samples to include in each audio block (default 1024), is the sampling rate, in units of , is the total number of audio blocks;

[0222] The time deviation calculation submodule is used to calculate the time deviation of each rendered frame. With video frames Video time deviation , and calculate each rendered frame With audio block Audio time deviation ;

[0223] ,

[0224] in, Rendering frame timestamp; For video frames timestamp, For audio blocks timestamp;

[0225] Special effects rendering results Is the rendered frame Part of, the value of k is consistent with i, rendering frame Composed of multiple fragment rendering results;

[0226] Synchronous frame insertion submodule is used for and , then determine and 、 Synchronize and jump to the final synthesis submodule; otherwise, start the interpolation compensation mechanism:

[0227] 1) Video interpolation: If , then by the video frame and adjacent video frames Linear interpolation to generate intermediate frames ; ; The linear interpolation formula is as follows:

[0228] ,

[0229] in, adjacent video frames The start time of

[0230] 2) Audio resampling: When When the audio block and Compress the duration and remove redundant segments using the WSOLA (Waveform Similarity Overlap-Add) algorithm to speed up audio matching. ;

[0231] when When the audio block and Stretch the duration and insert similar waveform segments through the WSOLA algorithm to slow down the audio to match ;

[0232] The timestamp synchronization phase aligns video frames and audio chunks using a bidirectional matching algorithm. For each rendered frame (Timestamp ), calculate its difference with the video frame and audio blocks Time deviation:

[0233] ,

[0234] in, Rendering frame timestamp; For video frames timestamp, For audio blocks timestamp;

[0235] like and , then determine and 、 Otherwise, start the interpolation compensation mechanism:

[0236] The WSOLA algorithm executes the following steps:

[0237] a. Segmented windowing: Divide the audio block into pieces of length (window length, unit: number of samples) overlapping sub-segments, the overlapping rate of adjacent sub-segments is 50% (i.e. step length ).

[0238] B. Similarity matching: For each target sub-segment, under the time offset constraint (maximum allowable offset, unit: number of samples) within the range of the source segment with the highest waveform similarity, and its offset is recorded as ( ).

[0239] c. Additive synthesis: Add the matched source sub-segments to the output buffer in a cross-fade manner to generate a stretched / compressed audio stream, with the synchronization error converged to .

[0240] Display frame synthesis submodule is used to render the frame through off-screen rendering technology Overlay onto video frame or intermediate frames The final display frame is generated by eliminating the synthesis boundary through the video and special effects mixing module ;

[0241] The synchronized rendering frame and audio and video data enter the synthesis stage. The rendering frame (special effect frame) is synthesized through off-screen rendering technology. Overlay onto video frame or interpolated frames , apply the edge ablation algorithm of the video and special effects mixing module to eliminate the synthesis boundary and generate the final display frame The color mixing formula is:

[0242] ,

[0243] in The transparency of the special effect is determined by user parameters or dynamic calculation (such as the flame effect ).

[0244] Detail enhancement submodule for decomposing the final display frame via a Laplacian pyramid It is a multi-scale feature layer to separate high-frequency details and low-frequency infrastructure. After parallel reconstruction on distributed rendering nodes, the features are fused through pyramid inverse transform. At the same time, the learnable convolution kernel of sub-pixel convolution and sub-pixel spatial reconstruction operation are used to enhance the high-frequency details of the synthesized frame, and finally updated to the final display frame. ;

[0245] Multi-scale features are reconstructed through the Laplacian pyramid, and sub-pixel convolution is used to enhance high-frequency details. High-resolution, low-bitrate synthetic videos are generated simultaneously to meet the real-time interaction needs of film and television industrialization and cloud games.

[0246] Laplacian pyramid reconstruction and sub-pixel convolution work together to achieve high-resolution, low-bitrate video synthesis through multi-scale feature fusion and high-frequency detail enhancement. The specific technical solution is: first, use the Laplacian pyramid to convert the final display frame The image is decomposed into multi-scale feature layers, generating image layers of varying resolutions using a Gaussian pyramid. Adjacent layers are then differentiated to extract high-frequency details (such as edges and textures) and low-frequency underlying structures, achieving feature separation. This is then reconstructed in parallel across distributed rendering nodes: the low-frequency layers preserve color and lighting information, while the high-frequency layers specifically enhance dynamic effects details (such as particle lighting or fluid motion). Finally, features at all scales are fused using an inverse pyramid transform to form a basic reconstructed frame.

[0247] Based on this basic reconstructed frame, sub-pixel convolution technology further optimizes high-frequency details. Learnable convolution kernels are applied to the reconstructed high-frequency feature layer. This process directly generates super-resolution details through sub-pixel spatial reorganization (e.g., rearranging the channel data of the low-resolution feature map into high-resolution pixels), avoiding the blurring artifacts caused by traditional interpolation. This process utilizes a residual connection mechanism to enhance high-frequency components in moving areas (such as character hair and environmental splashes) while preserving the natural transitions of the original low-frequency content, achieving a balance between visual fidelity and computational efficiency.

[0248] The synergy between the two enables the system to reduce data redundancy through pyramid decomposition and restore fine details through sub-pixel convolution (PSNR increased by 3.2dB), and finally output video frames that meet the dual requirements of film-level 4K / 120fps output and real-time interaction with cloud games (≤16ms / frame). The output video frames are based on the final display frame. to update.

[0249] Encapsulate the output submodule and remap the audio blocks by timestamp and the final display frame Synchronize, and then use the audio and video encapsulator to synchronize the audio blocks and the final display frame Encapsulate and output the final output video file in timestamp order .

[0250] The synchronous output of audio and video streams is achieved by timestamp remapping. The output timeline is defined as , whose timestamp precision is 1 microsecond ( ). For each frame , the associated audio block Need to meet:

[0251] ,

[0252] If not, a silent block is inserted or the poor audio sample is discarded (a poor audio sample is determined to be one with a signal-to-noise ratio (SNR) less than 10dB) to ensure audio and video synchronization;

[0253] The error handling mechanism defines a dynamic adjustment threshold to address the problem of synchronization deviation accumulation. :

[0254] ,

[0255] in is the attenuation coefficient (default 0.1), is the number of processed frames; The basic threshold represents the lower limit of the synchronization tolerance allowed. represents the synchronization reference timestamp of the kth frame; The timestamp of the frame rendered for frame k;

[0256] like If the maximum tolerance value (such as 50ms) is exceeded, the resynchronization process is triggered: the current queue is discarded, and the latest key frame is re-decoded and synchronized.

[0257] The final audio and video encapsulator (such as FFmpeg's AVMuxer) will Write the final output video file in timestamp order , encoding parameters are selected by preset rules:

[0258] - Video encoding: Using H.265 / HEVC encoding, bit rate bps( 、 is the resolution width and height);

[0259] - Audio encoding: AAC-LC encoding, bit rate , the sampling rate maintains the original SR value;

[0260] - Packaging format: MP4 or MOV, supporting streaming and multi-platform playback.

[0261] The chip of the cloud rendering node is based on a 7nm process chip, its heat dissipation system uses immersion liquid cooling technology, and its shader fast algorithm uses noise pre-calculation or Taylor truncation to reduce the energy consumption of the cloud rendering node;

[0262] This embodiment also uses a heuristic algorithm to allocate rendering tasks to the slices based on the computational complexity of the slices and the real-time energy efficiency ratio of the cloud rendering nodes, and reduces the energy consumption of the cloud rendering nodes to 10% of that in stand-alone mode;

[0263] Cloud-native low-power optimization: Based on a 7nm chip process, immersion liquid cooling (PUE ≤ 1.1), and fast shader algorithms (noise pre-calculation and Taylor truncation), energy consumption is reduced to 10% of stand-alone mode.

[0264] Shaders are the core computing units of cloud rendering nodes.

[0265] Noise precomputation generates 1024×1024 resolution texture maps for dynamic noise (such as Perlin noise for flames and clouds) offline, allowing for fast sampling by cloud rendering nodes during runtime. This technology converts the real-time noise calculations in the shader, which require O(n) complexity, into O(1) texture reads, reducing GPU computation instructions.

[0266] Taylor truncation uses polynomials of third order or less to approximate complex functions (such as sin / cos), reducing floating-point operations while maintaining a visual error of less than 0.01.

[0267] The core goal of cloud-native low-power optimization is to control overall energy consumption to less than 10% of traditional stand-alone mode through hardware architecture upgrades, dynamic resource scheduling, and algorithm acceleration. This process begins with hardware selection and heat dissipation design. Input parameters include the hardware configuration set of the cloud node. (Including chip process parameters , heat dissipation type etc.), global task queue (including the slices to be rendered and special effect parameters) (mainly used to manage rendering tasks, for task sorting and determining the before and after of task execution) and historical energy consumption data (Record the power consumption of the node under different loads). During the hardware configuration phase, select chips with a process below 7 nanometers (such as 5nm GPU), and its transistor dynamic power consumption It can be estimated by the formula:

[0268] ,

[0269] Where k is the process constant ( Craftsmanship )、 is the chip frequency (unit )、 is the operating voltage (unit V), is the transistor switching activity factor (default 0.3). Compared with the traditional 28nm chip ( ), the dynamic power consumption of the 7nm chip can be reduced to 32% of that of the 28nm chip at the same frequency (through the ratio 0.8 / 2.5≈0.32), thereby directly reducing the energy consumption of the computing unit.

[0270] The cooling system uses immersion liquid cooling technology, and its power usage efficiency (PUE) is calculated using the following formula:

[0271] ,

[0272] Compared with traditional air cooling system Liquid cooling can reduce heat dissipation-related energy consumption by approximately 37.1%. Calculated by comparing heat dissipation power consumption:

[0273] ,

[0274] in For the total power consumption of the node, the heat dissipation ratio in liquid cooling mode is reduced from 28.6% ((1.4-1) / 1.4) of air cooling to 4.76% ((1.05-1) / 1.05).

[0275] At the resource dynamic scheduling level, define nodes Real-time energy efficiency ratio for:

[0276]

[0277] in is the computing performance of node j, For real-time power consumption; the scheduler prioritizes slicing high-complexity tasks ( ) are allocated to nodes with high energy efficiency ratio ( ), ensuring that the computing throughput is maximized under unit energy consumption.

[0278] The computing performance and real-time power consumption of the node are obtained through the node IPMI interface. This is an existing technology and will not be described here.

[0279] , Calculated using known system information:

[0280] Indicates the average energy efficiency ratio (performance per unit power consumption) of all nodes in the cloud rendering cluster, which is used to measure the overall energy efficiency level of the cluster.

[0281]

[0282] M: total number of nodes in the cluster;

[0283] : Real-time energy efficiency ratio of node j.

[0284] The cluster energy efficiency ratio threshold is the energy efficiency ratio threshold when the task scheduler assigns tasks, and is used to screen high-efficiency nodes.

[0285] The task allocation strategy is implemented through a heuristic algorithm:

[0286] like and , then assign tasks To node j;

[0287] in is the cluster energy efficiency ratio threshold (the default is 1.2 times of For nodes The remaining memory capacity (in GB), For the task This strategy can reduce the active time of inefficient nodes, thereby reducing idle power consumption accumulation.

[0288] Heuristic algorithms are proposed in contrast to optimization algorithms. An optimal algorithm for a problem finds the optimal solution for each instance of that problem. A heuristic algorithm can be defined as an algorithm based on intuition or experience that, within acceptable cost (in terms of computational time and space), produces a feasible solution for each instance of the combinatorial optimization problem. The degree to which this feasible solution deviates from the optimal solution is generally unpredictable. Currently, heuristic algorithms primarily mimic natural object algorithms, including the ant colony algorithm, simulated annealing, and neural networks.

[0289] Software algorithm optimization focuses on low-power implementation of shader programs. Taking the noise perturbation calculation in flame special effects as an example, the traditional method uses real-time Simplex noise generation, and its computational complexity is By pre-calculating the noise texture (resolution 1024×1024), and perform bilinear interpolation at runtime to reduce the complexity to , reducing GPU computing load. The interpolation formula is:

[0290] ,

[0291] in is the integer part of the texture coordinate, and is the decimal part; Represents the sample value obtained by interpolation, and mix represents the bilinear interpolation function;

[0292] Global optimization of energy consumption is achieved through real-time monitoring and feedback mechanisms. Define cluster-level power consumption budget It is 10% of the energy consumption of the traditional stand-alone mode, that is:

[0293] ,

[0294] in is the equivalent number of single machines, is the basic power consumption of a single machine, For rendering power consumption, is the total task duration. Actual power consumption Through real-time collection of node sensors, if , the degradation strategy is triggered: automatically shutting down redundant nodes, reducing rendering resolution (from 4K to 1080P) or reducing the complexity of special effects (such as reducing the number of FBM layers from 4 to 2).

[0295] Precomputed noise textures achieve a significant reduction in computational complexity by shifting real-time noise generation to the preprocessing stage: Perlin / Simplex noise is baked offline in the preprocessing stage to a 1024×1024 static texture (storing the precomputed noise value per pixel). At runtime, the shader only needs to sample the texture once with O(1) complexity, and bilinear interpolation is automatically performed by the GPU texture unit (hardware accelerated). This is compared to traditional per-pixel real-time noise calculation (which involves O(n) operations such as lattice positioning, gradient hashing, and interpolation).

[0296] Equivalent number of single machines Obtained through real-world performance benchmarking: First, measuring the total time required for a single machine to render a complete video independently , and then measure the total time it takes for the cloud rendering cluster to complete the same video , finally according to the formula Calculated.

[0297] Basic power consumption of a single machine ( ) refers to the standby power consumption of the server when it is not performing any task (including basic power consumption such as CPU / memory), which is measured by IPMI sensor or external power meter. Actual power consumption ( ) is the real-time power consumption ( ) Integration in the time dimension: Continuously collect instantaneous power consumption through node sensors , accumulate the time from the start to the end of the task (formula: ), which reflects the total energy consumption rather than the instantaneous value.

[0298] Total task duration Total length of the original video Completely equivalent. The video segmentation strategy ensures that the total duration of all segments is equal to the original video duration, and the segmentation process does not introduce additional time overhead (the segment duration is nearly equal and divided according to the timeline). Rendering power consumption It is defined as the average incremental power consumption (unit: watt W) when a single machine performs rendering, not the power consumption per frame. Its value is obtained by measuring the total power consumption of a single machine at full load. Subtract basic power consumption Obtain, for example, measured 、 hour This power consumption covers the average dynamic energy consumption of the entire rendering process and is independent of the frame rate.

[0299] In the technical solution of the energy consumption optimization and degradation strategy, the system dynamically selects the degradation method through a feedback loop that monitors the cluster power consumption and performance indicators in real time: when the real-time total power consumption exceeds 120% of the budget and the remaining task duration is greater than 30 minutes, resolution degradation (4K→1440p→1080p) is triggered first, and sub-pixel convolution is used to upsample the low-resolution rendering results to the target 4K output to ensure resolution compliance; if there is still overconsumption, the complexity of special effects is reduced.

[0300] If the overall cluster load rate remains below 40% for 5 minutes, nodes with GPU utilization <10% and memory usage <20% are considered redundant nodes. These nodes are marked as "unschedulable" via the Kubernetes API, and a soft shutdown is performed using the IPMI command after the task is migrated.

[0301] The number of FBM layers is directly related to the complexity of special effects: each additional layer requires one more noise sampling and interpolation execution (for example, 4 layers of flame simulation core / outer flame / sparks require 120 instructions per pixel; 2 layers of basic form only require 45 instructions per pixel). Reducing the number of layers can significantly reduce the amount of real-time calculations.

[0302] The proof of the 10% energy consumption reduction relies on closed-loop control: the measured single-node power consumption after pre-calculation / Taylor truncation / dynamic degradation is 93.6W (traditional single machine power consumption is 936W). Cluster-level monitoring collects power consumption every 10 seconds. If it exceeds the budget by 5%, the strategy is triggered step by step. Ultimately, through sub-pixel convolution upsampling and complexity compression, the total energy consumption of the 20-node cluster is reduced to 2.272kWh, which is 12.1% of the traditional mode of 18.72kWh. Combined with liquid cooling and 7nm chip optimization, it is stabilized within the 10% threshold.

[0303] This embodiment also uses the optical flow consistency constraint and joint confidence evaluation module to correct motion distortion in distributed rendering and improve the visual fidelity of the synthesis result. Specifically, it includes:

[0304] 1) Optical flow consistency constraint:

[0305] Calculate the pixel motion vector (optical flow field) between adjacent frames and force the rendering result to meet the motion continuity. If the motion trajectory of the object rendered by the fragment exceeds the threshold value compared with the optical flow prediction (such as 5 pixels), triggering local re-rendering to eliminate inter-frame jitter.

[0306] 2) Joint confidence evaluation:

[0307] For dynamic characters, a confidence weight is assigned to each joint based on the visibility of the skeleton joints (occlusion analysis) and the credibility of the sensor data (such as the accuracy of the depth camera). . Low confidence joints ( )’s motion data is replaced by high-confidence adjacent joint interpolation to avoid abnormal postures.

[0308] It is worth noting that this does not refer to the human skeleton, but rather the concept of "rigid body node" in mechanics, which is used to describe the dynamic structure inside the special effects. For example:

[0309] In the flame special effects, the fire source serves as the "root joint" and the trajectory of the spark ejection serves as the "sub-joint chain" to form a hierarchical motion model; the cloud flow is discretized into grid control points (virtual joints), and turbulence mutations are constrained by confidence weights.

[0310] 3) Joint Optimization:

[0311] The optical flow error and joint confidence are combined to generate a motion distortion correction mask, which guides the cloud rendering nodes to prioritize the calculation of high-distortion areas (such as the edges of fast-moving limbs), thereby improving visual fidelity while ensuring real-time performance.

[0312] This embodiment focuses on the goal of "dynamic 3D special effects and low-power rendering synthesis in a cloud-native architecture" and builds an integrated process from special effects generation, hybrid synthesis, distributed rendering to energy efficiency optimization. First, dynamic 3D special effects are generated in real time based on a multi-scale noise model. The morphological evolution of natural phenomena such as flames and clouds is controlled through parameterized interfaces (such as density, flow rate, and color parameters), and real physical motion is simulated in combination with a timestamp-driven dynamic perturbation vector. In the hybrid synthesis stage, an orthogonal projection matrix is ​​used to render a two-dimensional background video, while a perspective projection matrix is ​​used to superimpose three-dimensional special effects. An edge ablation factor is introduced to eliminate jagged edges at the synthesis boundary, and a depth-based bottoming strategy is combined to ensure that the background layer is always at the bottom of the rendering layer to avoid visual occlusion conflicts.

[0313] To meet the needs of large-scale video processing, a distributed, decode-free sharding strategy is proposed: the video is divided into multiple tasks along the timeline. These tasks are then dynamically allocated to a cloud node cluster for parallel rendering, combining real-time node computing power assessment (such as GPU load rate and memory availability) with task complexity weights (number of frames and special effects). Each node synchronously generates special effect frames based on the global timeline, using off-screen rendering technology to save intermediate results and compress transmission to reduce network bandwidth usage. Regarding energy efficiency optimization, a 7nm process chip and an immersion liquid cooling system are integrated, combined with low-power shader algorithms such as pre-calculated noise textures and Taylor expansion truncation to reduce real-time computing overhead. At the same time, optical flow consistency constraints and joint confidence assessment modules are used to correct motion distortion in distributed rendering and improve the visual fidelity of the synthesized results.

[0314] A bidirectional timestamp alignment mechanism is designed for audio and video synchronization, enabling rapid keyframe location through jump decoding. Bilinear interpolation is used on the video stream to compensate for interframe gaps between slice boundaries, and the audio stream duration is adjusted using the WSOLA algorithm to match the special effects timing. In the final output stage, multi-scale features are reconstructed using a Laplacian pyramid, and sub-pixel convolution is used to enhance high-frequency details. High-resolution, low-bitrate composite video is simultaneously generated to meet the real-time interactive needs of film and television industrialization and cloud gaming. The entire process achieves elastic scaling of computing power through a cloud-edge-end collaborative architecture, ensuring visual quality while reducing overall energy consumption to less than 10% of traditional stand-alone models. This provides an efficient and green technical foundation for the large-scale application of dynamic 3D special effects.

[0315] Example 2

[0316] refer to Figure 2 This embodiment provides a dynamic 3D special effects rendering synthesis method based on a cloud native architecture, including the following steps:

[0317] S1. Split the original video into slices and dynamically allocate the slices according to the real-time load status of the cloud rendering node. Then, the multi-cloud rendering node renders the slices in parallel through the dynamic 3D special effects generation module to obtain the slice rendering results, and splices the slice rendering results to output the composite video file. ; The fragment rendering result includes a special effects frame sequence;

[0318] S2. Decode the video stream and audio stream of the original video separately to obtain video frames and audio blocks. Align the special effect frames with the video frames and audio blocks through a two-way matching algorithm. Then, synthesize the special effect frames with the video frames through a video and special effect synthesis module to generate the final display frame. Reconstruct multi-scale features through the Laplacian pyramid, and use sub-pixel convolution to enhance high-frequency details to update the final display frame. Finally, write the updated final display frame and audio blocks into the synthesized video file in timestamp order through the audio and video encapsulator. .

[0319] Example 3

[0320] refer to Figure 3 , Figure 3 is a schematic diagram of the structure of a dynamic 3D special effects rendering and synthesis device based on a cloud-native architecture in this embodiment. The dynamic 3D special effects rendering and synthesis device 20 based on a cloud-native architecture in this embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, the steps in the above-described method embodiment are implemented. Alternatively, when the processor 21 executes the computer program, the functions of each module / unit in the above-described apparatus embodiments are implemented.

[0321] Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the dynamic 3D special effects rendering and synthesis device 20 based on the cloud native architecture. For example, the computer program can be divided into the modules in Example 2. For the specific functions of each module, please refer to the working process of the device described in the above embodiment, and no further details will be given here.

[0322] The dynamic 3D special effects rendering and synthesis device 20 based on the cloud native architecture may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will appreciate that the schematic diagram is merely an example of the dynamic 3D special effects rendering and synthesis device 20 based on the cloud native architecture and does not limit the dynamic 3D special effects rendering and synthesis device 20 based on the cloud native architecture. The dynamic 3D special effects rendering and synthesis device 20 based on the cloud native architecture may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the dynamic 3D special effects rendering and synthesis device 20 based on the cloud native architecture may also include input and output devices, network access devices, buses, and the like.

[0323] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the dynamic 3D special effects rendering and synthesis device 20 based on the cloud-native architecture, and utilizes various interfaces and lines to connect the various parts of the dynamic 3D special effects rendering and synthesis device 20 based on the cloud-native architecture.

[0324] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements the various functions of the cloud-native architecture-based dynamic 3D special effects rendering and synthesis device 20 by running or executing the computer programs and / or modules stored in the memory 22 and accessing the data stored in the memory 22. The memory 22 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or an image playback function); the data storage area may store data generated based on the use of the mobile phone (such as audio data and a phone book). Furthermore, the memory 22 may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0325] If the modules / units integrated in the cloud-native architecture-based dynamic 3D special effects rendering and synthesis device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the processes in the above-mentioned method embodiments by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed by the processor 21, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practices in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practices, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0326] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0327] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 Multiple processes and / or boxes Figure 1 A device that performs the functions specified in one or more boxes.

[0328] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0329] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0330] The parts of the present invention that are not described in detail are prior art. It is obvious to those skilled in the art that the present invention is not limited to the details of the above-mentioned exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and it is intended that all changes that fall within the meaning and scope of equivalent elements are included in the present invention.

Claims

1. A dynamic 3D special effects rendering and synthesis system based on cloud native architecture, characterized by: Includes the following modules: A dynamic 3D special effects generation module is used to dynamically generate 3D special effects based on a multi-scale noise model according to the timestamp of the current rendering frame, pixel screen coordinates, and basic special effects parameters, and dynamically adjust the shape and color of the 3D special effects through parameterized control; The video and special effects synthesis module is used to eliminate synthesis boundary defects by using a hybrid projection matrix switching strategy combined with a color mixing equation and an edge ablation algorithm for 3D special effects and 2D video frames, and perform depth control to obtain the final display frame; The rendering control module is used to slice the original video and dynamically allocate the slices according to the real-time load status of the cloud rendering node. Then, the multi-cloud rendering node uses the dynamic 3D special effects generation module to render the slices in parallel to obtain the slice rendering results, and then splice the slice rendering results to output a composite video file. ; The fragment rendering result includes a special effects frame sequence; The audio and video synchronization module is used to decode the video stream and audio stream of the original video respectively to obtain video frames and audio blocks, quickly locate key frames through jump decoding, use bilinear interpolation to compensate for the frame gaps at the slice boundary of the video stream, and adjust the duration of the audio stream through the WSOLA algorithm to match the special effect timing to align the special effect frame with the video frame and audio block. Then, the special effect frame and the video frame are synthesized to generate the final display frame through the video and special effect synthesis module, and multi-scale features are reconstructed through the Laplace pyramid. Sub-pixel convolution is used to enhance high-frequency details to update the final display frame. Finally, the updated final display frame and audio block are written into the synthetic video file in timestamp order through the audio and video encapsulator. .

2. The dynamic 3D special effects rendering and synthesis system based on cloud native architecture according to claim 1 is characterized in that: The rendering control module includes: Video segmentation submodule, used to divide the video into segments according to the number of cloud rendering nodes Use the fragmentation formula to split the original video into shards and serialize the metadata of each shard into a JSON format file The segments contain continuous video frames with nearly equal duration. The metadata of the segments include the start timestamp, end timestamp and special effect parameter subset of the segments. The segmentation formula is shown as follows: ， in, is the total number of video frames, ; is the total length of the video; is the video frame rate; N ≥1; The dynamic load balancing scheduling submodule is used to allocate the slices to the cloud rendering node with the lowest real-time load through a greedy strategy according to the real-time load status of the cloud rendering node; the cloud rendering node calls the dynamic 3D special effects generation module to render according to the start timestamp, end timestamp and special effects parameter subset in the slice metadata, and generates the slice rendering result ; The rendering result splicing submodule is used to render the fragments in chronological order. Perform stitching and eliminate the frame gaps at the boundaries of the tile rendering results by aligning the timestamps to output a composite video file .

3. The dynamic 3D special effects rendering and synthesis system based on cloud native architecture according to claim 2 is characterized in that: The audio and video synchronization module includes: The audio and video decoding submodule is used to decode the original audio and video files through the FFmpeg library. Video stream and audio streams Perform independent decoding to obtain video frame sequence and audio block sequence ; Video decoding uses jump decoding technology to convert video frames Quickly locate the target time point based on the key frame index. The formula is: ， in, For video frames The start time of the video, FPS is the video frame rate, in frames per second; is the start time in seconds; is the frame number; is the total number of frames of the original video; Audio decoding maps audio blocks to timestamps The timestamps are aligned to the continuous time axis, and the formula is: ， in For audio blocks The starting time on the continuous time axis, The number of samples contained in each audio block, is the sampling rate, in units of , is the total number of audio blocks; The time deviation calculation submodule is used to calculate the time deviation of each rendered frame. With video frames Video time deviation , and calculate each rendered frame With audio block Audio time deviation ; ， in, Rendering frame timestamp; For video frames timestamp, For audio blocks timestamp; Synchronous frame insertion submodule is used for and , then determine and 、 Synchronize and jump to the final synthesis submodule; otherwise, start the interpolation compensation mechanism: like , then by the video frame and adjacent video frames Linear interpolation to generate intermediate frames ; ; The linear interpolation formula is as follows: ， in, adjacent video frames The start time of when When the audio block and Compress the duration and delete redundant segments through the WSOLA algorithm to speed up audio matching ; when When the audio block and Stretch the duration and insert similar waveform segments through the WSOLA algorithm to slow down the audio to match ; Display frame synthesis submodule is used to render the frame through off-screen rendering technology Overlay onto video frame or intermediate frames The final display frame is generated by eliminating the synthesis boundary through the video and special effects mixing module ; Detail enhancement submodule for decomposing the final display frame via a Laplacian pyramid It is a multi-scale feature layer to separate high-frequency details and low-frequency infrastructure. After parallel reconstruction on distributed rendering nodes, the features are fused through pyramid inverse transform. At the same time, the learnable convolution kernel of sub-pixel convolution and sub-pixel spatial reconstruction operation are used to enhance the high-frequency details of the synthesized frame, and finally updated to the final display frame. ; Encapsulate the output submodule and remap the audio blocks by timestamp and the final display frame Synchronize, and then use the audio and video encapsulator to synchronize the audio blocks and the final display frame Encapsulate and output the final output video file in timestamp order .

4. The dynamic 3D special effects rendering and synthesis system based on cloud native architecture according to claim 1, characterized in that: The dynamic three-dimensional special effects generation module includes: Initialize the submodule to define input parameters and convert pixel screen coordinates Perform normalization to obtain normalized coordinates The input parameters include the timestamp t of the current rendered frame, the pixel screen coordinates And special effect basic parameters; the normalization process is based on the formula and Conduct; among them, and Respectively represent the width and height of the screen resolution; ; Synthetic noise generation submodule is used to generate the noise according to the normalized coordinates and the timestamp of the currently rendered frame The improved Simplex noise algorithm is used to generate basic noise values, and natural textures are generated by superimposing multiple layers of basic noise values ​​through fractal Brownian motion. , and then introduce the perturbation vector Get the synthetic noise value ; The special effect color generation submodule is used to use different noise processing strategies to synthesize noise values ​​according to the different types of 3D special effects. Perform morphological and color processing to generate corresponding three-dimensional special effects and output special effect color values ; 3D special effects generation submodule, used to convert special effects color values Convert to sRGB space through Gamma correction to get the correction effect color value , and then correct the special effect color value With background color value By transparency Mix according to the mixing formula to get the final color value of the pixel , complete the generation of three-dimensional special effects; The mixing formula is shown below: ， in is a user-defined transparency parameter, .

5. The dynamic 3D special effects rendering and synthesis system based on cloud native architecture according to claim 1 is characterized in that: The video and special effects synthesis module includes: 2D rendering submodule, used to set the projection matrix of the rendering pipeline to an orthographic projection matrix , 2D video frames are rendered into the color buffer through the rendering pipeline ; The color buffer Stores the original pixel color values ​​of the two-dimensional video frame in ; The color buffer The size is ,in is the width of the screen resolution; is the height of the screen resolution; The orthographic projection matrix As shown below: ， in, is the width of the resolution of the 2D video frame; is the height of the resolution of the 2D video frame; 3D special effects color calculation submodule, used to set the projection matrix of the rendering pipeline to the perspective projection matrix , 3D effects through the model view matrix and the perspective projection matrix Transform to screen space, generate fragments containing depth information, and calculate special effect color values ​​based on the fragment shader ; The perspective projection matrix As shown below: ， in, ; FOV is the camera field of view; aspect Camera viewing aspect ratio; near is the near clipping plane; far is the far clipping plane; 2D video and special effects mixing submodule, used to convert special effects color values ​​into Superimpose the original pixel color value of the two-dimensional video frame to obtain the mixed color value ; The color mixing equation is as follows: ， Among them, the source factor By the transparency parameter of the special effect Dynamic control, Target factor Fixed to ; The edge correction submodule is used to correct the mixed color values ​​through the edge ablation algorithm Correction is performed to eliminate boundary defects and obtain the final corrected mixed color .

6. The cloud-native architecture-based dynamic 3D special effects rendering and synthesis system according to claim 2, characterized in that: The rendering result splicing submodule also includes: For fragment rendering results The overlapping intervals between the two are smoothly transitioned using a weighted hybrid algorithm; the weighted hybrid algorithm is shown in the following formula: ， in , The timestamp of the current frame; is the end timestamp of the i-th shard; The start timestamp of the i+1th shard.

7. The cloud-native architecture-based dynamic 3D special effects rendering and synthesis system according to claim 4, characterized in that: The improved Simplex noise algorithm described in the synthetic noise generation submodule is implemented based on the following formula: ， in is the noise frequency parameter, which is used to control the density of the noise texture; It is the noise dynamic speed parameter, which determines how fast the noise changes over time.

8. The cloud-native architecture-based dynamic 3D special effects rendering and synthesis system according to claim 4, characterized in that: The noise processing strategy in the special effect color generation submodule includes: For the flame effect, the formula Calculate the flame intensity and use the formula Adjust the base color of the flame to get the flame color , set the special effect color value Set to ;in 、 、 The base colors of the flames The red, green, and blue color components of the image; the clamp function is used to limit the value to between 0 and 1; For cloud effects, according to the flow direction of the clouds Adjust the noise coordinates, the formula is , and combined with the density of the clouds pass Generate cloud color , set the special effect color value Set to ; Among them, the smoothstep function is used to smoothly interpolate within the threshold range; For rain effects, based on the density of rain Rain color Perform binarization processing: When When the rain color Set to 1.0, otherwise the rain color is 0.0, generate the raindrop shape according to the noise value distribution, and then set the special effect color value Set to .

9. The cloud-native architecture-based dynamic 3D special effects rendering and synthesis system according to claim 5, characterized in that: The edge ablation algorithm in the edge correction submodule is shown in the following formula: ， in, is the ablation factor, based on the normalized coordinates of the current pixel calculate: ， in, and are normalized coordinates.

10. A dynamic 3D special effects rendering and synthesis method based on a cloud native architecture, based on the system according to any one of claims 1 to 9, characterized in that: The following steps are involved: S1. Split the original video into slices and dynamically allocate the slices according to the real-time load status of the cloud rendering node. Then, the multi-cloud rendering node renders the slices in parallel through the dynamic 3D special effects generation module to obtain the slice rendering results, and splices the slice rendering results to output a composite video file. ; The fragment rendering result includes a special effects frame sequence; S2. Decode the video stream and audio stream of the original video separately to obtain video frames and audio blocks. Align the special effect frames with the video frames and audio blocks through a two-way matching algorithm. Then, synthesize the special effect frames with the video frames through a video and special effect synthesis module to generate the final display frame. Reconstruct multi-scale features through the Laplacian pyramid, and use sub-pixel convolution to enhance high-frequency details to update the final display frame. Finally, write the updated final display frame and audio blocks into the synthesized video file in timestamp order through the audio and video encapsulator. .

Images