Virtual element synthesis control system and method applied to 3D animation

Abstract

The application discloses a virtual element synthesis control system and method applied to 3D animation, relates to the technical field of image processing, and extracts picture data in a rendering process and stores the picture data in a GPU when the GPU renders a scene picture in 3D animation; a mask prediction model is constructed by using a convolutional neural network, a three-dimensional space occupation mask generated when a virtual element interacts with a scene object is predicted through the mask prediction model; a learning network is constructed by using a neural network algorithm, and the corresponding outgoing light intensity of the incident light intensity in the picture is calculated; the contact shadow and the ambient light obstruction of the virtual element projected on the scene object in the 3D animation picture are calculated; the self-luminous color of the virtual element is calculated by using the outgoing light intensity, the contact shadow and the ambient light obstruction; after the virtual element synthesis of each frame picture in the 3D animation is carried out, the time sequence consistency is strengthened, and the 3D animation after the virtual element synthesis is output.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a virtual element synthesis control system and method applied to 3D animation. Background Technology

[0002] In the 1950s, *Fantasy Island* was considered the world's first 3D animated film, marking the initial formation of 3D animation technology. In 1962, Ivan Sutherland, the father of computer graphics, developed the Sketchpad system, achieving interactive drawing with a light pen for the first time. In the early 1970s, the development of 3D-assisted animation systems gradually emerged, such as the system at Ohio State University capable of shading. In the 1980s, with the deepening of computer graphics theory, 3D object modeling technology made progress, leading to key parameter interpolation methods, kinematic algorithms, and dynamic algorithms for animation generation. In 1989, James Cameron created the first liquid CG creature in *The Abyss*. In 1995, Pixar's *Toy Story*, as the first fully CG feature film, was released, heralding the arrival of the 3D animation era. Computer graphics provided the foundation for the synthesis of virtual elements in 3D animation, including technologies such as 3D modeling, lighting calculation, shadow processing, and material and texture representation. For example, deep learning-based texture synthesis technology can generate realistic natural surface textures, and improved lighting calculation and shadow processing algorithms enhance the visual effects of animation. Modern animation engines such as Maya, Unreal Engine, and Unity are constantly being optimized, introducing technologies such as ray tracing and real-time physical simulation, which significantly improves the realism of animation effects and rendering speed, and provides more powerful tools for virtual element compositing.

[0003] However, in the compositing of virtual elements in current 3D animation, it is still unavoidable that traditional green screen keying or manual ROTO cannot perfectly handle the interaction between semi-transparent, reflective, and refractive elements (such as smoke) and the scene. Virtual elements look like they are "stuck" onto the scene, without true optical blending; the lighting, shadows, and reflections of virtual elements need to be manually matched frame by frame to the lighting environment of the real scene, which is tedious and error-prone. Summary of the Invention

[0004] The purpose of this invention is to provide a virtual element synthesis control system and method for 3D animation, so as to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for controlling the compositing of virtual elements in 3D animation, the method comprising the following steps:

[0007] S100. When the GPU renders the scene in 3D animation, set the number of shaders output to the rendering target in the rendering channel to be greater than one; extract the scene data during the rendering process and store it in the GPU.

[0008] Furthermore, the specific steps for extracting image data during the rendering process and storing it in the GPU are as follows:

[0009] S101. When compositing virtual elements, extract scene images from the 3D animation, render the scene images, configure the number of shader outputs to the rendering target in the rendering channel to be greater than one, and extract the image data during the rendering process. The image data includes world position texture, world normal texture, depth texture, roughness texture, and HDR ambient lighting information. The world position texture stores the 3D coordinates (x, y, z) of each pixel in world space, and the world normal texture stores the surface unit normal vector (N) corresponding to each pixel. x N y， N z The depth texture stores the depth value D of each pixel, the roughness texture stores the material properties of the object surface, and the HDR ambient lighting information includes the scene's global illumination, environment cubemap, and spherical harmonics. All image data is stored in the GPU and forms a geometric buffer G-Buffer.

[0010] Shader outputs to multiple rendering targets, enabling the simultaneous acquisition of rich image data and avoiding data loss caused by single outputs, thus providing a foundation for subsequent multi-dimensional calculations. Key data such as world position, normals, and depth are stored in the GPU and form a geometry buffer (G-Buffer). The GPU's parallel processing capabilities significantly improve data reading and subsequent calculation efficiency. Furthermore, the integrated G-Buffer allows subsequent models and algorithms to directly access multi-dimensional data, reducing data transmission losses.

[0011] S200. Construct a mask prediction model using a convolutional neural network, taking the extracted image data as input; predict the three-dimensional space occupancy mask that should be generated when virtual elements interact with scene objects using the mask prediction model.

[0012] Furthermore, the specific steps for predicting the 3D spatial occupancy of the mask when virtual elements interact with scene objects using the mask prediction model are as follows:

[0013] S201. A mask prediction model is constructed using a lightweight editor-decoder convolutional neural network. The image data of virtual elements and corresponding scenes during the synthesis of historical virtual elements are extracted to train the mask prediction model. The mask prediction model learns the spatial occlusion and enclosing relationship between virtual elements and scene images through training.

[0014] S202. Merge the world position texture, world normal texture, and depth texture in the geometry buffer (G-Buffer) to form a multi-channel tensor. Specifically, extract three position values ​​from the world position texture, three normal vector values ​​from the world normal texture, and depth values ​​from the depth texture to obtain seven feature values. Construct a multi-channel tensor with seven features for each pixel. Input the multi-channel tensor into the mask prediction model.

[0015] The encoder downsamples to extract the spatial geometric features of the scene, and the decoder upsamples to predict the probability map.

[0016] The mask prediction model outputs a probability mask map M with the same resolution as the display screen. occ (p), where P represents a pixel, M occ (p)∈[0,1], the probability masking map represents the masking probability of a virtual element occupying space at pixel P and causing it to occupy the scene image;

[0017] A preset probability threshold is used to determine the occlusion probability of each pixel in the probability masking image. When the occlusion probability is greater than the probability threshold, it is determined that the corresponding pixel is occluded by a virtual element.

[0018] It adopts a lightweight encoder-decoder architecture, which reduces the consumption of computing resources while ensuring prediction accuracy, and adapts to the needs of real-time rendering of 3D animation. By judging the pixel occlusion status through probability threshold, it can handle complex scenes such as partial occlusion more delicately than traditional binary masking, and improve the realism of the synthesized image.

[0019] S300: Extract incident light intensity and illumination direction data from the image data, construct a learning network using a neural network algorithm, and calculate the outgoing light intensity corresponding to the incident light intensity in the image.

[0020] Furthermore, the specific steps for calculating the output light intensity corresponding to the incident light intensity in the image are as follows:

[0021] S301. For each pixel occluded by a virtual element, extract the incident light intensity and lighting direction from the HDR ambient lighting information based on the world position texture and world normal texture; construct a fully connected network using a neural network algorithm, input the incident light intensity into the fully connected network, and output the outgoing light intensity.

[0022] S302. The specific calculation of output light intensity in a fully connected network is as follows: Extract the absorption coefficient and scattering coefficient of the virtual element for light intensity, and use the absorption coefficient and scattering coefficient to calculate the transmittance of the virtual element. The formula is:

[0023] ;

[0024] In the formula, T(t) represents the transmittance of the virtual element, and β aβ represents the absorption coefficient. s The scattering coefficient is represented by t, which represents the time it takes for light to travel from entering the virtual element to exiting it; exp represents the exponential function.

[0025] Extract the depth values ​​of light entering and exiting the virtual element from the depth texture, calculate the depth difference between the two to obtain the light travel distance, and then calculate the outgoing light intensity using the incident light intensity, the virtual element's transmittance, and the travel distance. The formula is as follows:

[0026] ;

[0027] In the formula, L out L represents the intensity of the emitted light, k represents the distance the light travels, and L represents the distance the light travels. int This indicates the intensity of the incident light.

[0028] By constructing a learning network using neural networks and combining incident light intensity and direction extracted from HDR ambient lighting information, the system can adapt to the lighting characteristics of different scenes and accurately output the emitted light intensity. Absorption and scattering coefficients are introduced to calculate the transmittance of virtual elements, and the light intensity is corrected based on the light travel distance, ensuring that light intensity changes conform to physical laws and avoiding any unnatural blending of virtual elements with scene lighting.

[0029] S400: Calculate the contact shadows and ambient light occlusions of virtual elements projected onto scene objects in 3D animation.

[0030] Furthermore, the specific steps for calculating the contact shadows and ambient occlusion of virtual elements projected onto scene objects in a 3D animation are as follows:

[0031] S401, Based on the world position texture, at pixel P of the virtual element v Starting from a given location, emit a ray towards the light source in the scene. Set sampling points along the ray and extract the depth value D(P) of each sampling point in the scene. sample Extract the depth value D at the corresponding pixel position of the depth texture in the scene. scene When D(P) sample ) <D scene At that time, it is determined that the sampling point penetrates the scene object, and the virtual element pixel P v When an object in the scene occludes the area, it falls under the "contact shadow area".

[0032] When D(P) sample )≥D scene When determining the virtual element pixel P v It does not belong to the "contact shadow area";

[0033] S402. Measure the ray distance between the ray and the point of contact between the ray and the scene object when the ray penetrates the scene object. Professionals set the maximum effective distance according to the requirements of the virtual elements, and use the maximum effective distance to calculate the shadow intensity within the contact shadow area. The formula is:

[0034] ;

[0035] In the formula, S contact G represents the shadow intensity within the contact shadow area, and G represents the ray distance between the ray and the point of contact with the scene object. max This represents the maximum effective distance, and n represents the shading transition coefficient;

[0036] S403, In the scene, the same pixel P of the virtual element v Centered on pixel P v Sampling directions are set within the normal hemisphere. For each sampling direction, the same depth value comparison method as in S401 is used to determine whether the virtual element pixel is occluded by scene objects. After the determination, the number of sampling directions that are determined to be occluded by scene objects is obtained. The proportion of the number of sampling directions that are determined to be occluded by scene objects in all sampling directions is calculated as the ambient light occlusion factor AO(p).

[0037] By emitting rays towards the light source and comparing depth values, the contact shadow range of virtual elements can be accurately determined, solving the problems of edge blurring or positional deviation in traditional shadow calculations. It supports setting the maximum effective distance and shadow transition coefficient, allowing adjustment of shadow effects based on the characteristics of virtual elements; the ambient light occlusion factor is calculated by sampling direction proportion, realistically simulating the effect of ambient light being blocked in a scene, enhancing the sense of depth and three-dimensionality of the image.

[0038] S500 calculates the self-emission color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion; extracts the background color and the spatial proportion mask predicted by the mask prediction model, and performs physically based volumetric blending to output the final color of the image.

[0039] Furthermore, the specific steps for performing physically based volumetric blending to output the final color of the image are as follows:

[0040] S501. Calculate the self-emission color of virtual elements using emitted light intensity, contact shadows, and ambient occlusion. The formula is:

[0041] ;

[0042] In the formula, C vol Indicates the self-illuminating color of the virtual element;

[0043] S502. Extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physically based volumetric blending to output the final color of the image. The formula is:

[0044] ;

[0045] In the formula, C final (p) represents the final color of the image, C bg (p) indicates the background color.

[0046] By combining emitted light intensity, contact shadows, and ambient light occlusion to calculate self-emission color, the glowing effect of virtual elements is highly matched with the surrounding ambient lighting and shadow conditions, improving visual consistency. Through weighted mixing of mask and background color, following the principles of physical optics, a natural fusion of virtual elements and scene imagery is achieved, avoiding compositing artifacts and making the final image more realistic.

[0047] S600 performs virtual element compositing on each frame of the 3D animation, then enhances the temporal consistency, and outputs the 3D animation after virtual element compositing.

[0048] Furthermore, the specific steps for outputting the 3D animation after compositing virtual elements are as follows:

[0049] S601. Analyze each frame of the 3D animation based on the time sequence, extract the position of each pixel in the previous frame and the position of each pixel in the current frame, obtain the scene motion vector based on the difference between the pixel positions of adjacent frames, and use the scene motion vector to reproject the final color of the image output in the previous frame to the current frame.

[0050] S602. Mix the final color of the image calculated and output in the current frame with the final color of the image after reprojection, using the following formula:

[0051] ;

[0052] In the formula, C final (p) t C represents the final image after blending. final (p) traw C represents the final color of the image calculated and output in the current frame. final (p) trep This represents the final color of the image to which the image is reprojected; α represents the blending weight, which is set by the staff.

[0053] S603. The final color of each frame is mixed to obtain the final mixed image of each frame, and then integrated to obtain the 3D animation after the virtual elements are synthesized. By setting the blending weight to integrate the colors of the current frame and the reprojection frame, the consistency of timing is further enhanced, the instability of the image caused by single frame calculation errors is avoided, and the visual coherence of the overall animation is improved.

[0054] A virtual element compositing control system for 3D animation, comprising a data acquisition module, a model training module, a light intensity calculation module, a shadow analysis module, a compositing module, and a timing optimization module;

[0055] The data acquisition module is used to set the number of shader outputs to the rendering target in the rendering channel to be greater than one when the GPU renders the scene in 3D animation; extract the scene data during the rendering process and store it in the GPU;

[0056] The model training module is used to construct a mask prediction model using a convolutional neural network, taking the extracted image data as input; the mask prediction model predicts the three-dimensional space occupancy mask that should be generated when virtual elements interact with scene objects.

[0057] The light intensity calculation module is used to extract incident light intensity and illumination direction data from the image data, and to construct a learning network using a neural network algorithm to calculate the outgoing light intensity corresponding to the incident light intensity in the image.

[0058] The shadow analysis module is used to calculate the contact shadows and ambient light occlusions of virtual elements projected onto scene objects in 3D animation images;

[0059] The compositing module is used to calculate the self-emitting color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion; extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physical volume blending to output the final color of the image.

[0060] The timing optimization module is used to enhance the timing consistency of each frame in the 3D animation by compositing virtual elements and then outputting the 3D animation after virtual element compositing.

[0061] The shadow analysis module includes a shadow intensity unit and an ambient light occlusion factor unit;

[0062] The shadow intensity unit is used to measure the distance between the ray and the contact point of the scene object when the ray penetrates the scene object. Professionals set the maximum effective distance according to the needs of virtual elements and use the maximum effective distance to calculate the shadow intensity in the contact shadow area.

[0063] The ambient light occlusion factor unit is used to calculate the proportion of all sampling directions that are determined to be occluded by scene objects as the ambient light occlusion factor.

[0064] The synthesis module includes a self-illuminating color unit and a synthesis unit;

[0065] The self-emitting color unit is used to calculate the self-emitting color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion.

[0066] The compositing unit is used to extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physical volume blending to output the final color of the image.

[0067] Compared with the prior art, the beneficial effects of the present invention are:

[0068] 1. This invention integrates physical optics principles and precise spatial calculations throughout the entire process, enabling virtual elements and 3D scenes to be highly integrated in terms of lighting, shadows, and spatial position, thus solving the pain point of "strong virtuality" in traditional compositing.

[0069] 2. This invention employs a lightweight neural network model, GPU parallel storage and processing, and optimized ray tracing and sampling algorithms to control computational complexity while ensuring the effect, thus meeting the requirements for real-time 3D animation synthesis.

[0070] 3. This invention supports the customization of multiple parameters, which can be adapted to different types of virtual elements and scene styles, thus improving the versatility of the solution. Attached Figure Description

[0071] Figure 1 This is a module distribution diagram of the virtual element compositing control system of the present invention applied to 3D animation;

[0072] Figure 2 This is a schematic diagram illustrating the steps of the virtual element synthesis control method of the present invention applied to 3D animation. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] Example: Figures 1-2 As shown, the present invention provides a technical solution.

[0075] A method for controlling the compositing of virtual elements in 3D animation, the method comprising the following steps:

[0076] S100. When the GPU renders the scene in 3D animation, set the number of shaders output to the rendering target in the rendering channel to be greater than one; extract the scene data during the rendering process and store it in the GPU.

[0077] The specific steps for extracting image data during the rendering process and storing it in the GPU are as follows:

[0078] S101. When compositing virtual elements, extract scene images from the 3D animation, render the scene images, configure the number of shader outputs to the rendering target in the rendering channel to be greater than one, and extract the image data during the rendering process. The image data includes world position texture, world normal texture, depth texture, roughness texture, and HDR ambient lighting information. The world position texture stores the 3D coordinates (x, y, z) of each pixel in world space, and the world normal texture stores the surface unit normal vector (N) corresponding to each pixel. x N y， N z The depth texture stores the depth value D of each pixel, the roughness texture stores the material properties of the object surface, and the HDR ambient lighting information includes the scene's global illumination, environment cubemap, and spherical harmonics. All image data is stored in the GPU and forms a geometric buffer G-Buffer.

[0079] Shader outputs to multiple rendering targets, enabling the simultaneous acquisition of rich image data and avoiding data loss caused by single outputs, thus providing a foundation for subsequent multi-dimensional calculations. Key data such as world position, normals, and depth are stored in the GPU and form a geometry buffer (G-Buffer). The GPU's parallel processing capabilities significantly improve data reading and subsequent calculation efficiency. Furthermore, the integrated G-Buffer allows subsequent models and algorithms to directly access multi-dimensional data, reducing data transmission losses.

[0080] S200. Construct a mask prediction model using a convolutional neural network, taking the extracted image data as input; predict the three-dimensional space occupancy mask that should be generated when virtual elements interact with scene objects using the mask prediction model.

[0081] The specific steps for predicting the 3D spatial occupancy of the mask when virtual elements interact with scene objects using a mask prediction model are as follows:

[0082] S201. A mask prediction model is constructed using a lightweight editor-decoder convolutional neural network. The image data of virtual elements and corresponding scenes during the synthesis of historical virtual elements are extracted to train the mask prediction model. The mask prediction model learns the spatial occlusion and enclosing relationship between virtual elements and scene images through training.

[0083] S202. Merge the world position texture, world normal texture, and depth texture in the geometry buffer (G-Buffer) to form a multi-channel tensor. Specifically, extract three position values ​​from the world position texture, three normal vector values ​​from the world normal texture, and depth values ​​from the depth texture to obtain seven feature values. Construct a multi-channel tensor with seven features for each pixel. Input the multi-channel tensor into the mask prediction model.

[0084] The encoder downsamples to extract the spatial geometric features of the scene, and the decoder upsamples to predict the probability map.

[0085] The mask prediction model outputs a probability mask map M with the same resolution as the display screen. occ (p), where P represents a pixel, M occ (p)∈[0,1], the probability masking map represents the masking probability of a virtual element occupying space at pixel P and causing it to occupy the scene image;

[0086] A preset probability threshold is used to determine the occlusion probability of each pixel in the probability masking image. When the occlusion probability is greater than the probability threshold, it is determined that the corresponding pixel is occluded by a virtual element.

[0087] It adopts a lightweight encoder-decoder architecture, which reduces the consumption of computing resources while ensuring prediction accuracy, and adapts to the needs of real-time rendering of 3D animation. By judging the pixel occlusion status through probability threshold, it can handle complex scenes such as partial occlusion more delicately than traditional binary masking, and improve the realism of the synthesized image.

[0088] S300: Extract incident light intensity and illumination direction data from the image data, construct a learning network using a neural network algorithm, and calculate the outgoing light intensity corresponding to the incident light intensity in the image.

[0089] The specific steps for calculating the output light intensity corresponding to the incident light intensity in the image are as follows:

[0090] S301. For each pixel occluded by a virtual element, extract the incident light intensity and lighting direction from the HDR ambient lighting information based on the world position texture and world normal texture; construct a fully connected network using a neural network algorithm, input the incident light intensity into the fully connected network, and output the outgoing light intensity.

[0091] S302. The specific calculation of output light intensity in a fully connected network is as follows: Extract the absorption coefficient and scattering coefficient of the virtual element for light intensity, and use the absorption coefficient and scattering coefficient to calculate the transmittance of the virtual element. The formula is:

[0092] ;

[0093] In the formula, T(t) represents the transmittance of the virtual element, and β aβ represents the absorption coefficient. s The scattering coefficient is represented by t, which represents the time it takes for light to travel from entering the virtual element to exiting it; exp represents the exponential function.

[0094] Extract the depth values ​​of light entering and exiting the virtual element from the depth texture, calculate the depth difference between the two to obtain the light travel distance, and then calculate the outgoing light intensity using the incident light intensity, the virtual element's transmittance, and the travel distance. The formula is as follows:

[0095] ;

[0096] In the formula, L out L represents the intensity of the emitted light, k represents the distance the light travels, and L represents the distance the light travels. int This indicates the intensity of the incident light.

[0097] By constructing a learning network using neural networks and combining incident light intensity and direction extracted from HDR ambient lighting information, the system can adapt to the lighting characteristics of different scenes and accurately output the emitted light intensity. Absorption and scattering coefficients are introduced to calculate the transmittance of virtual elements, and the light intensity is corrected based on the light travel distance, ensuring that light intensity changes conform to physical laws and avoiding any unnatural blending of virtual elements with scene lighting.

[0098] S400: Calculate the contact shadows and ambient light occlusions of virtual elements projected onto scene objects in 3D animation.

[0099] The specific steps for calculating the contact shadows and ambient occlusion of virtual elements projected onto scene objects in a 3D animation are as follows:

[0100] S401, Based on the world position texture, at pixel P of the virtual element v Starting from a given location, emit a ray towards the light source in the scene. Set sampling points along the ray and extract the depth value D(P) of each sampling point in the scene. sample Extract the depth value D at the corresponding pixel position of the depth texture in the scene. scene When D(P) sample ) <D scene At that time, it is determined that the sampling point penetrates the scene object, and the virtual element pixel P v When an object in the scene occludes the area, it falls under the "contact shadow area".

[0101] When D(P) sample )≥D scene When determining the virtual element pixel P v It does not belong to the "contact shadow area";

[0102] S402. Measure the ray distance between the ray and the point of contact between the ray and the scene object when the ray penetrates the scene object. Professionals set the maximum effective distance according to the requirements of the virtual elements, and use the maximum effective distance to calculate the shadow intensity within the contact shadow area. The formula is:

[0103] ;

[0104] In the formula, S contact G represents the shadow intensity within the contact shadow area, and G represents the ray distance between the ray and the point of contact with the scene object. max This represents the maximum effective distance, and n represents the shading transition coefficient;

[0105] S403, In the scene, the same pixel P of the virtual element v Centered on pixel P v Sampling directions are set within the normal hemisphere. For each sampling direction, the same depth value comparison method as in S401 is used to determine whether the virtual element pixel is occluded by scene objects. After the determination, the number of sampling directions that are determined to be occluded by scene objects is obtained. The proportion of the number of sampling directions that are determined to be occluded by scene objects in all sampling directions is calculated as the ambient light occlusion factor AO(p).

[0106] By emitting rays towards the light source and comparing depth values, the contact shadow range of virtual elements can be accurately determined, solving the problems of edge blurring or positional deviation in traditional shadow calculations. It supports setting the maximum effective distance and shadow transition coefficient, allowing adjustment of shadow effects based on the characteristics of virtual elements; the ambient light occlusion factor is calculated by sampling direction proportion, realistically simulating the effect of ambient light being blocked in a scene, enhancing the sense of depth and three-dimensionality of the image.

[0107] S500 calculates the self-emission color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion; extracts the background color and the spatial proportion mask predicted by the mask prediction model, and performs physically based volumetric blending to output the final color of the image.

[0108] The specific steps for performing physically based volumetric blending to output the final color of the image are as follows:

[0109] S501. Calculate the self-emission color of virtual elements using emitted light intensity, contact shadows, and ambient occlusion. The formula is:

[0110] ;

[0111] In the formula, C vol This represents the self-illumination color of the virtual element, and the calculated self-illumination color is set to the RGB three-channel color.

[0112] S502. Extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physically based volumetric blending to output the final color of the image. The formula is:

[0113] ;

[0114] In the formula, C final (p) represents the final color of the image, C bg (p) indicates the background color.

[0115] By combining emitted light intensity, contact shadows, and ambient light occlusion to calculate self-emission color, the glowing effect of virtual elements is highly matched with the surrounding ambient lighting and shadow conditions, improving visual consistency. Through weighted mixing of mask and background color, following the principles of physical optics, a natural fusion of virtual elements and scene imagery is achieved, avoiding compositing artifacts and making the final image more realistic.

[0116] S600 performs virtual element compositing on each frame of the 3D animation, then enhances the temporal consistency, and outputs the 3D animation after virtual element compositing.

[0117] The specific steps for outputting the 3D animation after compositing virtual elements are as follows:

[0118] S601. Analyze each frame of the 3D animation based on the time sequence, extract the position of each pixel in the previous frame and the position of each pixel in the current frame, obtain the scene motion vector based on the difference between the pixel positions of adjacent frames, and use the scene motion vector to reproject the final color of the image output in the previous frame to the current frame.

[0119] S602. Mix the final color of the image calculated and output in the current frame with the final color of the image after reprojection, using the following formula:

[0120] ;

[0121] In the formula, C final (p) t C represents the final image after blending. final (p) traw C represents the final color of the image calculated and output in the current frame. final (p) trep This represents the final color of the image to which the image is reprojected; α represents the blending weight, which is set by the staff.

[0122] S603. The final color of each frame is mixed to obtain the final mixed image of each frame, and then integrated to obtain the 3D animation after the virtual elements are synthesized. By setting the blending weight to integrate the colors of the current frame and the reprojection frame, the consistency of timing is further enhanced, the instability of the image caused by single frame calculation errors is avoided, and the visual coherence of the overall animation is improved.

[0123] A virtual element compositing control system for 3D animation, comprising a data acquisition module, a model training module, a light intensity calculation module, a shadow analysis module, a compositing module, and a timing optimization module;

[0124] The data acquisition module is used to set the number of shader outputs to the rendering target in the rendering channel to be greater than one when the GPU renders the scene in 3D animation; extract the scene data during the rendering process and store it in the GPU;

[0125] The model training module is used to construct a mask prediction model using a convolutional neural network, taking the extracted image data as input; the mask prediction model predicts the three-dimensional space occupancy mask that should be generated when virtual elements interact with scene objects.

[0126] The light intensity calculation module is used to extract incident light intensity and illumination direction data from the image data, and to construct a learning network using a neural network algorithm to calculate the outgoing light intensity corresponding to the incident light intensity in the image.

[0127] The shadow analysis module is used to calculate the contact shadows and ambient light occlusions of virtual elements projected onto scene objects in 3D animation images;

[0128] The compositing module is used to calculate the self-emitting color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion; extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physical volume blending to output the final color of the image.

[0129] The timing optimization module is used to enhance the timing consistency of each frame in the 3D animation by compositing virtual elements and then outputting the 3D animation after virtual element compositing.

[0130] The shadow analysis module includes a shadow intensity unit and an ambient light occlusion factor unit;

[0131] The shadow intensity unit is used to measure the distance between the ray and the contact point of the scene object when the ray penetrates the scene object. Professionals set the maximum effective distance according to the needs of virtual elements and use the maximum effective distance to calculate the shadow intensity in the contact shadow area.

[0132] The ambient light occlusion factor unit is used to calculate the proportion of all sampling directions that are determined to be occluded by scene objects as the ambient light occlusion factor.

[0133] The synthesis module includes a self-illuminating color unit and a synthesis unit;

[0134] The self-emitting color unit is used to calculate the self-emitting color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion.

[0135] The compositing unit is used to extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physical volume blending to output the final color of the image.

[0136] Example: Taking the synthesis of virtual flame elements with indoor scenes (including tables, chairs, windows, and chandelier light sources) in 3D animation as an example, 5,000 sets of historical synthesis data of "virtual flame + different indoor scenes" (including scene images, flame positions, real occlusion masking, and lighting parameters) were collected for training of the masking prediction model.

[0137] Create a custom rendering pipeline in the Unity engine, and set the output to 3 rendering targets (corresponding to "position-normal-depth", "roughness" and "HDR ambient lighting") through HLSL shaders, ensuring that the number of shader outputs is greater than 1; collect image data and build a 4-channel G-Buffer;

[0138] A lightweight U-Net (4 layers of downsampling in the encoder and 4 layers of upsampling in the decoder) is used as the editor-decoder convolutional neural network. The input dimension is 7 (3 position values ​​+ 3 normal vector values ​​+ 1 depth value), and the output dimension is 1 (masking probability).

[0139] The multi-channel tensor is input into the trained model. The encoder extracts scene spatial geometric features (such as the outlines of tables and chairs, and the planar structure of windows) through 3×3 convolution downsampling. The decoder outputs a 1920×1080 probability mask map M through transposed convolution upsampling. occ (p) (e.g., pixel M at the center of the flame) occ (p)=0.98, the table and chairs are not obscured (M pixels). occ (p)=0.02).

[0140] The preset probability threshold is 0.5. When Mocc(p) > 0.5, it is determined that the pixel is occluded by the virtual flame.

[0141] Based on the world location texture and world normal texture, extract the incident light intensity L from the HDR ambient lighting information. int =450 lux (the chandelier light source direction is (0,-1,0), that is, vertically downward).

[0142] The absorption coefficient β of the virtual flame a =0.3 (unit: cm⁻¹), scattering coefficient β s =0.5, according to the formula The calculated value is T(t)≈0.000335; the final calculated output light intensity is L. out =0.000335×0.5×450×10≈0.754lux;

[0143] Assume the calculated shadow intensity is 0.64 and the ambient light occlusion factor is 0.375; C vol =0.754×(1−0.64)×0.375≈0.754×0.36×0.375≈0.102; set to RGB three-channel color approximately (0.1,0.1,0.1);

[0144] Extract the scene background color Cbg(p) (e.g., Pv corresponds to a light wood color desktop background color, RGB value (0.8, 0.6, 0.4)), and the spatial proportion mask Mocc(p) output by the mask prediction model is 0.98.

[0145] The final color of the image is obtained by compositing virtual elements = (0.116, 0.112, 0.108).

[0146] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A virtual element compositing control method applied to 3D animation, characterized in that: The method includes the following steps: S100. When the GPU renders the scene in 3D animation, set the number of shaders output to the rendering target in the rendering channel to be greater than one; extract the scene data during the rendering process and store it in the GPU. S200. Construct a mask prediction model using a convolutional neural network, taking the extracted image data as input; predict the three-dimensional space occupancy mask that should be generated when virtual elements interact with scene objects using the mask prediction model. The specific steps of predicting the 3D spatial occupancy mask generated when virtual elements interact with scene objects using the mask prediction model are as follows: The world position texture, world normal texture, and depth texture in the geometry buffer (G-Buffer) are merged to form a multi-channel tensor. Specifically, three position values ​​from the world position texture, three normal values ​​from the world normal texture, and depth values ​​from the depth texture are extracted and integrated to obtain seven feature values. A multi-channel tensor with seven features is constructed for each pixel. The multi-channel tensor is then input into the mask prediction model. The encoder downsamples to extract the spatial geometric features of the scene, and the decoder upsamples to predict the probability map. The mask prediction model outputs a probability mask map M with the same resolution as the display screen. occ (p), where P represents a pixel, M occ (p)∈[0,1], the probability masking map represents the masking probability of a virtual element occupying the space at pixel P and causing it to occupy the scene; A preset probability threshold is used to determine the masking probability of each pixel in the probability masking image. When the masking probability is greater than the probability threshold, it is determined that the corresponding pixel is masked by a virtual element. S300: Extract incident light intensity and illumination direction data from the image data, construct a learning network using a neural network algorithm, and calculate the outgoing light intensity corresponding to the incident light intensity in the image. The specific output light intensity corresponding to the incident light intensity in the calculation image is: The incident light intensity and illumination direction are extracted from the HDR ambient lighting information; a fully connected network is constructed using a neural network algorithm, and the incident light intensity is input into the fully connected network to output the outgoing light intensity; The specific calculation of output light intensity in a fully connected network is as follows: extract the absorption coefficient and scattering coefficient of the virtual element for light intensity, and use the absorption coefficient and scattering coefficient to calculate the transmittance of the virtual element; Extract the depth values ​​of light entering and exiting virtual elements from the depth texture, calculate the difference between the two depth values ​​to obtain the distance the light travels, and use the incident light intensity, the transmittance of the virtual element, and the distance traveled to calculate the outgoing light intensity. S400: Calculate the contact shadows and ambient light occlusions of virtual elements projected onto scene objects in 3D animation. S500 calculates the self-emission color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion; extracts the background color and the spatial proportion mask predicted by the mask prediction model, and performs physically based volumetric blending to output the final color of the image. S600 performs virtual element compositing on each frame of the 3D animation, then enhances the temporal consistency, and outputs the 3D animation after virtual element compositing.

2. The virtual element compositing control method for 3D animation according to claim 1, characterized in that: The specific steps in S100 for extracting image data during the rendering process and storing it in the GPU are as follows: S101. When compositing virtual elements, extract scene images from the 3D animation, render the scene images, configure the number of shader outputs to the rendering target in the rendering channel to be greater than one, and extract the image data during the rendering process. The image data includes world position texture, world normal texture, depth texture, roughness texture, and HDR ambient lighting information. The world position texture stores the 3D coordinates (x, y, z) of each pixel in world space, and the world normal texture stores the surface unit normal vector (N) corresponding to each pixel. x N y， N z The depth texture stores the depth value D of each pixel, the roughness texture stores the material properties of the object surface, and the HDR ambient lighting information includes the scene's global illumination, environment cubemap, and spherical harmonics. All image data is stored in the GPU and forms a geometric buffer G-Buffer.

3. The virtual element compositing control method for 3D animation according to claim 2, characterized in that: The specific steps in S200 for predicting the three-dimensional space occupation of the mask when virtual elements interact with scene objects using the mask prediction model are as follows: S201. A mask prediction model is constructed using a lightweight editor-decoder convolutional neural network. The mask prediction model is trained by extracting the image data of virtual elements and corresponding scenes during the synthesis of historical virtual elements. The mask prediction model learns the spatial occlusion and enclosing relationship between virtual elements and scene images through training.

4. The virtual element compositing control method for 3D animation according to claim 3, characterized in that: The specific steps for calculating the outgoing light intensity corresponding to the incident light intensity in the image in S300 are as follows: S301. For each pixel occluded by a virtual element, extract the incident light intensity and lighting direction from the HDR ambient lighting information based on the world position texture and world normal texture; construct a fully connected network using a neural network algorithm, input the incident light intensity into the fully connected network, and output the outgoing light intensity. S302. The specific calculation of output light intensity in a fully connected network is as follows: Extract the absorption coefficient and scattering coefficient of the virtual element for light intensity, and use the absorption coefficient and scattering coefficient to calculate the transmittance of the virtual element. The formula is: ； In the formula, T(t) represents the transmittance of the virtual element, and β a β represents the absorption coefficient. s The scattering coefficient is represented by t, which represents the time it takes for light to travel from entering the virtual element to exiting it; exp represents the exponential function. Extract the depth values ​​of light entering and exiting the virtual element from the depth texture, calculate the depth difference between the two to obtain the light travel distance, and then calculate the outgoing light intensity using the incident light intensity, the virtual element's transmittance, and the travel distance. The formula is as follows: ； In the formula, L out L represents the intensity of the emitted light, k represents the distance the light travels, and L represents the distance the light travels. int This indicates the intensity of the incident light.

5. The virtual element compositing control method for 3D animation according to claim 4, characterized in that: The specific steps in S400 for calculating the contact shadows and ambient occlusion of virtual elements projected onto scene objects in a 3D animation are as follows: S401, Based on the world position texture, at pixel P of the virtual element v Starting from a given location, emit a ray towards the light source in the scene. Set sampling points along the ray and extract the depth value D(P) of each sampling point in the scene. sample Extract the depth value D at the pixel location corresponding to the depth texture in the scene. scene When D(P) sample ) <D scene At that time, it is determined that the sampling point penetrates the scene object, and the virtual element pixel P v When an object in the scene occludes the area, it falls under the "contact shadow area". When D(P) sample )≥D scene When determining the virtual element pixel P v It does not belong to the "contact shadow area"; S402. Measure the ray distance between the ray and the point of contact between the ray and the scene object when the ray penetrates the scene object. Professionals set the maximum effective distance according to the requirements of the virtual elements, and use the maximum effective distance to calculate the shadow intensity within the contact shadow area. The formula is: ； In the formula, S contact G represents the shadow intensity within the contact shadow area, and G represents the ray distance between the ray and the point of contact with the scene object. max This represents the maximum effective distance, and n represents the shading transition coefficient; S403, In the scene, the same pixel P of the virtual element v Centered on pixel P v The sampling direction is set within the normal hemisphere. For each sampling direction, the same depth value comparison method as in S401 is used to determine whether the virtual element pixel is occluded by scene objects. After determining the number of sampling directions that are determined to be occluded by scene objects, the percentage of the total number of sampling directions that are determined to be occluded by scene objects is calculated as the ambient light occlusion factor AO(p).

6. The virtual element compositing control method for 3D animation according to claim 5, characterized in that: The specific steps for performing physical volumetric blending to output the final color of the image in the S500 are as follows: S501. Calculate the self-emission color of virtual elements using emitted light intensity, contact shadows, and ambient occlusion. The formula is: ； In the formula, C vol Indicates the self-illuminating color of the virtual element; S502. Extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physically based volumetric blending to output the final color of the image. The formula is: ； In the formula, C final (p) represents the final color of the image, C bg (p) indicates the background color.

7. The virtual element compositing control method for 3D animation according to claim 6, characterized in that: The specific steps for outputting the 3D animation after virtual element synthesis in S600 are as follows: S601. Analyze each frame of the 3D animation based on the time sequence, extract the position of each pixel in the previous frame and the position of each pixel in the current frame, obtain the scene motion vector based on the difference between the pixel positions of adjacent frames, and use the scene motion vector to reproject the final color of the image output in the previous frame to the current frame. S602. Mix the final color of the image calculated and output in the current frame with the final color of the image after reprojection, using the following formula: ； In the formula, C final (p) t C represents the final image after blending. final (p) traw C represents the final color of the image calculated and output in the current frame. final (p) trep This represents the final color of the image to which the image is reprojected; α represents the blending weight, which is set by the staff. S603. Mix the final colors of each frame to obtain the final mixed image of each frame, and integrate them to obtain the 3D animation after outputting virtual elements.

8. A virtual element compositing control system applied to 3D animation, characterized in that: The virtual element synthesis control system includes a data acquisition module, a model training module, a light intensity calculation module, a shadow analysis module, a synthesis module, and a timing optimization module; The data acquisition module is used to set the number of shader outputs to the rendering target in the rendering channel to be greater than one when the GPU renders the scene in 3D animation; extract the scene data during the rendering process and store it in the GPU; The model training module is used to construct a mask prediction model using a convolutional neural network, taking the extracted image data as input; the mask prediction model predicts the three-dimensional space occupancy mask that should be generated when virtual elements interact with scene objects. The light intensity calculation module is used to extract incident light intensity and illumination direction data from the image data, and to construct a learning network using a neural network algorithm to calculate the outgoing light intensity corresponding to the incident light intensity in the image. The shadow analysis module is used to calculate the contact shadows and ambient light occlusions of virtual elements projected onto scene objects in 3D animation images; The compositing module is used to calculate the self-emitting color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion; extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physical volume blending to output the final color of the image. The timing optimization module is used to enhance the timing consistency of each frame in the 3D animation by compositing virtual elements and then outputting the 3D animation after virtual element compositing.

9. The virtual element compositing control system for 3D animation according to claim 8, characterized in that: The shadow analysis module includes a shadow intensity unit and an ambient light occlusion factor unit; The shadow intensity unit is used to measure the distance between the ray and the contact point of the scene object when the ray penetrates the scene object. Professionals set the maximum effective distance according to the needs of virtual elements and use the maximum effective distance to calculate the shadow intensity in the contact shadow area. The ambient light occlusion factor unit is used to calculate the proportion of all sampling directions that are determined to be occluded by scene objects as the ambient light occlusion factor.

10. The virtual element compositing control system for 3D animation according to claim 8, characterized in that: The synthesis module includes a self-illuminating color unit and a synthesis unit; The self-emitting color unit is used to calculate the self-emitting color of virtual elements using emitted light intensity, contact shadows, and ambient light occlusion. The compositing unit is used to extract the background color and the spatial proportion mask predicted by the mask prediction model, and perform physical volume blending to output the final color of the image.

Images