Dynamic digital human high-fidelity real-time rendering method and device, equipment and storage medium

Abstract

This application relates to a high-fidelity real-time rendering method, apparatus, device, and storage medium for dynamic digital humans. It includes receiving multimodal driving signals, mapping them to expression and pose parameters, inputting a parameterized human model to generate standard spatial geometry and skinning weights, and constructing a three-dimensional Gaussian sputtering field to deform to the target pose to obtain dynamic geometric data; arranging sparse virtual cameras in the target pose space, and simultaneously rendering and generating a sparse reference view set containing color and depth within a single rendering call; determining dense virtual viewpoints based on the display device's viewpoint parameters, and generating depth range textures for each dense viewpoint through downsampling depth reprojection and point sputtering techniques; determining ray step intervals using the depth range textures, sampling along the rays and fusing signed distance values ​​truncated from multiple viewpoints to generate surface position textures; projecting the surface positions onto sparse view sampled colors, fusing to generate a dense view color image, and adapting it for output to different display devices.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a method, apparatus, device and storage medium for high-fidelity real-time rendering of dynamic digital humans. Background Technology

[0002] With the rapid development of metaverse, virtual reality, and intelligent interaction technologies, dynamic digital humans, as a medium connecting the physical world and digital space, are playing an increasingly important role in scenarios such as virtual live streaming, film and television entertainment, and intelligent customer service. The industry's requirements for digital humans have upgraded from static model presentations to highly realistic, emotionally natural, and interactive dynamic intelligent agents.

[0003] Currently, existing technologies primarily rely on light field display based on independent rendering of multiple views. This involves first deploying dozens or even hundreds of virtual cameras in virtual space, performing complete geometric processing, lighting calculations, and pixel shading for each camera's perspective to generate color and depth images for all viewpoints. These images are then synthesized into a stereoscopic light field view through light field encoding. To improve rendering efficiency, geometry shaders or GPU instantiation techniques can be used to accelerate multi-view rendering, reducing communication overhead between the CPU and GPU by copying primitives or submitting draw calls in batches. In terms of geometric representation, traditional triangular mesh models are typically used, combined with physically based rendering (PBR) materials for lighting calculations.

[0004] The above-mentioned technologies require independent and complete geometric processing and lighting calculations for each viewpoint. As the number of viewpoints increases, the computational load increases linearly. When the number of viewpoints reaches dozens, it is difficult to achieve real-time frame rates even with GPU acceleration. This is especially true when dealing with complex skin subsurface scattering and hair lighting models of digital humans, where the computational overhead is even greater, making it inconvenient to use.

[0005] Based on this, this application provides a method, apparatus, device, and storage medium for high-fidelity real-time rendering of dynamic digital humans. Summary of the Invention

[0006] To address the issue that the computational load increases linearly with the number of viewpoints due to the need for independent and complete geometric processing and lighting calculations for each viewpoint, and that even with GPU acceleration, it is difficult to achieve a real-time frame rate when the number of viewpoints reaches dozens, especially when dealing with complex skin subsurface scattering and hair lighting models of digital humans, where the computational overhead is even greater and the system is inconvenient to use, this application provides a method, apparatus, device, and storage medium for high-fidelity real-time rendering of dynamic digital humans.

[0007] Firstly, this application provides a high-fidelity real-time rendering method for dynamic digital humans, employing the following technical solution: including: The system receives multimodal driving signals, maps them to facial expression parameters and posture parameters, inputs them into a parameterized human body model, and generates standard spatial geometry and skin weights. Based on the standard spatial geometry, it constructs a three-dimensional scene representation and deforms it to the target posture space according to the posture parameters and the skin weights to obtain dynamic geometric data. Sparse virtual cameras are deployed in the target pose space. A deferred rendering pipeline is used to render the views of all sparse virtual cameras simultaneously within a single draw call, generating a sparse reference view set containing color textures and depth textures. The dense virtual viewpoint is determined based on the viewpoint parameters of the target display device. Based on the depth texture in the sparse reference view set, each sparse viewpoint pixel is projected to the dense virtual viewpoint screen space by downsampling depth reprojection. The depth contribution of each pixel is accumulated by point splashing technology to generate the depth range texture of each dense virtual viewpoint. For each dense virtual viewpoint, the ray stepping interval is determined by the depth range texture, sampling is performed along the ray direction, and the sampling points are transformed to the observation space of each sparse viewpoint. The truncated signed distance value is calculated using the depth texture, and the scene surface position is located to generate the surface position texture of each dense virtual viewpoint. The surface position texture is projected onto the screen space of each sparse viewpoint, sparse color textures are sampled, color weights are calculated based on depth differences and fused to generate the final color image of the target dense virtual viewpoint; according to the display device type, the final color image is synthesized into a suitable display image and rendered.

[0008] Preferably, the received multimodal driving signal is mapped to expression parameters and posture parameters, input to a parameterized human body model, and standard spatial geometry and skin weights are generated; a three-dimensional scene representation is constructed based on the standard spatial geometry, and deformed to the target posture space according to the posture parameters and the skin weights to obtain dynamic geometric data, including: The system receives multimodal driving signals, including audio streams, video streams, or inertial sensor data streams; extracts Mel frequency cepstral coefficients and fundamental frequency features from the audio stream, extracts two-dimensional coordinates of facial landmarks and head posture from the video stream, and extracts limb joint rotation angles from the inertial sensor data; and fuses the extracted feature vectors through a temporal convolutional network to generate a low-dimensional driving latent code. The driving latent code is input into a pre-trained multilayer perceptron and mapped to expression parameters and posture parameters, respectively. The expression parameters are used to control the activation level of facial muscle units, and the posture parameters are used to control the axial rotation of the whole skeleton. The facial expression parameters and pose parameters are input into the parametric human body model, which is either a FLAME model or an SMPL-X model. The model outputs the standard space geometry, which is the set of vertex positions in the standard space, with each vertex containing three-dimensional spatial coordinates. The skin weights, facial detail normal maps, and diffuse maps in the standard space corresponding to each vertex are also obtained. Based on the standard spatial geometry, normal map, and diffuse map, a three-dimensional scene representation is constructed. The three-dimensional scene representation is a three-dimensional Gaussian sputtering field, where each Gaussian element contains a position vector, covariance matrix, color vector, and opacity value. The total number of Gaussian elements is adaptively determined according to the geometric complexity of the digital human model. Based on the rotation of each bone in the pose parameters, the transformation matrix of each bone is calculated; for each Gaussian element position in the 3D scene representation, the transformation matrix is ​​weighted and mixed using the corresponding skin weight, and the deformed position is calculated according to the linear blending skin formula to obtain the dynamic geometric data. The dynamic geometric data contains a set of transformed Gaussian elements, and each transformed Gaussian element includes a deformed position vector, a deformed covariance matrix, a color vector, and an opacity value.

[0009] Preferably, the step of arranging sparse virtual cameras in the target pose space, employing a deferred rendering pipeline, and simultaneously rendering the views of all sparse virtual cameras within a single draw call to generate a sparse reference view set containing color and depth textures includes: Based on the field of view and viewing distance of the target display device, several sparse virtual cameras are arranged in the target posture space; the sparse virtual cameras adopt a convergent arrangement, with the optical axes of each camera intersecting at the center of gravity of the digital human model; The dynamic geometry data is input into the geometry processing stage of the deferred rendering pipeline; using multi-rendering target technology, a geometry buffer array is generated simultaneously for all sparse virtual cameras within a single render call. The geometry buffer array contains the normal texture, diffuse color texture, world space position texture, and material property texture for each sparse virtual camera. The normal texture in the geometric buffer array is compressed and encoded using the Lambert equal area projection method, converting the three-dimensional normal vector into two-dimensional texture coordinates for storage; the diffuse color texture is transformed using a color space based on luminance extraction, converting the red-green-blue color space into a luminance-color difference color space, and then compressed using a chromaticity subsampling format to obtain the compressed geometric buffer array. A shading rate texture is generated based on the screen space contour mask of the digital human model; the shading rate is set to 1:1 for the digital human pixel region within the contour mask; the shading rate is set to 4:4 for the non-digital human background region outside the contour mask; the shading rate texture is bound to the variable rate shading stage of the graphics pipeline, so that the pixel shader executes at a reduced frequency in the background region. During the lighting calculation phase of the deferred rendering pipeline, the lighting results of each sparse virtual camera are calculated pixel by pixel based on the geometry buffer array, and the color texture and depth texture of each sparse virtual camera are obtained to form the sparse reference view set.

[0010] Preferably, the step of determining dense virtual viewpoints based on the viewpoint parameters of the target display device, and projecting each sparse viewpoint pixel onto the dense virtual viewpoint screen space through downsampling depth reprojection based on the depth texture in the sparse reference view set, includes: Obtain the viewpoint parameters of the target display device, including the total number of viewpoints, horizontal field of view, viewpoint spacing, and zero parallax plane position; The number of dense virtual viewpoints is determined based on the viewpoint parameters. The dense virtual viewpoints are arranged in an equally spaced staggered manner in the horizontal direction. The staggered arrangement means that the projection center of each dense virtual viewpoint is offset by a fixed distance in the horizontal direction, and the optical axes of each viewpoint are parallel to each other. For each sparse virtual camera and each dense virtual viewpoint, establish a coordinate transformation from the sparse viewpoint screen space to the dense virtual viewpoint screen space. The coordinate transformation includes, in sequence: transforming the sparse viewpoint screen coordinates to normalized device coordinates, transforming them to the sparse viewpoint observation space through inverse perspective projection, transforming them to world space through inverse observation matrix, and then transforming them to the dense virtual viewpoint screen space through the dense virtual viewpoint observation matrix and perspective projection matrix. Based on the staggered arrangement of dense virtual viewpoints, a horizontal translation component is introduced into the perspective projection matrix of the dense virtual viewpoints. The translation component is proportional to the distance between the dense virtual viewpoint and the central viewpoint.

[0011] Preferably, the step of accumulating the depth contribution of each pixel using point splatter technology to generate the depth range texture for each dense virtual viewpoint includes: Obtain the depth textures from the sparse reference view set, reduce the resolution of each depth texture, and obtain downsampled depth textures; For each pixel in the downsampled depth texture, its projection coordinates in the screen space of each dense virtual viewpoint are calculated using the constructed screen space transformation relationship, and the depth value at the projection coordinates is recorded. For each dense virtual viewpoint, an accumulation buffer is allocated in screen space for each pixel location. The accumulation buffer is used to store the minimum and maximum depth values ​​at that pixel location from the projection contributions of all sparse viewpoints. For each pixel in the downsampled depth texture of each sparse viewpoint, the region of influence is determined by a radially symmetric Gaussian kernel function centered on its projected coordinates, and each target pixel within the coverage of the Gaussian kernel is taken as a contributing receiver. Calculate the distance weight between the depth value of the current projected pixel and the target pixel. Based on the distance weight, add the current depth value to the accumulation buffer of the depth range of the target pixel. If the current depth value is less than the recorded minimum depth value, update the minimum depth value. If it is greater than the maximum depth value, update the maximum depth value. The minimum and maximum depth values ​​in the cumulative buffer of each dense virtual viewpoint are written into the red and green channels of the dual-channel texture, respectively, to form the depth range texture corresponding to the dense virtual viewpoint. The resolution of the depth range texture is the same as that of the downsampled depth texture. Each pixel in the texture stores the minimum starting depth and maximum ending depth of the light step corresponding to the screen position. All the depth range textures of dense virtual viewpoints are stored as a texture array in order of viewpoint number.

[0012] Preferably, determining the ray step interval for each dense virtual viewpoint using the depth range texture includes: A light field encoded mask texture is generated asynchronously. The light field encoded mask texture is consistent with the physical resolution of the target display device. Each pixel in the texture stores the viewpoint number corresponding to the pixel position. The light field encoded mask texture is generated based on the raster tilt angle of the light field display device and the number of views in the horizontal slit period. Based on the light field encoded mask texture, determine the dense virtual viewpoint number of the corresponding pixel pixel by pixel; obtain the observation matrix and perspective projection matrix corresponding to the dense virtual viewpoint number; for each pixel in the screen space, use the pixel coordinates and the observation matrix and perspective projection matrix to calculate the unit vector of the ray direction pointing from the viewpoint origin through the target pixel to the scene. Simultaneously, the minimum and maximum depth values ​​corresponding to the current pixel coordinates are sampled from the depth range texture of the corresponding dense virtual viewpoint. The minimum depth value is divided by the Z-axis component of the unit vector of the ray direction to obtain the minimum ray stepping parameter, and the maximum depth value is divided by the Z-axis component of the unit vector of the ray direction to obtain the maximum ray stepping parameter, thereby determining the ray stepping interval.

[0013] Preferably, the step of sampling along the ray direction, transforming the sampling points to the observation space of each sparse viewpoint, calculating truncated signed distance values ​​using depth textures, locating the scene surface position, and generating a surface position texture for each dense virtual viewpoint includes: Starting from the minimum ray stepping parameter and ending from the maximum ray stepping parameter, the sampling point is moved successively along the ray direction with an adaptive step size to obtain the world space coordinates of the current sampling point. For each sparse virtual camera, the world space coordinates are transformed to the screen space of the sparse virtual camera through the observation matrix and perspective projection matrix of the corresponding sparse virtual camera to obtain the screen space coordinates and the corresponding observation space depth value. Based on screen space coordinates, sample the depth texture of the corresponding sparse virtual camera to obtain the surface depth value recorded by the sparse viewpoint; calculate the difference between the observation space depth value and the surface depth value to obtain a signed distance value; truncate the signed distance value. If the absolute value of the signed distance value is greater than a preset truncation distance, set the truncated signed distance value as the boundary value between positive and negative truncation distances; otherwise, leave it unchanged to obtain the truncated signed distance value of the corresponding sparse viewpoint. For each sparse virtual camera, a neighborhood window of a predetermined size is taken in the depth texture with the screen space coordinates as the center, and the variance of the depth values ​​within the window is calculated. If the world space coordinates are outside the view frustum after being transformed to the observation space of the sparse viewpoint, the weight is set to zero. Otherwise, the preset threshold is divided by the square root of the variance and compared with one to obtain the minimum value, which is used to characterize the confidence of the depth measurement of the target sparse viewpoint. The truncated signed distance values ​​of all sparse viewpoints are weighted and summed according to their corresponding fusion weights to obtain the fused truncated signed distance value; the ray stepping parameter of the current sampling point is recorded, and the ray stepping parameter is increased by one step before sampling continues, and the fused truncated signed distance value of the next sampling point is calculated; when the fused truncated signed distance values ​​of two adjacent sampling points have opposite signs, it indicates that the ray has passed through the scene surface in this interval, and the surface intersection position is located using a linear interpolation formula; The world space coordinates of the surface intersection points are stored in the surface position texture of the screen pixel of the corresponding dense virtual viewpoint; if no sign change of the fused truncated signed distance value is detected within the ray stepping interval, the corresponding surface position texture is marked as invalid background.

[0014] Secondly, this application discloses a high-fidelity real-time rendering device for dynamic digital humans, which adopts the following technical solution, including: The geometric data module is used to receive multimodal driving signals, map them into facial expression parameters and posture parameters, input them into a parameterized human body model, and generate standard spatial geometry and skin weights; based on the standard spatial geometry, a three-dimensional scene representation is constructed, and the model is deformed to the target posture space according to the posture parameters and the skin weights to obtain dynamic geometric data; The virtual camera module is used to deploy sparse virtual cameras in the target pose space. It uses a deferred rendering pipeline to render the views of all sparse virtual cameras simultaneously within a single draw call, generating a sparse reference view set containing color and depth textures. The depth texture module is used to determine dense virtual viewpoints based on the viewpoint parameters of the target display device. Based on the depth texture in the sparse reference view set, each sparse viewpoint pixel is projected to the dense virtual viewpoint screen space by downsampling depth reprojection, and the depth contribution of each pixel is accumulated by point splashing technology to generate the depth range texture of each dense virtual viewpoint. The virtual viewpoint module is used to determine the ray stepping interval with the depth range texture for each dense virtual viewpoint, sample along the ray direction, transform the sampled points to the observation space of each sparse viewpoint, calculate the truncated signed distance value using the depth texture, locate the scene surface position, and generate the surface position texture of each dense virtual viewpoint. The image rendering module is used to project the surface position texture onto the screen space of each sparse viewpoint, sample sparse color texture, calculate color weights based on depth differences and fuse them to generate the final color image of the target dense virtual viewpoint; and synthesize the final color image into a suitable display image and render it according to the display device type.

[0015] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the above-described dynamic digital human high-fidelity real-time rendering method.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above for high-fidelity real-time rendering of a dynamic digital human.

[0017] In summary, this application receives multimodal driving signals such as audio, video, or inertial sensors, maps them to facial expression and pose parameters, and inputs them into a parameterized human model to generate standard spatial geometry and skinning weights. Then, based on the standard spatial geometry, a 3D Gaussian sputtering scene representation is constructed. The model is then deformed to the target pose space according to the pose parameters and skinning weights to obtain dynamic geometric data. Next, sparse virtual cameras are deployed in the target pose space, and a deferred rendering pipeline is used. Multiple rendering targets are employed to simultaneously generate a sparse reference view set containing color and depth textures within a single render call. Variable rate shading is used to reduce the shading overhead in non-digital human regions. Finally, based on the target… The viewpoint parameters of the display device determine dense virtual viewpoints. Sparse depth textures are used for downsampling depth reprojection, and point splashing technology is employed to generate depth range textures for each dense virtual viewpoint. Then, for each dense virtual viewpoint, the ray stepping interval is determined using the depth range texture. Sampling is performed along the ray direction, and the sampling points are transformed to the observation space of each sparse viewpoint. Truncated signed distance values ​​are calculated using depth textures and weighted fusion is performed to accurately locate the scene surface position and generate surface position textures. Finally, the surface position is projected back to the sparse view to sample colors and fused based on depth differences to generate the final color image of the dense virtual viewpoint. This image is then synthesized into a suitable display image output according to the display device type. This multimodal driving mechanism enables real-time expression and motion generation of dynamic digital humans. Sparse view reuse significantly reduces computational redundancy, and TSDF ray casting and color space fusion ensure high-fidelity surface details. Real-time rendering frame rates of 60 to 120 frames per second can be achieved on ordinary computing hardware. Simultaneously, it supports various display devices such as light field displays and VR headsets, significantly improving the real-time performance, fidelity, and interactive responsiveness of dynamic digital human rendering. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a high-fidelity real-time rendering method for dynamic digital humans.

[0019] Figure 2 This is a structural block diagram of a high-fidelity real-time rendering device for dynamic digital humans. Detailed Implementation

[0020] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0021] This application relates to the fields of computer graphics and 3D rendering technology. With the rapid development of metaverse, virtual reality, and intelligent interaction technologies, digital humans, as an important medium connecting the physical world and digital space, are being widely used in scenarios such as virtual live streaming, film and television entertainment, online education, and intelligent customer service. The market's requirements for digital humans have evolved from static appearance to highly realistic, emotionally natural, and interactive dynamic intelligent agents.

[0022] However, existing rendering methods generally face the challenge of balancing high fidelity and real-time performance: traditional multi-view independent rendering methods have extremely high computational redundancy, making it difficult to meet real-time frame rate requirements; while neural radiation field-based methods can generate high-quality images, their training and inference speeds are slow, making them unable to support dynamic driving; and depth reprojection-based methods are prone to losing geometric details, producing holes and artifacts. Furthermore, all of the above methods lack the ability to respond to multimodal driving signals such as audio and video, making it difficult to achieve real-time synchronization of facial expressions and movements.

[0023] Based on this, this application proposes a high-fidelity real-time rendering method for dynamic digital humans. It generates dynamic geometric data by driving a parameterized human model through multimodal signals, and significantly reduces rendering overhead by using sparse reference views combined with multiple rendering targets and variable rate shading technology. Furthermore, it introduces optimization strategies such as downsampling depth reprojection, point splashing, TSDF ray casting, and color space fusion to achieve real-time frame rate while ensuring high-fidelity rendering quality.

[0024] Reference Figure 1 The embodiments of this application include at least steps S10 to S50.

[0025] S10 receives multimodal driving signals, maps them to facial expression parameters and posture parameters, inputs a parameterized human body model, generates standard spatial geometry and skin weights; constructs a 3D scene representation based on the standard spatial geometry, and deforms it to the target posture space according to the posture parameters and skin weights to obtain dynamic geometric data.

[0026] S20: Arrange sparse virtual cameras in the target pose space, use a deferred rendering pipeline to render the views of all sparse virtual cameras simultaneously within a single draw call, and generate a sparse reference view set containing color textures and depth textures.

[0027] S30: Determine dense virtual viewpoints based on the viewpoint parameters of the target display device. Based on the depth texture in the sparse reference view set, project each sparse viewpoint pixel to the dense virtual viewpoint screen space through downsampling depth reprojection. Then, use point splashing technology to accumulate the depth contribution of each pixel to generate the depth range texture of each dense virtual viewpoint.

[0028] S40: For each dense virtual viewpoint, determine the ray stepping interval with depth range texture, sample along the ray direction, transform the sampled points to the observation space of each sparse viewpoint, calculate the truncated signed distance value using depth texture, locate the scene surface position, and generate the surface position texture of each dense virtual viewpoint.

[0029] S50 projects the surface position texture onto the screen space of each sparse viewpoint, samples the sparse color texture, calculates color weights based on depth differences and fuses them to generate the final color image of the target dense virtual viewpoint; according to the display device type, the final color image is synthesized into a suitable display image and rendered.

[0030] Specifically, the system generates facial and motion parameters for the digital human based on driving signals such as audio or video. It then uses a parametric human model to obtain 3D geometry and skinning weights under standard poses, which are then deformed to the target pose to form dynamic geometric data. Based on this, only color and depth information for a small number of sparse viewpoints are rendered, using multi-target rendering technology to complete the rendering of multiple perspectives simultaneously. Subsequently, based on the dense viewpoints required by the target display device, the sparse depth map is downsampled, reprojected, and sputtered to generate the depth range for each dense viewpoint. Finally, step-sampling is performed along the ray direction within the depth range, and the truncated signed distance value is calculated and combined with the sparse depth texture to determine the scene surface position, obtaining the surface position texture for each dense viewpoint, laying the foundation for subsequent color synthesis.

[0031] In some embodiments, step S10 specifically includes the following steps: receiving a multimodal driving signal, the multimodal driving signal including an audio stream, a video stream, or an inertial sensor data stream; extracting Mel-frequency cepstral coefficients and fundamental frequency features from the audio stream, extracting two-dimensional coordinates of facial landmarks and head posture from the video stream, and extracting limb joint rotation angles from the inertial sensor data; fusing the extracted feature vectors through a temporal convolutional network to generate a low-dimensional driving latent code; inputting the driving latent code into a pre-trained multilayer perceptron, which is mapped to expression parameters and posture parameters, respectively, the expression parameters being used to control the activation level of facial muscle units, and the posture parameters being used to control the axial rotation of the whole skeleton; Facial expression and pose parameters are input into a parametric human model, which can be either a FLAME model or an SMPL-X model. The model outputs standard space geometry, i.e., a set of vertex positions in standard space, with each vertex containing three-dimensional spatial coordinates. The model also obtains the skinning weights corresponding to each vertex and the facial detail normal map and diffuse map in standard space. Based on the standard space geometry, normal map, and diffuse map, a three-dimensional scene representation is constructed. The three-dimensional scene representation is a three-dimensional Gaussian sputtering field, where each Gaussian element contains a position vector, covariance matrix, color vector, and opacity value. The total number of Gaussian elements is adaptively determined according to the geometric complexity of the digital human model. Based on the rotation of each bone in the pose parameters, the transformation matrix of each bone is calculated. For each Gaussian element position in the 3D scene representation, the transformation matrix is ​​weighted and mixed using its corresponding skin weight. The deformed position is calculated according to the linear blending skin formula to obtain dynamic geometric data. The dynamic geometric data contains a set of transformed Gaussian elements. Each transformed Gaussian element includes a deformed position vector, a deformed covariance matrix, a color vector, and an opacity value.

[0032] The formula for linear hybrid skin deformation is: ; in, The Gaussian element position vector (three-dimensional coordinates) before deformation. K is the deformed position vector; K is the total number of bones; j is the bone index; The skin weight of the j-th bone corresponding to this Gaussian element, with a value range of [0,1]. The rotation matrix for the j-th bone (calculated from the axis angle rotation in the pose parameters); Let be the translation vector of the j-th bone. The deformed Gaussian elements also include the deformed covariance matrix, whose transformation formula is: ,in The original covariance matrix, The transformed covariance matrix, express The transpose of the vector is used. The color vector and opacity values ​​remain unchanged. This results in dynamic geometric data containing the transformed Gaussian set of elements.

[0033] Specifically, features are extracted from audio, video, or inertial sensors, fused into driving latent codes via a temporal network, and then mapped to expression and pose parameters that control facial and limb movements. These two parameters are input into a parameterized human model to obtain 3D vertex positions, skin weights, normals, and diffuse maps in a standard pose, and based on this, a 3D Gaussian sputtering field composed of a large number of Gaussian primitives is constructed. Subsequently, the bone transformation matrix is ​​calculated based on the pose parameters, and the position and covariance matrix of each Gaussian primitive are weighted and deformed using skin weights, ultimately generating a deformed set of Gaussian primitives, which serves as dynamic geometric data for subsequent rendering.

[0034] In some embodiments, step S20 specifically includes the following steps: Arranging a plurality of sparse virtual cameras in the target pose space according to the field of view and viewing distance of the target display device; the sparse virtual cameras are arranged in a convergent manner, with the optical axes of each camera intersecting at the center of gravity of the digital human model; inputting dynamic geometric data into the geometry processing stage of the deferred rendering pipeline; using multi-rendering target technology, generating a geometry buffer array for all sparse virtual cameras simultaneously within a single rendering call; the geometry buffer array contains the normal texture, diffuse color texture, world space position texture, and material property texture of each sparse virtual camera; compressing and encoding the normal texture in the geometry buffer array using the Lambert equal area projection method, converting the three-dimensional normal vector into two-dimensional texture coordinates for storage; and using a color space transformation based on luminance extraction to convert the diffuse color texture from the red-green-blue color space to a luminance-color difference color space, and encoding and compressing it using a chromaticity subsampling format to obtain the compressed geometry buffer array. Based on the screen space contour mask of the digital human model, a shading rate texture is generated; for the digital human pixel region within the contour mask, the shading rate is set to 1:1; for the non-digital human background region outside the contour mask, the shading rate is set to 4:4; the shading rate texture is bound to the variable rate shading stage of the graphics pipeline, so that the pixel shader executes at a reduced frequency in the background region; in the lighting calculation stage of the deferred rendering pipeline, the lighting results of each sparse virtual camera are calculated pixel by pixel based on the geometry buffer array, and the color texture and depth texture of each sparse virtual camera are obtained to form a sparse reference view set.

[0035] Specifically, multiple converging sparse virtual cameras are deployed in the target pose space, with their optical axes converging at the center of gravity of the digitizer. Dynamic geometric data is fed into the deferred rendering pipeline, and multi-render target technology is used to simultaneously generate geometric buffer arrays for all sparse cameras within a single render call, containing textures such as normals, colors, positions, and materials. To reduce memory usage, the normal texture is compressed using Lambert equal-area projection, and the color texture is converted to a luminance-chromaticity space and subjected to chromaticity subsampling. Simultaneously, a shading rate texture is generated based on the digitizer's screen outline, with a lower shading rate set for the background area, causing the pixel shader to execute less frequently, thereby reducing computational overhead. Finally, pixel-by-pixel rendering is performed during the lighting calculation stage, outputting the color and depth textures of each sparse camera to form a sparse reference view set.

[0036] In some embodiments, step S30 specifically includes the following steps: obtaining viewpoint parameters of the target display device, including the total number of viewpoints, horizontal field of view, viewpoint spacing, and zero parallax plane position; determining the number of dense virtual viewpoints based on the viewpoint parameters, wherein each dense virtual viewpoint is arranged in a staggered manner with equal spacing in the horizontal direction, the staggered arrangement means that the projection center of each dense virtual viewpoint is successively offset by a fixed spacing in the horizontal direction, and the optical axes of each viewpoint are parallel to each other; establishing a coordinate transformation from the sparse viewpoint screen space to the dense virtual viewpoint screen space for each sparse virtual camera and each dense virtual viewpoint; the coordinate transformation includes, in sequence: transforming the sparse viewpoint screen coordinates to normalized device coordinates, transforming to the sparse viewpoint observation space through inverse perspective projection, transforming to world space through inverse observation matrix, and then transforming to the dense virtual viewpoint screen space through the dense virtual viewpoint observation matrix and perspective projection matrix; and introducing a horizontal translation component into the perspective projection matrix of the dense virtual viewpoint according to the staggered arrangement of the dense virtual viewpoints, wherein the translation component is proportional to the spacing between the dense virtual viewpoint and the central viewpoint.

[0037] Specifically, this technical solution first determines the number of dense virtual viewpoints based on the viewpoint parameters of the target display device, and then arranges them at equal intervals and staggered in the horizontal direction, with the optical axes of each viewpoint parallel to each other. Subsequently, a coordinate transformation from sparse screen space to dense screen space is established for each sparse virtual camera and dense virtual viewpoint. This transformation sequentially passes through normalized device coordinates, inverse perspective projection, and inverse observation matrix to reach world space, and then through the observation matrix and perspective projection matrix of the dense viewpoint to reach its screen space. Simultaneously, according to the staggered arrangement of the dense viewpoints, a horizontal translation component is introduced into the perspective projection matrix, which is proportional to the distance of the current viewpoint from the center viewpoint. Through the above transformation, pixel-level mapping between sparse and dense viewpoints is achieved, laying the foundation for subsequent depth reprojection.

[0038] Furthermore, step S30 also includes the following steps: obtaining depth textures from the sparse reference view set, reducing the resolution of each depth texture to one-quarter or one-half of the original resolution to obtain downsampled depth textures; for each pixel in the downsampled depth texture, using the constructed screen space transformation relationship, calculating its projection coordinates in the screen space of each dense virtual viewpoint, and recording the depth value at the projection coordinates; for each dense virtual viewpoint, allocating an accumulation buffer in the screen space for each pixel position, the accumulation buffer being used to store the minimum and maximum depth values ​​at that pixel position from the projection contributions of all sparse viewpoints; for each pixel in the downsampled depth texture of each sparse viewpoint, using its projection coordinates as the center, using a radially symmetric Gaussian kernel function to determine the influence region, and taking each target pixel within the coverage area of ​​the Gaussian kernel as a contribution receiver; Calculate the distance weight between the depth value of the current projected pixel and the target pixel. Based on the distance weight, accumulate the current depth value into the accumulation buffer of the depth range of the target pixel. If the current depth value is less than the recorded minimum depth value, update the minimum depth value; if it is greater than the maximum depth value, update the maximum depth value. Write the minimum and maximum depth values ​​in the accumulation buffer of each dense virtual viewpoint into the red and green channels of the dual-channel texture, respectively, to form the depth range texture of the corresponding dense virtual viewpoint. The resolution of the depth range texture is consistent with the resolution of the downsampled depth texture. Each pixel in the texture stores the minimum starting depth and maximum ending depth of the light step corresponding to the screen position. Store the depth range textures of all dense virtual viewpoints in a texture array according to the viewpoint number order.

[0039] Specifically, the sparse depth texture is downsampled to reduce computation. Then, using the previously established screen space transformation, each downsampled depth pixel is projected onto the screen coordinates of the dense viewpoint. For each pixel of each dense viewpoint, an accumulation buffer is set to record the minimum and maximum depth contributions from projections from different sparse viewpoints. During projection, an influence region is determined using a Gaussian kernel function centered on the projection point. Within this region, each target pixel receives a depth contribution based on distance weights, and the minimum and maximum depth values ​​of that pixel are dynamically updated. Finally, the accumulated results for each dense viewpoint are stored in the red and green channels of a dual-channel texture, forming a depth range texture (storing the start and end depths of ray steps). The depth range textures of all dense viewpoints are grouped into a texture array by number for subsequent surface localization.

[0040] In some embodiments, step S40 specifically includes the following steps: asynchronously generating a light field encoded mask texture, the light field encoded mask texture being consistent with the physical resolution of the target display device, and each pixel in the texture storing the viewpoint number corresponding to that pixel position; the light field encoded mask texture is generated based on the raster tilt angle of the light field display device and the number of views in the horizontal slit period, and the specific calculation formula is as follows: ; in, Subpixel index The corresponding viewpoint number; k is the horizontal index of the sub-pixel; The vertical index of the sub-pixel; This is the horizontal distance from the top-left corner of the sub-pixel to the origin of the coordinate system. This refers to the horizontal offset between multiple viewpoints; The tilt angle of the grating; The number of views in the horizontal slit cycle; This represents the total number of views.

[0041] Based on the light field encoded mask texture, the dense virtual viewpoint number to which the corresponding pixel belongs is determined pixel by pixel; the observation matrix and perspective projection matrix corresponding to the dense virtual viewpoint number are obtained; for each pixel in the screen space, the unit vector of the ray direction pointing from the viewpoint origin through the target pixel to the scene is calculated using the pixel coordinates, observation matrix, and perspective projection matrix; at the same time, the minimum depth value and maximum depth value corresponding to the current pixel coordinates are sampled from the depth range texture of the corresponding dense virtual viewpoint, the minimum depth value is divided by the Z-axis component of the ray direction unit vector to obtain the minimum ray step parameter, the maximum depth value is divided by the Z-axis component of the ray direction unit vector to obtain the maximum ray step parameter, and the ray step interval is determined.

[0042] The formula for calculating the ray stepping interval parameters is: ; ; in, and These are the minimum and maximum parameters for ray stepping (i.e., the distances from the start and end points of the stepping to the viewpoint). and These are the minimum and maximum depth values ​​stored in the depth range texture, respectively; The z-axis component of the unit vector in the ray direction (along the forward direction of the viewpoint).

[0043] Specifically, the system asynchronously generates a light field-coded mask texture with the same resolution as the display device. Each pixel stores the viewpoint number corresponding to that location, calculated from parameters such as the raster tilt angle and the number of viewpoints. Then, for each pixel on the screen, its corresponding dense virtual viewpoint is determined based on the mask texture, and the viewpoint's observation and projection matrix is ​​obtained, thereby calculating the ray direction originating from the viewpoint and passing through the pixel. Simultaneously, the minimum and maximum depth values ​​at that pixel location are sampled from the depth range texture corresponding to the viewpoint, and divided by the Z-axis component of the ray direction vector to obtain the start and end parameters of the ray stepping, thus determining the effective stepping range of the ray in space, preparing for subsequent surface intersection location searches.

[0044] Furthermore, step S40 also includes the following steps: starting from the minimum ray stepping parameter and ending at the maximum ray stepping parameter, the sampling point is moved successively along the ray direction with an adaptive step size to obtain the world space coordinates of the current sampling point; for each sparse virtual camera, the world space coordinates are transformed to the screen space of the sparse virtual camera through the observation matrix and perspective projection matrix of the corresponding sparse virtual camera to obtain the screen space coordinates and the corresponding observation space depth value; based on the screen space coordinates, the depth texture of the corresponding sparse virtual camera is sampled to obtain the surface depth value recorded by the sparse viewpoint; the difference between the observation space depth value and the surface depth value is calculated to obtain the signed distance value; the signed distance value is truncated, and if the absolute value of the signed distance value is greater than the preset truncation distance, the truncated signed distance value is set as the boundary value of the positive and negative truncation distances, otherwise it remains unchanged to obtain the truncated signed distance value of the corresponding sparse viewpoint. The calculation formula for the truncated signed distance value is: ; in, The truncated signed distance value for the i-th sparse viewpoint; The surface depth value sampled in the sparse viewpoint depth texture; This is the depth value (i.e., the z-coordinate of the observation space) after the current sampling point is transformed to the observation space of this sparse viewpoint. This means restricting x to... Within the interval, T is the preset cutoff distance (a positive real number).

[0045] For each sparse virtual camera, a neighborhood window of a predetermined size is taken in the depth texture, centered on the screen space coordinates, and the variance of the depth values ​​within the window is calculated. If the world space coordinates are outside the view frustum after transformation to the observation space of the sparse viewpoint, the weight is set to zero; otherwise, a preset threshold is divided by the square root of the variance and compared with one, and the minimum value is taken to obtain the fusion weight of the sparse viewpoint. The fusion weight is used to characterize the confidence of the depth measurement of the target sparse viewpoint. The formula for calculating the fusion weight is: ; in, Let be the fusion weight for the i-th sparse viewpoint; The preset small positive threshold (e.g., 0.01 or 0.01). Variance within the neighborhood window of the depth texture.

[0046] The truncated signed distance values ​​of all sparse viewpoints are weighted and summed according to their corresponding fusion weights to obtain the fused truncated signed distance value. The ray step parameter of the current sampling point is recorded. The ray step parameter is increased by one step size before sampling continues, and the fused truncated signed distance value of the next sampling point is calculated. When the fused truncated signed distance values ​​of two adjacent sampling points have opposite signs, it indicates that the ray has passed through the scene surface in that interval. The surface intersection point is located using a linear interpolation formula. The surface intersection point position is equal to the previous sampling point position plus the previous sampling point's fused truncated signed distance value divided by the difference between the fused truncated signed distance values ​​of the previous and subsequent sampling points multiplied by the step size. ; in, The interpolated surface intersection points (world space coordinates); The world space coordinates of the previous sampling point; and The fused signed distance values ​​of the previous sampling point and the current sampling point are respectively truncated. The step size is the distance between adjacent sampling points.

[0047] Store the world space coordinates of the surface intersection points into the surface position texture of the corresponding dense virtual viewpoint's screen pixel; if no sign change of the fused truncated signed distance value is detected within the ray stepping interval, mark the corresponding surface position texture as invalid background.

[0048] Specifically, within a defined ray stepping interval, spatial points are sampled progressively from the starting point to the ending point. For each sampled point, it is transformed to the viewpoint of each sparse camera, and the signed distance between that point and the sparse depth recording surface is calculated and truncated to obtain the truncated signed distance value. Simultaneously, the fusion weight for each sparse viewpoint is calculated based on the local variance of the sparse depth texture (the smaller the variance, the greater the weight; points outside the view frustum have zero weight). The weighted sum of the truncated signed distance values ​​of all sparse viewpoints yields the fusion value. When the sign of the fusion value of adjacent sampled points changes from positive to negative, it indicates that the ray has passed through the object surface. At this point, linear interpolation is used to accurately calculate the coordinates of the surface intersection point, which is then stored in the surface position texture of the corresponding dense viewpoint. If there is no sign change throughout, it is marked as an invalid background.

[0049] In some embodiments, step S50 specifically includes the following steps: For each dense virtual viewpoint, a surface position texture is acquired, which stores the world space coordinates of scene surface points corresponding to each screen pixel under that viewpoint. For each surface point, it is projected onto the screen space of each sparse virtual camera to obtain the corresponding screen space coordinates. Based on the screen space coordinates, the color texture of the corresponding sparse virtual camera is sampled to obtain the color value of the sparse viewpoint at that surface point. Simultaneously, the surface point is transformed to the observation space of the sparse virtual camera to obtain the observation space depth value, and the surface depth value recorded by the viewpoint is sampled from the depth texture of the sparse virtual camera. The absolute value of the difference between the two is calculated as the depth difference.

[0050] To integrate the color contributions from multiple sparse viewpoints, this embodiment introduces an adaptive color weight calculation method based on depth and viewpoint differences. For each sparse viewpoint, its color weight is the product of three parts: depth difference weight, viewpoint difference weight, and confidence weight. The depth difference weight is calculated using a Gaussian function: the smaller the depth difference, the larger the weight, as shown in the formula below: ; in, For the depth difference weight of the i-th sparse viewpoint; The square of the depth difference is the square of the difference between the observation depth and the recording depth at the surface point. The sensitivity to depth differences is controlled by the square of the preset Gaussian kernel bandwidth.

[0051] The weight of the viewpoint difference is taken as the cosine of the angle between the line of sight of the sparse viewpoint and the normal direction of the surface point; the smaller the angle, the greater the weight. ; in, The viewpoint difference weight; n is the unit normal vector of the surface point; Let be the unit line-of-sight direction from the surface point to the optical center of the camera at the i-th sparse viewpoint; the dot product result is set to the maximum value of 0 to ensure non-negativity.

[0052] The confidence weight is directly adopted from the fusion weight of the sparse viewpoint at the surface point calculated in step S40, to reflect the reliability of the depth measurement. The final color weight of the sparse viewpoint is obtained by multiplying the three weights together. The corresponding formula is as follows: ; The color values ​​of each sparse viewpoint are weighted, normalized, and summed to obtain the final color value of the current pixel of the dense virtual viewpoint. If the color weights of all sparse viewpoints are zero (e.g., a surface point is outside the view frustum of all sparse viewpoints), then the color of that pixel is set to the background color.

[0053] ; in, The final color value (RGB components); The color vector is obtained by sampling the i-th sparse viewpoint. If the color weights of all sparse viewpoints are zero.

[0054] After color calculations are completed for all dense virtual viewpoints, different compositing strategies are executed based on the target display device type. If the target display device is a light field display, a pre-generated light field encoded mask texture is obtained. This mask texture records the viewpoint number corresponding to each sub-pixel position of the light field display. According to the mapping relationship of the mask texture, the pixel values ​​in the color images of each dense virtual viewpoint are written into the corresponding sub-pixel positions of the light field stereoscopic view, and the final light field encoded image is synthesized. If the target display device is a VR headset or a regular 2D display, the center viewpoint or left and right views in the dense virtual viewpoints are directly output as the display image. If the target display device supports multi-view naked-eye 3D display, the dense virtual viewpoint images are synthesized in a column-interlaced or row-interlaced manner according to the viewpoint arrangement format required by the device. Finally, the synthesized display image is transmitted to the target display device for rendering and presentation.

[0055] This embodiment has the following creative improvements: First, it introduces an adaptive color fusion method that combines depth difference, viewpoint difference, and confidence weight, which effectively reduces color distortion and edge artifacts caused by reprojection, and is especially suitable for rapidly changing geometric shapes in dynamic digital human scenes; Second, it supports adaptive output compositing for multiple types of display devices, not limited to light field displays, but also compatible with VR headsets, naked-eye 3D screens, etc., expanding the application scope of the technology; Third, by reusing the fusion weights calculated in step S40, it reduces redundant calculations and maintains real-time rendering efficiency.

[0056] The implementation principle of a high-fidelity real-time rendering method for dynamic digital humans in this application is as follows: The system receives multimodal driving signals such as audio, video, or inertial sensors, maps them into expression parameters and posture parameters, and inputs them into a parameterized human model to generate standard spatial geometry and skin weights. Then, a three-dimensional Gaussian sputtering scene representation is constructed based on the standard spatial geometry, and the dynamic geometric data is obtained by deforming to the target posture space according to the posture parameters and skin weights. Then, sparse virtual cameras are arranged in the target posture space, a deferred rendering pipeline is used, and a sparse reference view set containing color texture and depth texture is generated simultaneously in a single render call using multi-rendering technology. At the same time, variable rate shading is used to reduce the non-digital human areas. The process involves several steps: First, shading overhead is reduced. Then, dense virtual viewpoints are determined based on the viewpoint parameters of the target display device. Sparse depth textures are used for downsampling depth reprojection, and point splatter technology is employed to generate depth range textures for each dense virtual viewpoint. Next, for each dense virtual viewpoint, the ray stepping interval is determined using the depth range texture. Sampling is performed along the ray direction, and the sampling points are transformed to the observation space of each sparse viewpoint. Truncated signed distance values ​​are calculated using depth textures and weighted fusion is performed to accurately locate the scene surface position and generate surface position textures. Finally, the surface position is projected back to the sparse view to sample colors and fused based on depth differences to generate the final color image of the dense virtual viewpoint. This image is then synthesized into a suitable display image output according to the display device type. This multimodal driving mechanism enables real-time expression and motion generation of dynamic digital humans. Sparse view reuse significantly reduces computational redundancy, and TSDF ray casting and color space fusion ensure high-fidelity surface details. Real-time rendering frame rates of 60 to 120 frames per second can be achieved on ordinary computing hardware. Simultaneously, it supports various display devices such as light field displays and VR headsets, significantly improving the real-time performance, fidelity, and interactive responsiveness of dynamic digital human rendering.

[0057] Figure 1 This is a flowchart illustrating a high-fidelity real-time rendering method for dynamic digital humans in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0058] Based on the same technical concept, referring to Figure 2This application also provides a high-fidelity real-time rendering device for dynamic digital humans, which adopts the following technical solution: The device includes: The geometric data module receives multimodal driving signals, maps them to facial expression parameters and pose parameters, inputs them into a parameterized human body model, and generates standard spatial geometry and skin weights. Based on the standard spatial geometry, it constructs a 3D scene representation and deforms it to the target pose space according to the pose parameters and skin weights to obtain dynamic geometric data. The virtual camera module is used to deploy sparse virtual cameras in the target pose space. It uses a deferred rendering pipeline to render the views of all sparse virtual cameras simultaneously within a single draw call, generating a sparse reference view set containing color and depth textures. The depth texture module is used to determine dense virtual viewpoints based on the viewpoint parameters of the target display device. Based on the depth texture in the sparse reference view set, the pixels of each sparse viewpoint are projected to the screen space of the dense virtual viewpoint through downsampling depth reprojection, and the depth contribution of each pixel is accumulated by point splashing technology to generate the depth range texture of each dense virtual viewpoint. The virtual viewpoint module is used to determine the ray stepping interval with depth range texture for each dense virtual viewpoint, sample along the ray direction, transform the sampled points to the observation space of each sparse viewpoint, calculate the truncated signed distance value using depth texture, locate the scene surface position, and generate the surface position texture of each dense virtual viewpoint. The image rendering module projects the surface position texture onto the screen space of each sparse viewpoint, samples the sparse color texture, calculates and fuses color weights based on depth differences, and generates the final color image of the target dense virtual viewpoint; according to the display device type, the final color image is synthesized into a suitable display image and rendered.

[0059] In some embodiments, the geometric data module is specifically used to receive multimodal driving signals, including audio streams, video streams, or inertial sensor data streams; extract Mel frequency cepstral coefficients and fundamental frequency features from the audio stream, extract two-dimensional coordinates of facial landmarks and head posture from the video stream, and extract limb joint rotation angles from the inertial sensor data; and fuse the extracted feature vectors through a temporal convolutional network to generate a low-dimensional driving latent code. The driving latent code is input into the pre-trained multilayer perceptron and mapped to expression parameters and pose parameters, respectively. The expression parameters are used to control the activation level of facial muscle units, and the pose parameters are used to control the axial rotation of the whole skeleton. The facial expression and pose parameters are input into the parametric human body model, which is either a FLAME model or an SMPL-X model. The model outputs standard space geometry, which is the set of vertex positions in standard space, with each vertex containing three-dimensional spatial coordinates. The model also obtains the skin weights corresponding to each vertex and the facial detail normal map and diffuse map in standard space. Based on standard spatial geometry, normal maps, and diffuse maps, a 3D scene representation is constructed. The 3D scene representation is a 3D Gaussian sputtering field, where each Gaussian element contains a position vector, a covariance matrix, a color vector, and an opacity value. The total number of Gaussian elements is adaptively determined according to the geometric complexity of the digital human model. Based on the rotation of each bone in the pose parameters, the transformation matrix of each bone is calculated. For each Gaussian element position in the 3D scene representation, the transformation matrix is ​​weighted and mixed using its corresponding skin weight. The deformed position is calculated according to the linear blending skin formula to obtain dynamic geometric data. The dynamic geometric data contains a set of transformed Gaussian elements. Each transformed Gaussian element includes a deformed position vector, a deformed covariance matrix, a color vector, and an opacity value.

[0060] In some embodiments, the virtual camera module is specifically used to arrange a number of sparse virtual cameras in the target pose space according to the field of view and viewing distance of the target display device; the sparse virtual cameras adopt a convergent arrangement, and the optical axes of each camera intersect at the center of gravity of the digital human model; Dynamic geometry data is input into the geometry processing stage of the deferred rendering pipeline; using multi-render target technology, geometry buffer arrays are generated simultaneously for all sparse virtual cameras within a single draw call. The geometry buffer arrays contain the normal texture, diffuse color texture, world space position texture, and material property texture for each sparse virtual camera. The normal texture in the geometry buffer array is compressed and encoded using the Lambert equal area projection method, converting the three-dimensional normal vector into two-dimensional texture coordinates for storage; the diffuse color texture is transformed using a color space based on luminance extraction, converting the red-green-blue color space into a luminance-color difference color space, and then compressed using a chromaticity subsampling format to obtain the compressed geometry buffer array. A shading rate texture is generated based on the screen space contour mask of the digital human model; the shading rate is set to 1:1 for the digital human pixel region within the contour mask; the shading rate is set to 4:4 for the non-digital human background region outside the contour mask; the shading rate texture is bound to the variable rate shading stage of the graphics pipeline, so that the pixel shader executes at a reduced frequency in the background region. During the lighting calculation phase of the deferred rendering pipeline, the lighting results of each sparse virtual camera are calculated pixel by pixel based on the geometry buffer array, and the color texture and depth texture of each sparse virtual camera are obtained to form a sparse reference view set.

[0061] In some embodiments, the depth texture module is specifically used to obtain the viewpoint parameters of the target display device, including the total number of viewpoints, the horizontal field of view, the viewpoint spacing, and the position of the zero parallax plane. The number of dense virtual viewpoints is determined based on the viewpoint parameters. The dense virtual viewpoints are arranged in an equally spaced staggered manner in the horizontal direction. The staggered arrangement means that the projection center of each dense virtual viewpoint is offset by a fixed distance in the horizontal direction, and the optical axes of each viewpoint are parallel to each other. For each sparse virtual camera and each dense virtual viewpoint, establish a coordinate transformation from the sparse viewpoint screen space to the dense virtual viewpoint screen space. The coordinate transformation includes, in sequence: transforming the sparse viewpoint screen coordinates to normalized device coordinates, transforming them to the sparse viewpoint observation space through inverse perspective projection, transforming them to world space through inverse observation matrix, and then transforming them to the dense virtual viewpoint screen space through the dense virtual viewpoint observation matrix and perspective projection matrix. Based on the staggered arrangement of dense virtual viewpoints, a horizontal translation component is introduced into the perspective projection matrix of the dense virtual viewpoints. The translation component is proportional to the distance between the dense virtual viewpoint and the central viewpoint.

[0062] In some embodiments, the depth texture module is specifically used to obtain depth textures in a sparse reference view set, reduce the resolution of each depth texture, and obtain downsampled depth textures. For each pixel in the downsampled depth texture, the projected coordinates in the screen space of each dense virtual viewpoint are calculated using the constructed screen space transformation relationship, and the depth value at the projected coordinates is recorded. For each dense virtual viewpoint, an accumulation buffer is allocated in screen space for each pixel location. The accumulation buffer is used to store the minimum and maximum depth values ​​at that pixel location from the projection contributions of all sparse viewpoints. For each pixel in the downsampled depth texture of each sparse viewpoint, the region of influence is determined by a radially symmetric Gaussian kernel function centered on its projected coordinates, and each target pixel within the coverage of the Gaussian kernel is taken as a contributing receiver. Calculate the distance weight between the depth value of the current projected pixel and the target pixel. Based on the distance weight, add the current depth value to the accumulation buffer of the depth range of the target pixel. If the current depth value is less than the recorded minimum depth value, update the minimum depth value. If it is greater than the maximum depth value, update the maximum depth value. The minimum and maximum depth values ​​in the cumulative buffer of each dense virtual viewpoint are written into the red and green channels of the dual-channel texture, respectively, to form the depth range texture of the corresponding dense virtual viewpoint. The resolution of the depth range texture is the same as that of the downsampled depth texture. Each pixel in the texture stores the minimum starting depth and maximum ending depth of the light step corresponding to the screen position. Store the depth range textures of all dense virtual viewpoints in a texture array according to the viewpoint number order.

[0063] In some embodiments, the virtual viewpoint module is specifically used to asynchronously generate a light field encoded mask texture. The light field encoded mask texture is consistent with the physical resolution of the target display device. Each pixel in the texture stores the viewpoint number corresponding to the pixel position. The light field encoded mask texture is generated based on the raster tilt angle of the light field display device and the number of views in the horizontal slit period. Based on the light field encoded mask texture, determine the dense virtual viewpoint number of the corresponding pixel pixel by pixel; obtain the observation matrix and perspective projection matrix corresponding to the dense virtual viewpoint number; for each pixel in the screen space, use the pixel coordinates and the observation matrix and perspective projection matrix to calculate the unit vector of the ray direction pointing from the viewpoint origin through the target pixel to the scene. Simultaneously, the minimum and maximum depth values ​​corresponding to the current pixel coordinates are sampled from the depth range texture of the corresponding dense virtual viewpoint. The minimum depth value is divided by the Z-axis component of the unit vector of the ray direction to obtain the minimum ray stepping parameter, and the maximum depth value is divided by the Z-axis component of the unit vector of the ray direction to obtain the maximum ray stepping parameter, thus determining the ray stepping interval.

[0064] In some embodiments, the virtual viewpoint module is specifically used to move the sampling point along the ray direction with an adaptive step size, starting from the minimum ray step parameter and ending at the maximum ray step parameter, to obtain the world space coordinates of the current sampling point. For each sparse virtual camera, the world space coordinates are transformed to the screen space of the sparse virtual camera through the observation matrix and perspective projection matrix of the corresponding sparse virtual camera to obtain the screen space coordinates and the corresponding observation space depth value. Based on screen space coordinates, sample the depth texture of the corresponding sparse virtual camera to obtain the surface depth value recorded by the sparse viewpoint; calculate the difference between the observation space depth value and the surface depth value to obtain the signed distance value; truncate the signed distance value. If the absolute value of the signed distance value is greater than the preset truncation distance, set the truncated signed distance value as the boundary value between the positive and negative truncation distances; otherwise, leave it unchanged to obtain the truncated signed distance value of the corresponding sparse viewpoint. For each sparse virtual camera, a neighborhood window of a predetermined size is taken in the depth texture with the screen space coordinates as the center, and the variance of the depth values ​​within the window is calculated. If the world space coordinates are outside the view frustum after being transformed to the observation space of the sparse viewpoint, the weight is set to zero. Otherwise, the preset threshold is divided by the square root of the variance and compared with one to obtain the minimum value, which is used to characterize the confidence of the depth measurement of the target sparse viewpoint. The truncated signed distance values ​​of all sparse viewpoints are weighted and summed according to their corresponding fusion weights to obtain the fused truncated signed distance value. The ray stepping parameter of the current sampling point is recorded. The ray stepping parameter is increased by one step and sampling continues. The fused truncated signed distance value of the next sampling point is calculated. When the fused truncated signed distance values ​​of two adjacent sampling points have opposite signs, it indicates that the ray has passed through the scene surface in this interval. The intersection point of the surface is located using a linear interpolation formula. Store the world space coordinates of the surface intersection points into the surface position texture of the corresponding dense virtual viewpoint's screen pixel; if no sign change of the fused truncated signed distance value is detected within the ray stepping interval, mark the corresponding surface position texture as invalid background.

[0065] This application also discloses a control device.

[0066] Specifically, the control device includes a memory and a processor, the memory storing a computer program that can be loaded by the processor and executed using the aforementioned high-fidelity real-time rendering method for dynamic digital humans.

[0067] This application also discloses a computer-readable storage medium.

[0068] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed as described above in the high-fidelity real-time rendering method for dynamic digital humans. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0069] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A high-fidelity real-time rendering method for dynamic digital humans, characterized in that, include: Receive multimodal driving signals, map them into facial expression parameters and pose parameters, input them into a parameterized human body model, and generate standard spatial geometry and skinning weights; A three-dimensional scene representation is constructed based on the standard spatial geometry, and the scene is deformed to the target pose space according to the pose parameters and the skin weights to obtain dynamic geometric data; Sparse virtual cameras are deployed in the target pose space. A deferred rendering pipeline is used to render the views of all sparse virtual cameras simultaneously within a single draw call, generating a sparse reference view set containing color textures and depth textures. The dense virtual viewpoint is determined based on the viewpoint parameters of the target display device. Based on the depth texture in the sparse reference view set, each sparse viewpoint pixel is projected to the dense virtual viewpoint screen space by downsampling depth reprojection. The depth contribution of each pixel is accumulated by point splashing technology to generate the depth range texture of each dense virtual viewpoint. For each dense virtual viewpoint, the ray stepping interval is determined by the depth range texture, sampling is performed along the ray direction, and the sampling points are transformed to the observation space of each sparse viewpoint. The truncated signed distance value is calculated using the depth texture, and the scene surface position is located to generate the surface position texture of each dense virtual viewpoint. The surface position texture is projected onto the screen space of each sparse viewpoint, sparse color textures are sampled, color weights are calculated based on depth differences and fused to generate the final color image of the target dense virtual viewpoint; according to the display device type, the final color image is synthesized into a suitable display image and rendered.

2. The method for high-fidelity real-time rendering of dynamic digital humans according to claim 1, characterized in that, The received multimodal driving signal is mapped to facial expression parameters and posture parameters, input to the parameterized human body model, and generates standard spatial geometry and skinning weights. A 3D scene representation is constructed based on the standard spatial geometry, and deformed to the target pose space according to the pose parameters and the skin weights to obtain dynamic geometric data, including: The system receives multimodal driving signals, including audio streams, video streams, or inertial sensor data streams; extracts Mel frequency cepstral coefficients and fundamental frequency features from the audio stream, extracts two-dimensional coordinates of facial landmarks and head posture from the video stream, and extracts limb joint rotation angles from the inertial sensor data; and fuses the extracted feature vectors through a temporal convolutional network to generate a low-dimensional driving latent code. The driving latent code is input into a pre-trained multilayer perceptron and mapped to expression parameters and posture parameters, respectively. The expression parameters are used to control the activation level of facial muscle units, and the posture parameters are used to control the axial rotation of the whole skeleton. The facial expression parameters and pose parameters are input into the parametric human body model, which is either a FLAME model or an SMPL-X model. The model outputs the standard space geometry, which is the set of vertex positions in the standard space, with each vertex containing three-dimensional spatial coordinates. The skin weights, facial detail normal maps, and diffuse maps in the standard space corresponding to each vertex are also obtained. Based on the standard spatial geometry, normal map, and diffuse map, a three-dimensional scene representation is constructed. The three-dimensional scene representation is a three-dimensional Gaussian sputtering field, where each Gaussian element contains a position vector, covariance matrix, color vector, and opacity value. The total number of Gaussian elements is adaptively determined according to the geometric complexity of the digital human model. Based on the rotation of each bone in the pose parameters, the transformation matrix of each bone is calculated; for each Gaussian element position in the 3D scene representation, the transformation matrix is ​​weighted and mixed using the corresponding skin weight, and the deformed position is calculated according to the linear blending skin formula to obtain the dynamic geometric data. The dynamic geometric data contains a set of transformed Gaussian elements, and each transformed Gaussian element includes a deformed position vector, a deformed covariance matrix, a color vector, and an opacity value.

3. The method for high-fidelity real-time rendering of dynamic digital humans according to claim 1, characterized in that, The process involves arranging sparse virtual cameras in the target pose space, employing a deferred rendering pipeline, and simultaneously rendering the views of all sparse virtual cameras within a single draw call to generate a sparse reference view set containing color and depth textures, including: Based on the field of view and viewing distance of the target display device, several sparse virtual cameras are arranged in the target posture space; the sparse virtual cameras adopt a convergent arrangement, with the optical axes of each camera intersecting at the center of gravity of the digital human model; The dynamic geometry data is input into the geometry processing stage of the deferred rendering pipeline; using multi-rendering target technology, a geometry buffer array is generated simultaneously for all sparse virtual cameras within a single render call. The geometry buffer array contains the normal texture, diffuse color texture, world space position texture, and material property texture for each sparse virtual camera. The normal texture in the geometric buffer array is compressed and encoded using the Lambert equal area projection method, converting the three-dimensional normal vector into two-dimensional texture coordinates for storage; the diffuse color texture is transformed using a color space based on luminance extraction, converting the red-green-blue color space into a luminance-color difference color space, and then compressed using a chromaticity subsampling format to obtain the compressed geometric buffer array. A shading rate texture is generated based on the screen space contour mask of the digital human model; the shading rate is set to 1:1 for the digital human pixel region within the contour mask; the shading rate is set to 4:4 for the non-digital human background region outside the contour mask; the shading rate texture is bound to the variable rate shading stage of the graphics pipeline, so that the pixel shader executes at a reduced frequency in the background region. During the lighting calculation phase of the deferred rendering pipeline, the lighting results of each sparse virtual camera are calculated pixel by pixel based on the geometry buffer array, and the color texture and depth texture of each sparse virtual camera are obtained to form the sparse reference view set.

4. The method for high-fidelity real-time rendering of dynamic digital humans according to claim 1, characterized in that, The step of determining dense virtual viewpoints based on the viewpoint parameters of the target display device, and projecting each sparse viewpoint pixel onto the dense virtual viewpoint screen space through downsampling depth reprojection based on the depth texture in the sparse reference view set, includes: Obtain the viewpoint parameters of the target display device, including the total number of viewpoints, horizontal field of view, viewpoint spacing, and zero parallax plane position; The number of dense virtual viewpoints is determined based on the viewpoint parameters. The dense virtual viewpoints are arranged in an equally spaced staggered manner in the horizontal direction. The staggered arrangement means that the projection center of each dense virtual viewpoint is offset by a fixed distance in the horizontal direction, and the optical axes of each viewpoint are parallel to each other. For each sparse virtual camera and each dense virtual viewpoint, establish a coordinate transformation from the sparse viewpoint screen space to the dense virtual viewpoint screen space. The coordinate transformation includes, in sequence: transforming the sparse viewpoint screen coordinates to normalized device coordinates, transforming them to the sparse viewpoint observation space through inverse perspective projection, transforming them to world space through inverse observation matrix, and then transforming them to the dense virtual viewpoint screen space through the dense virtual viewpoint observation matrix and perspective projection matrix. Based on the staggered arrangement of dense virtual viewpoints, a horizontal translation component is introduced into the perspective projection matrix of the dense virtual viewpoints. The translation component is proportional to the distance between the dense virtual viewpoint and the central viewpoint.

5. The method for high-fidelity real-time rendering of dynamic digital humans according to claim 4, characterized in that, The method of accumulating the depth contribution of each pixel using point splatter technology to generate the depth range texture of each dense virtual viewpoint includes: Obtain the depth textures from the sparse reference view set, reduce the resolution of each depth texture, and obtain downsampled depth textures; For each pixel in the downsampled depth texture, its projection coordinates in the screen space of each dense virtual viewpoint are calculated using the constructed screen space transformation relationship, and the depth value at the projection coordinates is recorded. For each dense virtual viewpoint, an accumulation buffer is allocated in screen space for each pixel location. The accumulation buffer is used to store the minimum and maximum depth values ​​at that pixel location from the projection contributions of all sparse viewpoints. For each pixel in the downsampled depth texture of each sparse viewpoint, the region of influence is determined by a radially symmetric Gaussian kernel function centered on its projected coordinates, and each target pixel within the coverage of the Gaussian kernel is taken as a contributing receiver. Calculate the distance weight between the depth value of the current projected pixel and the target pixel. Based on the distance weight, add the current depth value to the accumulation buffer of the depth range of the target pixel. If the current depth value is less than the recorded minimum depth value, update the minimum depth value. If it is greater than the maximum depth value, update the maximum depth value. The minimum and maximum depth values ​​in the cumulative buffer of each dense virtual viewpoint are written into the red and green channels of the dual-channel texture, respectively, to form the depth range texture corresponding to the dense virtual viewpoint. The resolution of the depth range texture is the same as that of the downsampled depth texture. Each pixel in the texture stores the minimum starting depth and maximum ending depth of the light step corresponding to the screen position. All the depth range textures of dense virtual viewpoints are stored as a texture array in order of viewpoint number.

6. The method for high-fidelity real-time rendering of dynamic digital humans according to claim 5, characterized in that, The stepping interval for each dense virtual viewpoint, determined by the depth range texture, includes: A light field encoded mask texture is generated asynchronously. The light field encoded mask texture is consistent with the physical resolution of the target display device. Each pixel in the texture stores the viewpoint number corresponding to the pixel position. The light field encoded mask texture is generated based on the raster tilt angle of the light field display device and the number of views in the horizontal slit period. Based on the light field encoded mask texture, determine the dense virtual viewpoint number of the corresponding pixel pixel by pixel; obtain the observation matrix and perspective projection matrix corresponding to the dense virtual viewpoint number; for each pixel in the screen space, use the pixel coordinates and the observation matrix and perspective projection matrix to calculate the unit vector of the ray direction pointing from the viewpoint origin through the target pixel to the scene. Simultaneously, the minimum and maximum depth values ​​corresponding to the current pixel coordinates are sampled from the depth range texture of the corresponding dense virtual viewpoint. The minimum depth value is divided by the Z-axis component of the unit vector of the ray direction to obtain the minimum ray stepping parameter, and the maximum depth value is divided by the Z-axis component of the unit vector of the ray direction to obtain the maximum ray stepping parameter, thereby determining the ray stepping interval.

7. The method for high-fidelity real-time rendering of dynamic digital humans according to claim 6, characterized in that, The process of sampling along the ray direction, transforming the sampling points to the observation space of each sparse viewpoint, calculating truncated signed distance values ​​using depth textures, locating the scene surface position, and generating the surface position texture for each dense virtual viewpoint includes: Starting from the minimum ray stepping parameter and ending from the maximum ray stepping parameter, the sampling point is moved successively along the ray direction with an adaptive step size to obtain the world space coordinates of the current sampling point. For each sparse virtual camera, the world space coordinates are transformed to the screen space of the sparse virtual camera through the observation matrix and perspective projection matrix of the corresponding sparse virtual camera to obtain the screen space coordinates and the corresponding observation space depth value. Based on screen space coordinates, sample the depth texture of the corresponding sparse virtual camera to obtain the surface depth value recorded by the sparse viewpoint; calculate the difference between the observation space depth value and the surface depth value to obtain a signed distance value; truncate the signed distance value. If the absolute value of the signed distance value is greater than a preset truncation distance, set the truncated signed distance value as the boundary value between positive and negative truncation distances; otherwise, leave it unchanged to obtain the truncated signed distance value of the corresponding sparse viewpoint. For each sparse virtual camera, a neighborhood window of a predetermined size is taken in the depth texture with the screen space coordinates as the center, and the variance of the depth values ​​within the window is calculated. If the world space coordinates are outside the view frustum after being transformed to the observation space of the sparse viewpoint, the weight is set to zero. Otherwise, the preset threshold is divided by the square root of the variance and compared with one to obtain the minimum value, which is used to characterize the confidence of the depth measurement of the target sparse viewpoint. The truncated signed distance values ​​of all sparse viewpoints are weighted and summed according to their corresponding fusion weights to obtain the fused truncated signed distance value; the ray stepping parameter of the current sampling point is recorded, and the ray stepping parameter is increased by one step before sampling continues, and the fused truncated signed distance value of the next sampling point is calculated; when the fused truncated signed distance values ​​of two adjacent sampling points have opposite signs, it indicates that the ray has passed through the scene surface in this interval, and the surface intersection position is located using a linear interpolation formula; The world space coordinates of the surface intersection points are stored in the surface position texture of the screen pixel of the corresponding dense virtual viewpoint; if no sign change of the fused truncated signed distance value is detected within the ray stepping interval, the corresponding surface position texture is marked as invalid background.

8. A high-fidelity real-time rendering device for dynamic digital humans, characterized in that, The device includes: The geometric data module is used to receive multimodal driving signals, map them into facial expression parameters and posture parameters, input them into a parameterized human body model, and generate standard spatial geometry and skin weights; based on the standard spatial geometry, a three-dimensional scene representation is constructed, and the model is deformed to the target posture space according to the posture parameters and the skin weights to obtain dynamic geometric data; The virtual camera module is used to deploy sparse virtual cameras in the target pose space. It uses a deferred rendering pipeline to render the views of all sparse virtual cameras simultaneously within a single draw call, generating a sparse reference view set containing color and depth textures. The depth texture module is used to determine dense virtual viewpoints based on the viewpoint parameters of the target display device. Based on the depth texture in the sparse reference view set, each sparse viewpoint pixel is projected to the dense virtual viewpoint screen space by downsampling depth reprojection, and the depth contribution of each pixel is accumulated by point splashing technology to generate the depth range texture of each dense virtual viewpoint. The virtual viewpoint module is used to determine the ray stepping interval with the depth range texture for each dense virtual viewpoint, sample along the ray direction, transform the sampled points to the observation space of each sparse viewpoint, calculate the truncated signed distance value using the depth texture, locate the scene surface position, and generate the surface position texture of each dense virtual viewpoint. The image rendering module is used to project the surface position texture onto the screen space of each sparse viewpoint, sample sparse color texture, calculate color weights based on depth differences and fuse them to generate the final color image of the target dense virtual viewpoint; and synthesize the final color image into a suitable display image and render it according to the display device type.

9. A control device, characterized in that, The device includes: A memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 7.

Images