Audio-driven three-dimensional digital human generation method and system based on neural radiation field

Abstract

The invention provides an audio-driven three-dimensional digital human generation method and system based on a neural radiation field, and the method comprises the steps: firstly constructing a posture keeping prior generation model, fusing audio features and facial posture features, learning a cross-modal interaction relation between two modal features through a self-attention mechanism and a cross-attention mechanism, and obtaining a posture keeping prior generation model; facial gesture priori information synchronized with the input audio is generated. Then, constructing a neural radiation field rendering model, performing efficient sampling on a three-dimensional space by adopting a three-plane Hash coding technology, and dynamically fusing facial posture priori and audio features through an attention mechanism; and finally, inputting the generated posture prior into a neural radiation field rendering model for volume rendering, and introducing light cumulative transmissivity calculation to generate a high-fidelity three-dimensional digital human. According to the method, the sense of reality and expressive force of the digital human are improved, the rendering calculation complexity is reduced, and dynamic coherent and lip sound synchronous real-time digital human generation is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer graphics and artificial intelligence, and in particular relates to a method and system for generating a three-dimensional digital human driven by audio based on a neural radiation field. Background Art

[0002] In applications such as virtual reality, augmented reality, and mixed reality, realistic digital humans need to be generated. Traditional digital human generation methods, such as those based on skeletal animation or expression capture, are difficult to generate high-fidelity, detailed digital humans. In recent years, the development of neural rendering technology has provided new ideas for high-quality digital human generation. Among them, NeRF (Neural Radiance Fields), as an emerging neural rendering technology, can generate high-quality new perspective images through implicit modeling of three-dimensional scenes by neural networks, providing a new technical means for digital human generation.

[0003] However, most existing NeRF models only consider static scenes and are difficult to apply to dynamic digital human generation. In addition, existing methods find it difficult to generate digital humans synchronized with the input audio signal, resulting in a lack of realism in the generated digital humans. Therefore, how to design an efficient and high-quality audio-driven digital human generation method is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an audio-driven three-dimensional digital human generation method and system based on neural radiation field, which is mainly used to solve the problems in the prior art that traditional 2D digital human synthesis lacks three-dimensional structural support, the rendering speed of 3D digital human dynamic synthesis is slow, and the expressions and movements are not natural enough and lack of realism.

[0005] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0006] In a first aspect, the present invention provides an audio-driven three-dimensional digital human generation method based on neural radiation field, comprising the following steps:

[0007] S1. Construct a posture-preserving prior generation model, extract the potential representation of audio features and facial posture feature points through convolutional neural network, and use self-attention mechanism and cross-attention mechanism to learn the cross-modal interaction relationship between audio features and facial posture feature points to generate facial posture prior information synchronized with the input audio;

[0008] S2. Build a neural radiation field rendering model, use three-plane hash coding technology to efficiently sample the three-dimensional space, and dynamically fuse facial posture feature points and audio features through the attention mechanism;

[0009] S3. Input the posture retention prior information generated in step S1 into the neural radiation field rendering model constructed in step S2, and introduce the light cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with the input audio.

[0010] In some embodiments, in S1, the training process of the posture-preserving prior generation model includes:

[0011] Audio features are extracted through two-dimensional convolution operations, and the latent representation of facial posture feature points is extracted through one-dimensional convolution;

[0012] The Transformer module is used to learn the relationship between audio features and facial posture feature points, and the information of the two modalities is fused through the cross-attention mechanism;

[0013] In the decoding stage, the facial posture feature points are used as queries, and the encoder output results are used as keys and values ​​to perform cross-attention fusion to generate the predicted values ​​of the lip feature points;

[0014] The first loss function including the predicted lip feature point error and temporal consistency loss is introduced to optimize the training process of the facial feature point generator.

[0015] In some embodiments, in S1, the audio feature is represented as a t , the facial pose feature points are represented as p t , for a t With p t Encode to get the initial audio Token {Z a ∈R T} and posture feature point Token{Z p ∈R T}, the encoding process is completed by position encoding γ(·):

[0016] Z a ,Z p =γ(a t ),γ(p t )

[0018] In some embodiments, in S1, when using the cross-attention mechanism, the audio features are used as the key (Key, K) and the value (Value, V), and the facial posture feature points are used as the query (Query, Q), to guide the generation or adjustment of the changes of the facial feature points, and to represent the changes of the lips according to the audio input:

[0019] CrossAttn(Z p ,Z a )=Attention(Q,K,V)

[0021] In some embodiments, in S1, when introducing the first loss function, it includes:

[0022] Custom positive and negative samples rely on two convolution modules to downsample facial posture feature points and audio features respectively to obtain two feature vectors of the same dimension;

[0023] The error is calculated using cross-entropy loss:

[0024]

[0025] in, Indicates the real lip feature points.

[0026] In some embodiments, in S2, when constructing the neural radiation field rendering model, it includes:

[0027] In the NeRF rendering process, the facial pose preservation prior information and the input image sequence are used as the input of NeRF, where the image sequence is parameterized by the three-dimensional coordinate X and the viewing direction d;

[0028] Three-plane hash coding technology is used to obtain spatial information, implicit function The expression is as follows:

[0029]

[0030] Where c and σ represent the color and density in the radiation field, respectively, and H is the result of the three-plane hash coding;

[0031] The information of audio features and facial posture feature points is extracted from multiple consecutive frames, and the consistency of the generated facial time is determined.

[0032] In some embodiments, in S2, it further includes:

[0033] The three-dimensional coordinates X of each frame are projected in three directions in two dimensions, and the spatial information H(X) of the frame is obtained by splicing. The specific expression is:

[0034]

[0035] in, Represents the projection of the plane direction.

[0036] In some embodiments, it also includes:

[0037] Audio Features and feature point features The cross attention calculation is performed by the following formula:

[0038]

[0039] The fused features are used to guide the volume rendering process of NeRF, and its rendering formula is:

[0040]

[0041] Among them, t 0 and t 1 are the near and far boundaries of the ray r, respectively, and T(t) is the distance from t to 0 The cumulative transmittance of and t is expressed as follows:

[0042]

[0043] In some embodiments, the neural radiance field rendering model uses a second loss function to reduce the error between the rendered image and the real image, and the second loss function is:

[0044]

[0045] in, is the set of camera rays, C gt is the color of the real image.

[0046] In a second aspect, the present invention provides a system for use in the above method, comprising:

[0047] The first processing module is used to build a posture-preserving prior generation model, extract the potential representation of audio features and facial posture feature points through a convolutional neural network, and use the self-attention mechanism and cross-attention mechanism to learn the cross-modal interaction relationship between audio features and facial posture feature points to generate facial posture prior information synchronized with the input audio;

[0048] The second processing module is used to build a neural radiation field rendering model, which uses three-plane hash coding technology to efficiently sample the three-dimensional space and dynamically fuses facial posture feature points and audio features through an attention mechanism;

[0049] The three-dimensional generation module is used to input the generated posture-keeping prior information into the constructed neural radiation field rendering model, and introduce the light cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with the input audio.

[0050] Compared with the prior art, the present invention has at least the following beneficial effects:

[0051] 1. By fusing audio features and facial posture features to construct a posture-preserving prior generation model, facial expressions and lip shapes that are synchronized with the input audio can be generated, thereby improving the realism of the generated three-dimensional digital human. Specifically, the present invention uses a convolutional neural network to extract the potential representation of audio and facial posture features, and learns the cross-modal interaction relationship between the two modal features through self-attention mechanisms and cross-attention mechanisms. In this way, the generated facial posture prior information not only includes the dynamic changes of facial posture, but also incorporates the timing information synchronized with the audio, so that the generated digital human can adjust facial expressions and lip shapes in real time according to the input audio, thereby enhancing the realism and interactivity of the character.

[0052] 2. The three-plane hash coding technology is used to efficiently sample the three-dimensional space, which reduces the computational complexity and improves the efficiency of neural radiation field rendering; the traditional NeRF model requires dense sampling of the three-dimensional space, which requires a large amount of calculation and is difficult to achieve real-time rendering; the present invention uses three-plane hash coding technology to map the three-dimensional space to three orthogonal planes, and splices the features of each plane to obtain a compact spatial representation. This method can significantly reduce the number of spatial sampling points and reduce the computational burden, while retaining sufficient spatial information. Therefore, the present invention can significantly improve the rendering speed while ensuring the rendering quality, and realize the real-time generation of digital humans;

[0053] 3. By integrating facial posture prior information and audio-driven features in the neural radiation field rendering process, a high-fidelity three-dimensional digital human with dynamic coherence and lip synchronization can be generated. Specifically, the present invention fuses the facial posture prior information output by the posture-preserving prior generation model with the input of the neural radiation field rendering model, and uses the attention mechanism to dynamically adjust the influence of posture priors and audio features on the rendering results. This fusion method can add explicit facial posture and audio-driven information on the basis of the implicit representation of the neural radiation field, so that the rendering process can be dynamically adjusted according to the input audio and posture priors to generate coherent and natural character movements and expressions. At the same time, by introducing the calculation of cumulative light transmittance, the present invention can also generate three-dimensional digital humans with high-fidelity details and realism, showing realistic skin, hair and clothing effects.

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The present invention is further described using the accompanying drawings, but the embodiments in the accompanying drawings do not constitute any limitation to the present invention. A person skilled in the art can obtain other drawings based on the following drawings without creative work.

[0056] Figure 1The present invention is a flowchart of a method for generating a three-dimensional digital human driven by audio based on a neural radiation field under an embodiment.

[0057] Figure 2 The present invention is a schematic diagram of a framework of an audio-driven three-dimensional digital human generation system based on a neural radiation field under an embodiment. DETAILED DESCRIPTION

[0058] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0060] The applicant found that:

[0061] The prior art has the following methods:

[0062] (1) Data-driven approach

[0063] Data-driven speaker synthesis technology aims to generate animation effects synchronized with facial movements based on input data (such as audio). Audio-driven technology achieves synchronization through the mapping relationship between audio and facial movements. Early methods are mostly based on generative adversarial networks (GANs) and use facial landmarks (such as lip shape) as constraints. In recent years, NeRF-based 3D reconstruction technology has gradually been applied to audio-driven synthesis, improving the quality and realism of generation. However, data-driven methods have high computational costs, high requirements on hardware equipment, and expensive training costs. There is also room for improvement in the naturalness and scalability of generated facial animations, and there are still challenges in improving scalability and naturalness of details.

[0064] (2) 2D digital human synthesis

[0065] In recent years, 2D digital human synthesis has made great progress with the development of deep learning technology, especially the application of generative adversarial networks (GANs) and diffusion models. Early GAN methods achieved the synthesis of facial animations through image generation models, but due to the lack of three-dimensional structure, the generated facial expressions and movements were still insufficient in realism. In contrast, 2D video generation methods based on diffusion models rely on space-time decomposition technology and perform well in visual generation quality and temporal consistency. Representative works such as Video Diffusion Models (VDM) use U-Net architecture to process joint training of images and videos, and ImagenVideo generates high-resolution videos through cascaded diffusion models. Although diffusion model-based technologies have made significant progress in generating high-fidelity videos, slow inference speed is still the main bottleneck of this type of method; in addition, although the GAN method provided an image-to-image generation path in the early days, it lacked three-dimensional support, resulting in a lack of realism in the generated effect.

[0066] (3) 3D digital human synthesis (implicit expression)

[0067] Since its proposal in 2020, the implicit neural radiance field (NeRF) technology has rapidly become a core tool in the field of 3D reconstruction and synthesis. AD-NeRF first proposed a 3D radiance field training method driven by audio data to generate high-quality dynamic facial animations. Subsequent studies such as RAD-NeRF and DFRF further encode spatial coordinates and audio features through a small network to achieve real-time synthesis, significantly reducing training costs. Although the NeRF method has achieved excellent performance in generating highly realistic 3D digital humans, the complexity and slow rendering speed brought by its implicit expression make it difficult to meet large-scale application needs. How to improve rendering efficiency and enhance the versatility of the model remains an important direction for future research.

[0068] In view of this, refer to Figure 1 The present invention provides a method for generating a three-dimensional digital human driven by audio based on a neural radiation field, comprising the following steps:

[0069] S1. Construct a posture-preserving prior generation model, extract the potential representation of audio features and facial posture feature points through convolutional neural network, and use self-attention mechanism and cross-attention mechanism to learn the cross-modal interaction relationship between audio features and facial posture feature points to generate facial posture prior information synchronized with the input audio;

[0070] S2. Build a neural radiation field rendering model, use three-plane hash coding technology to efficiently sample the three-dimensional space, and dynamically fuse facial posture feature points and audio features through the attention mechanism;

[0071] S3. Input the posture retention prior information generated in step S1 into the neural radiation field rendering model constructed in step S2, and introduce the light cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with the input audio.

[0072] In this work, we propose a two-stage framework that combines pose-preserving prior-driven features and NeRF rendering to synthesize high-fidelity audio-driven portrait sequences with better rendering quality:

[0073] Specifically, facial pose feature points and audio are used to form pose-preserving priors for synthesizing portraits. Facial pose feature points are composed of unified lip-sync feature points and pose feature points. Since facial pose feature points have a unique matching relationship with facial movements, we add them to the training to increase the driving information and the stability of the synthesized sequence, while also improving the generalization problem caused by training with only a small amount of audio. In order to obtain feature points corresponding to arbitrary audio, we implemented a Transformer encoder based on a cross-attention mechanism, which combines the features of the two modalities of pose feature point embedding and audio embedding, and finally decodes and synthesizes the lip-sync feature points.

[0074] In the radiance field network, in order to improve sampling accuracy and convergence speed, we use three-plane hash coding sampling to obtain spatial information. Using audio and facial landmarks as dynamic input, we also use the attention mechanism to fuse audio and facial landmarks to extract driving feature information, and finally connect it with spatial information as training data input into the portrait radiance field.

[0075] As an implementation, in S1, the training process of the posture-preserving prior generation model includes:

[0076] The audio features are extracted through two-dimensional convolution operation to obtain the audio feature representation a t ; For posture feature points, since they are essentially a one-dimensional point set data, the potential representation of facial posture feature points is extracted through one-dimensional convolution to obtain the feature point feature p t ;

[0077] The Transformer module is used to learn the relationship between audio features and facial posture feature points, and the information of the two modalities is fused through the cross-attention mechanism;

[0078] In the decoding stage, the facial posture feature points are used as queries, and the encoder output results are used as keys and values ​​to perform cross-attention fusion to generate the predicted values ​​of the lip feature points;

[0079] The first loss function including the predicted lip feature point error and temporal consistency loss is introduced to optimize the training process of the facial feature point generator.

[0080] In this embodiment, by introducing a facial feature point generator and using audio-driven posture retention prior information to guide NeRF rendering, the model's synchronization control of facial movements is enhanced, making the generated face movements more stable. This technology effectively avoids the inconsistency problem between audio and facial movements, and improves the temporal consistency and naturalness of the synthesized face.

[0081] Through the Transformer module and cross-attention mechanism, the deep fusion of audio features and facial posture features is achieved, and the accuracy of facial posture prior information is improved. The lip feature point prediction error and temporal consistency loss are introduced to optimize the training process of the facial feature point generator, making the generated facial posture prior more synchronized and coherent with the audio.

[0082] Preferably, in S1, the Transformer module is used to learn the relationship between the audio feature and the facial posture feature point feature, and the audio feature is represented as a t , the facial pose feature points are represented as p t , for a t With p t Encode to get the initial audio Token {Z a ∈R T} and posture feature point Token{Z p ∈R T}, the encoding process is completed by position encoding γ(·):

[0083] Z a ,Z p =γ(a t ),γ(p t )

[0085] Preferably, in S1, in order to better capture the information interaction between audio and feature points, this embodiment introduces a cross-attention mechanism to overcome the limitation that simple feature concatenation and self-attention mechanisms are difficult to learn multimodal information. Specifically, audio features are used as keys (Key, K) and values ​​(Value, V), because audio provides the basic conditions for generating facial posture feature points; facial posture feature points are used as queries (Query, Q) to guide the generation or adjustment of changes in facial feature points, and to represent changes in lips according to audio input, so as to help the model understand how lips should change according to specific audio input:

[0086] CrossAttn(Z p ,Z a )=Attention(Q,K,V)

[0088] In addition, in the decoding stage, this embodiment uses the posture feature point as the query, and the CrossAttn result output by the encoder as the key and value, and performs cross-attention fusion again to finally obtain the predicted value of the lip feature point

[0089]

[0090] In order to ensure the accuracy of generating feature points, this embodiment adopts the first loss function □ landmark , the loss function includes L 1 Error, and L set to ensure timing consistency 2 loss.

[0091] Specifically, in S1, in order to better infer the facial feature points synchronized with the audio, this embodiment introduces a new loss value calculation to quantify the synchronization between the facial feature points and the audio; based on the principles of CLIP technology and Wav2lip algorithm, a loss calculation based on 2D landmark and audio synchronization, namely L_sync, is proposed. The implementation of the first loss function includes:

[0092] Custom positive and negative samples rely on two convolution modules to downsample facial posture feature points and audio features respectively to obtain two feature vectors of the same dimension;

[0093] The error is calculated using cross-entropy loss:

[0094]

[0095] in, Indicates the real lip feature points.

[0096] This first loss function combines generation accuracy and timing consistency, ensuring the smoothness and accuracy of lip movements. Through this technical solution, the present invention can achieve accurate audio-driven lip synchronization generation and improve the timing consistency and naturalness of the synthesized portrait.

[0097] In this embodiment, the encoding method of audio features and facial posture feature points, the calculation process of the cross-attention mechanism, and the calculation method of the first loss function are further defined. Through position encoding and cross-attention mechanism, the temporal information and interaction relationship of audio features and facial posture feature points are fully utilized to improve the accuracy of posture prior information. The introduction of custom positive and negative samples and cross-entropy loss effectively optimizes the prediction accuracy of lip feature points, making the generated digital human lip movements more consistent with the input audio.

[0098] As an implementation, in S2, when constructing the neural radiation field rendering model, a NeRF-based renderer module (NERF-BASED RENDERER) is constructed in this embodiment to generate realistic speaker portrait images. This module combines the posture-preserving prior information generated from the input audio and performs portrait synthesis through the three-dimensional neural radiation field model, solving the problem of the lack of three-dimensional realism in 2D portraits in the prior art. Specifically:

[0099] In the NeRF rendering process, the facial pose preservation prior information and the input image sequence are used as the input of NeRF, where the image sequence is parameterized by the three-dimensional coordinate X and the viewing direction d;

[0100] In order to improve the sampling efficiency of spatial information and reduce the impact of hash conflicts on training, a three-plane hash coding technique is used to obtain spatial information. The expression is as follows:

[0101]

[0102] Where c and σ represent the color and density in the radiation field, respectively, and H is the result of the three-plane hash coding;

[0103] The information of audio features and facial posture feature points is extracted from multiple consecutive frames, and the consistency of the generated facial time is determined.

[0104] By adopting three-plane hash coding technology, the sampling efficiency of three-dimensional spatial information is significantly improved, and the hash conflict problem that may occur in traditional NeRF is avoided. Especially when processing dynamic scenes, the convergence speed and rendering efficiency of the model are improved, and the training and reasoning process are optimized.

[0105] Preferably, in S2, in order to deal with the hash collision problem in dynamic scenes and capture more delicate dynamic information, it also includes:

[0106] The three-dimensional coordinates X of each frame are projected in three directions in two dimensions, and the spatial information H(X) of the frame is obtained by splicing. The specific expression is:

[0107]

[0108] in, Represents the projection of the plane direction.

[0109] Preferably, in order to better extract the multimodal information of audio and facial feature points, an attention mechanism is used to perform feature fusion on the audio and facial feature points, specifically:

[0110] Audio Features and feature point features The cross attention calculation is performed by the following formula:

[0111]

[0112] The fused features are used to guide the volume rendering process of NeRF, and its rendering formula is:

[0113]

[0114] Among them, t 0 and t 1 are the near and far boundaries of the ray r, respectively, and T(t) is the distance from t to 0 The cumulative transmittance of and t is expressed as follows:

[0115]

[0116] Through the cross-modal attention mechanism, the audio and facial landmark features are deeply integrated, which improves the model's ability to understand and generate multimodal data, ensuring that the synthesized portraits can maintain high-quality dynamic performance under a variety of input conditions.

[0117] Preferably, the error between the rendered image and the real image is reduced by using a second loss function through the neural radiation field rendering model, and the second loss function is:

[0118]

[0119] in, is the set of camera rays, C gt is the color of the real image.

[0120] In this embodiment, the construction process of the neural radiation field rendering model is further defined, including the input of posture prior information and image sequences, three-plane hash coding, cross-attention fusion of audio features and posture feature points, and volume rendering of the neural radiation field. At the same time, a second loss function is introduced to optimize the training process of the neural radiation field rendering model. Through the three-plane hash coding technology, a compact spatial information representation is obtained, the computational complexity of spatial sampling is reduced, and the rendering efficiency is improved. The cross-attention mechanism realizes the dynamic fusion of audio features and facial posture feature points, and improves the spatiotemporal consistency in the rendering process. The introduction of light cumulative transmittance calculation and the second loss function optimizes the rendering quality of the neural radiation field and generates a high-fidelity, detail-rich three-dimensional digital human.

[0121] In some possible embodiments, an expression intensity control parameter is introduced into the posture retention prior generation model. By adjusting this parameter, the amplitude of expression changes in the generated facial posture prior information can be controlled. At the same time, in the neural radiation field rendering model, the fusion weights of facial feature points are dynamically adjusted according to the expression intensity parameter, so that the generated digital human facial expression can be adjusted between exaggeration and subtlety according to needs. Specifically:

[0122] 1. Add expression intensity control parameters to the input of the posture preservation prior generation model, and fuse them with audio features and facial posture feature points through the self-attention mechanism to generate facial posture prior information that matches the expression intensity.

[0123] 2. In the neural radiation field rendering model, the fusion weights of facial feature points are set according to the expression intensity parameter. When the expression intensity parameter is large, the fusion weights of facial feature points are increased to generate exaggerated expression changes; when the expression intensity parameter is small, the fusion weights of facial feature points are reduced to generate subtle expression changes.

[0124] 3. Design an expression intensity loss function to optimize the learning process of expression intensity control parameters by calculating the difference between the generated expression and the expected expression intensity, thereby improving the accuracy of expression control.

[0125] In this embodiment, by introducing the expression intensity control mechanism, the generated digital human can realize facial expression regulation from exaggerated to subtle according to application requirements, thereby improving the expressiveness and applicability of the digital human and meeting the requirements for fine control of facial expressions in different scenarios.

[0126] In some possible embodiments, this method can be applied to the generation of digital humans driven by audio at the whole body level. On the basis of the original audio-driven facial generation, human posture information is introduced as an additional input condition. By extending the posture-preserving prior generation model and the neural radiation field rendering model, human posture features are fused with audio features and facial features to generate full-body movements and facial expressions synchronized with the audio, and achieve full-body audio-driven digital human generation. Specifically:

[0127] 1. Use the human posture estimation algorithm to extract the key point information of the human body from the input video to obtain the human posture features.

[0128] 2. Expand the posture-preserving prior generation model, fuse human posture features with audio features and facial posture feature points, learn the cross-modal interaction relationship between the three through self-attention mechanism and cross-attention mechanism, and generate prior information of whole-body posture and facial expression.

[0129] 3. Expand the neural radiation field rendering model, introduce human posture features as additional conditional input, sample the three-dimensional space of the whole body through three-plane hash coding, and use the attention mechanism to dynamically fuse human posture, facial feature points and audio features.

[0130] 4. Introduce a full-body posture consistency loss function to optimize the full-body level audio-driven digital human generation process by calculating the difference between the generated full-body posture and the real posture.

[0131] In this embodiment, by integrating human posture information, audio-driven digital human generation is extended from the face to the whole body, so that the generated digital human can synchronously produce whole-body movements and facial expressions according to the audio information, thereby improving the realism and interactivity of the digital human and expanding the application scenarios of audio-driven digital human generation.

[0132] Second, refer to Figure 2 This embodiment provides an audio-driven three-dimensional digital human generation system based on neural radiation field, which uses the generation method of the above embodiments, including:

[0133] The first processing module is used to build a posture-preserving prior generation model, extract the potential representation of audio features and facial posture feature points through a convolutional neural network, and use the self-attention mechanism and cross-attention mechanism to learn the cross-modal interaction relationship between audio features and facial posture feature points to generate facial posture prior information synchronized with the input audio;

[0134] The second processing module is used to build a neural radiation field rendering model, which uses three-plane hash coding technology to efficiently sample the three-dimensional space and dynamically fuses facial posture feature points and audio features through an attention mechanism;

[0135] The three-dimensional generation module is used to input the generated posture-keeping prior information into the constructed neural radiation field rendering model, and introduce the light cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with the input audio.

[0136] This embodiment realizes the end-to-end generation of high-fidelity three-dimensional digital human from input audio through the collaborative work of the first processing module, the second processing module and the three-dimensional generation module. The system integrates key technologies such as audio drive, facial posture prior and neural radiation field rendering, and generates realistic digital human with coherent movements, rich expressions and lip synchronization, providing high-quality digital human assets for applications such as virtual reality, augmented reality and mixed reality.

[0137] The following is an example description:

[0138] Embodiment 1:

[0139] In order to compare the results generated by different algorithms, this embodiment shows the digital human inferred by each algorithm under three key frames under self-driving. The LA-NeRF column is the generation result of this embodiment, which is more accurate and clear than other methods. More accurate reconstruction results can be obtained in lip reconstruction. This phenomenon also proves that there is a many-to-many phenomenon between audio and mouth movement, which will cause blurred mouth reconstruction.

[0140] In addition, when the mouth has a large movement, such as pouting or opening the mouth, the method of this embodiment can synthesize the mouth movement more robustly and reconstruct more details. Since the posture is fixed, the mouth feature points in the landmark are what really drive the digital human.

[0141] Furthermore, in this embodiment, data-driven synthesis is performed on the same identity test set, and the quality of portrait reconstruction is compared. The following table shows the quantitative evaluation using different methods:

[0142] Table 1. Quantitative evaluation data of different methods

[0143]

[0144] Among them, the LA-NeRF column is the evaluation data of this embodiment, which uses peak signal-to-noise ratio (PSNR), image feature distance (FID), boundary distance (LMD) and learning perceptual image patch similarity (LPIPS) to measure the similarity between the generated image and the real image; the synchronization network confidence score (Sync) is used to evaluate lip synchronization.

[0145] The evaluation results are shown in Table 1. Compared with the best recent methods, they are:

[0146] (1) The method of this embodiment shows excellent quality in most indicators.

[0147] (2) Specifically, from the perspective of video reconstruction, the rendering effect of this embodiment maintains the best visual quality. It is observed that the wav2lip and makeittalk algorithms have low fidelity for image reconstruction and perform poorly on FID and LPIPS. ER-NeRF alleviates the impact of high-dimensional hash conflicts in sampling point encoding, has better face reconstruction results, and performs better in LMD.

[0148] In summary, compared with the prior art, the above embodiment provides an audio-driven 3D digital human generation method and system based on neural radiation field, and at least the following technical optimizations are made to address the defects of the prior art:

[0149] 1. Improve sampling accuracy and convergence speed: The traditional NeRF method faces the problems of low sampling efficiency and slow convergence speed in high-precision 3D scene reconstruction. We have greatly improved the efficiency of spatial information extraction by introducing three-plane hash coding sampling, and combined with facial feature points and audio data to achieve a more efficient sampling process, shorten training time, and improve the convergence speed of the model;

[0150] 2. Optimize dynamic feature fusion: Although NeRF performs well in static scene generation, it has the problem of unstable motion generation when processing dynamic characters. This invention uses a cross-modal attention mechanism to dynamically fuse facial feature points and audio information, making the generated motion smoother, improving the stability of facial expressions and lip synchronization, and reducing blur and flicker in dynamic scenes;

[0151] 3. Enhance the generalization ability of the model: The generalization ability of the existing NeRF model is weak, especially when there is insufficient training data, the generation quality is prone to decline. This paper introduces a posture preservation prior module based on audio and facial feature points. Through joint training of multimodal data, the stability and generalization ability of the model under different input conditions are improved, and the impact of data scarcity on generation quality is reduced;

[0152] Therefore, this method realizes an end-to-end audio-driven digital human generation process through the organic combination of the two stages of posture-preserving prior generation and neural radiation field rendering, simplifies the traditional process, and improves the generation efficiency and quality. Traditional digital human generation methods usually need to capture and model facial expressions, body movements and sounds separately, and then synchronize and fuse them. The process is complicated and prone to misalignment and unnaturalness. The present invention automatically completes the mapping from audio to three-dimensional digital humans through end-to-end neural network learning, avoiding complex manual processing and synchronization operations. Therefore, the present invention provides a simple, efficient, and fully automated audio-driven digital human generation solution, which greatly reduces the threshold and cost of digital human production and promotes the promotion and application of digital human technology.

[0153] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. An audio-driven three-dimensional digital human generation method based on neural radiation field, characterized in that: The following steps are involved: S1. Construct a posture-preserving prior generation model, extract the potential representation of audio features and facial posture feature points through convolutional neural network, and use self-attention mechanism and cross-attention mechanism to learn the cross-modal interaction relationship between audio features and facial posture feature points to generate facial posture prior information synchronized with the input audio; S2. Build a neural radiation field rendering model, use three-plane hash coding technology to efficiently sample the three-dimensional space, and dynamically fuse facial posture feature points and audio features through the attention mechanism; S3. Input the posture retention prior information generated in step S1 into the neural radiation field rendering model constructed in step S2, and introduce the light cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with the input audio.

2. The method according to claim 1, characterized in that In S1, the training process of the pose-preserving prior generative model includes: Audio features are extracted through two-dimensional convolution operations, and the latent representation of facial posture feature points is extracted through one-dimensional convolution; The Transformer module is used to learn the relationship between audio features and facial posture feature points, and the information of the two modalities is fused through the cross-attention mechanism; In the decoding stage, the facial posture feature points are used as queries, and the encoder output results are used as keys and values ​​to perform cross-attention fusion to generate the predicted values ​​of the lip feature points; The first loss function including the predicted lip feature point error and temporal consistency loss is introduced to optimize the training process of the facial feature point generator.

3. The method according to claim 2, characterized in that In S1, the audio feature is represented as a t , the facial pose feature points are represented as p t , for a t With p t Encode to get the initial audio Token {Z a ∈R T } and posture feature point Token{Z p ∈R t }, the encoding process is completed by position encoding γ(·): Z a ,Z p =γ(a t ),γ(p t ) 4. The method according to claim 3, characterized in that In S1, when using the cross-attention mechanism, the audio features are used as the key (Key, K) and value (Value, V), and the facial posture feature points are used as the query (Query, Q), which guides the generation or adjustment of the changes in facial feature points to represent the changes in the lips according to the audio input: CrossAttn(Z p ,Z a )=Attention(Q,K,V) 5. The method according to claim 4, characterized in that In S1, when the first loss function is introduced, it includes: Custom positive and negative samples rely on two convolution modules to downsample facial posture feature points and audio features respectively to obtain two feature vectors of the same dimension; The error is calculated using cross-entropy loss: in, Indicates the real lip feature points.

6. The method according to any one of claims 1 to 5, characterized in that: In S2, when building a neural radiation field rendering model, it includes: In the NeRF rendering process, the facial pose prior information is used together with the input image sequence as The input of NeRF is an image sequence parameterized by 3D coordinates X and viewing direction d. Three-plane hash coding technology is used to obtain spatial information, implicit function The expression is as follows: Where c and σ represent the color and density in the radiation field, respectively, and H is the result of the three-plane hash coding; The information of audio features and facial posture feature points is extracted from multiple consecutive frames, and the consistency of the generated facial time is determined.

7. The method according to claim 6, characterized in that In S2, it also includes: The three-dimensional coordinates X of each frame are projected in three directions in two dimensions, and the spatial information H(X) of the frame is obtained by splicing. The specific expression is: in, Represents the projection of the plane direction.

8. The method according to claim 7, characterized in that Also includes: Audio Features and feature point features The cross attention calculation is performed by the following formula: The fused features are used to guide the volume rendering process of NeRF, and its rendering formula is: Among them, t0 and t1 are the near and far boundaries of the light r, respectively, and T(t) is the cumulative transmittance of the light from t0 and t, expressed as follows:

9. The method according to claim 8, characterized in that The neural radiance field rendering model uses the second loss function to reduce the error between the rendered image and the real image. The second loss function is: in, is the set of camera rays, C gt is the color of the real image.

10. A system applied to the method according to any one of claims 1 to 9, characterized in that: include: The first processing module is used to build a posture-preserving prior generation model, extract the potential representation of audio features and facial posture feature points through a convolutional neural network, and use the self-attention mechanism and cross-attention mechanism to learn the cross-modal interaction relationship between audio features and facial posture feature points to generate facial posture prior information synchronized with the input audio; The second processing module is used to build a neural radiation field rendering model, which uses three-plane hash coding technology to efficiently sample the three-dimensional space and dynamically fuses facial posture feature points and audio features through an attention mechanism; The three-dimensional generation module is used to input the generated posture-keeping prior information into the constructed neural radiation field rendering model, and introduce the light cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with the input audio.

Images