A method and system for generating an audio-driven three-dimensional digital human based on a neural radiance field

Abstract

The application provides a kind of audio-driven three-dimensional digital human generation method and system based on neural radiance field, first, a pose preserving prior generation model is constructed, by fusing audio features and facial pose features, the cross-modal interaction relationship between the two modal features is learned using self-attention mechanism and cross-attention mechanism, to generate facial pose prior information synchronized with input audio. Then, a neural radiance field rendering model is constructed, three-plane hash coding technology is used to efficiently sample the three-dimensional space, and the facial pose prior and audio features are dynamically fused through the attention mechanism. Finally, the generated pose prior is input into the neural radiance field rendering model for volume rendering, and the light cumulative transmittance calculation is introduced to generate high-fidelity three-dimensional digital humans. The application improves the realism and expressiveness of digital humans, reduces the rendering computation complexity, and realizes real-time digital human generation with dynamic coherence and lip-synchronized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer graphics and artificial intelligence, and particularly relates to a neural radiance field-based audio-driven three-dimensional digital human generation method and system. BACKGROUND

[0002] In virtual reality, augmented reality, mixed reality and other applications, realistic digital humans need to be generated. Traditional digital human generation methods, such as methods based on skeletal animation or expression capture, are difficult to generate high-fidelity, detailed digital humans. In recent years, the development of neural rendering technology has provided a new approach to high-quality digital human generation. Among them, NeRF (Neural Radiance Fields) as a new neural rendering technology, implicitly models a three-dimensional scene through a neural network, and can generate high-quality new view images, providing a new technical means for digital human generation.

[0003] However, existing NeRF models mostly only consider static scenes and are difficult to apply to dynamic digital human generation. In addition, existing methods are difficult to generate digital humans synchronized with input audio signals, resulting in a lack of realism in the generated digital humans. Therefore, how to design an efficient and high-quality audio-driven digital human generation method is a technical problem to be solved. SUMMARY

[0004] In order to overcome the deficiencies of the prior art, the purpose of the present application is to provide a neural radiance field-based audio-driven three-dimensional digital human generation method and system, mainly used to solve the problems of unnatural expressions and insufficient realism caused by the lack of three-dimensional structural support in traditional 2D digital human synthesis and slow rendering speed in 3D digital human dynamic synthesis in the prior art.

[0005] To solve the above problems, the technical scheme adopted by the present application is as follows:

[0006] In a first aspect, the present application provides a neural radiance field-based audio-driven three-dimensional digital human generation method, comprising the following steps:

[0007] S1, a pose-preserving prior generation model is constructed, the latent representations of audio features and facial pose feature points are extracted through a convolutional neural network, and the cross-modal interaction relationship between the audio features and the facial pose feature points is learned using a self-attention mechanism and a cross-attention mechanism to generate facial pose prior information synchronized with the input audio;

[0008] S2, a neural radiance field rendering model is constructed, three-plane hash coding technology is used to efficiently sample a three-dimensional space, and facial pose feature points and audio features are dynamically fused through an attention mechanism;

[0009] S3, input the pose maintaining prior information generated in step S1 into the neural radiance field rendering model constructed in step S2, and introduce light ray cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with the input audio.

[0010] In some embodiments, in S1, the training process of the pose maintaining prior generation model includes:

[0011] Audio features are extracted by two-dimensional convolution operation, and latent representations of facial pose feature points are extracted by one-dimensional convolution;

[0012] The relationship between audio features and facial pose feature point features is learned by using a Transformer module, and the information of the two modalities is fused by cross-attention mechanism;

[0013] In the decoding stage, the facial pose feature points are taken as queries, and the results output by the encoder are taken as keys and values, cross-attention fusion is performed, and the prediction value of the lip feature points is generated;

[0014] A first loss function including prediction lip feature point error and timing consistency loss is introduced to optimize the training process of the facial feature point generator.

[0015] In some embodiments, in S1, the audio feature is represented as a t , the facial pose feature point is represented as p t , a t and p t are encoded to obtain initial audio Token{Z a ∈R T} and pose feature point Token{Z p ∈R T}, and the encoding process is completed by position encoding γ(·):

[0016] Z a ,Z p =γ(a t ),γ(p t )

[0017] In some embodiments, in S1, when using the cross-attention mechanism, the audio feature is taken as the key (Key, K) and the value (Value, V), and the facial pose feature point is taken as the query (Query, Q) to guide the generation or adjustment of the change of the facial feature point, and the change of the lip is represented according to the audio input:

[0018] CrossAttn(Z p ,Z a )=Attention(Q,K,V)

[0019] In some embodiments, in S1, when introducing the first loss function, comprising:

[0020] Custom positive and negative samples, relying on two convolution modules to downsample facial pose feature points and audio features respectively, to obtain two feature vectors of the same dimension;

[0021] The error is calculated using cross-entropy loss Cross-Entropy:

[0022]

[0023] Wherein, Indicates the real lip feature point.

[0024] In some embodiments, in S2, when constructing the neural radiance field rendering model, comprising:

[0025] In the NeRF rendering process, the facial pose maintains the prior information and the input image sequence as the input of NeRF, wherein the image sequence is parameterized by three-dimensional coordinates X and viewing direction d;

[0026] The three-plane hash coding technology is adopted to obtain the spatial information, and the implicit function The expression is as follows:

[0027]

[0028] Wherein, c and σ respectively represent the color and density in the radiance field, and H is the result of three-plane hash coding;

[0029] The information of audio features and facial pose feature points is extracted from multiple consecutive frames, and the consistency of the generated facial time is judged.

[0030] In some embodiments, in S2, further comprising:

[0031] The three-dimensional coordinates X of each frame are projected in three directions, and the spatial information H(X) of the frame is spliced, and the specific expression is:

[0032]

[0033] Wherein, Indicates the projection of the plane direction.

[0034] In some embodiments, further comprising:

[0035] Audio features And feature point features Cross-attention calculation is performed by the following formula:

[0036]

[0037] The fused features are used to guide the volume rendering process of NeRF, and the rendering formula is:

[0038]

[0039] Where t0 and t1 are the near and far boundaries of the ray r respectively, and T(t) is the cumulative transmittance of the ray from t0 to t, and the expression is as follows:

[0040]

[0041] In some embodiments, the neural radiance field rendering model reduces the error between the rendered image and the real image by using a second loss function, and the second loss function is:

[0042]

[0043] Where, is a set of camera rays, C gt is the color of the real image.

[0044] In a second aspect, the present application provides a system applied to the above method, comprising:

[0045] A first processing module is configured to construct a pose preservation prior generation model, extract latent representations of audio features and facial pose feature points through a convolutional neural network, and learn cross-modal interaction relationships between the audio features and the facial pose feature points by using a self-attention mechanism and a cross-attention mechanism to generate facial pose prior information synchronized with the input audio.

[0046] A second processing module is configured to construct a neural radiance field rendering model, use a three-plane hash coding technology to efficiently sample a three-dimensional space, and dynamically fuse facial pose feature points and audio features through an attention mechanism.

[0047] A three-dimensional generation module is configured to input the generated pose preservation prior information into the constructed neural radiance field rendering model, introduce ray cumulative transmittance calculation, and generate a high-fidelity three-dimensional digital human synchronized with the input audio.

[0048] Compared with the prior art, the present application has at least the following beneficial effects:

[0049] 1. By fusing audio features and facial pose features to construct a pose-preserving prior generation model, facial expressions and lip shapes synchronized with input audio can be generated, improving the realism of the generated digital human; Specifically, the present application uses a convolutional neural network to extract the latent representation of audio and facial pose features, and learns the cross-modal interaction between the two modal features through a self-attention mechanism and a cross-attention mechanism. In this way, the generated facial pose prior information not only contains the dynamic changes of facial pose, but also fuses the timing information synchronized with the audio, so that the generated digital human can adjust facial expressions and mouth shapes in real time according to the input audio, enhancing the realism and interactivity of the character;

[0050] 2. The three-plane hash coding technology is used to efficiently sample the three-dimensional space, reducing the computational complexity and improving the efficiency of neural radiation field rendering; The traditional NeRF model needs to densely sample the three-dimensional space, which has large amount of calculation and is difficult to realize real-time rendering; The present application uses three-plane hash coding technology to map three-dimensional space to three orthogonal planes, and splices the features of each plane to obtain a compact spatial representation. This method can significantly reduce the number of spatial sampling points, reduce the computational burden, and also retain sufficient spatial information. Therefore, the present application can significantly improve the rendering speed while ensuring the rendering quality, realizing the real-time generation of digital human;

[0051] 3. The facial pose prior information and audio driven features are fused in the process of neural radiation field rendering, so as to generate a high-fidelity three-dimensional digital human with dynamic and coherent lip synchronization. Specifically, the present application fuses the facial pose prior information output by the pose-preserving prior generation model with the input of the neural radiation field rendering model, and dynamically adjusts the influence of the pose prior and audio features on the rendering result by using the attention mechanism. This fusion method can add explicit facial pose and audio driven information on the basis of the implicit representation of neural radiation field, so that the rendering process can dynamically adjust according to the input audio and pose prior, generating coherent and natural character actions and expressions. At the same time, by introducing the calculation of light cumulative transmittance, the present application can also generate a three-dimensional digital human with high-fidelity details and realism, showing realistic skin texture, hair and clothing effects.

[0052] The present application will be further described in detail in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0053] The present application will be further described in conjunction with the accompanying drawings and specific embodiments.

[0054] Figure 1Fig. 1 is a flowchart of a method for generating an audio-driven three-dimensional digital human based on a neural radiance field according to an embodiment.

[0055] Figure 2 Fig. 2 is a schematic diagram of a system for generating an audio-driven three-dimensional digital human based on a neural radiance field according to an embodiment. DETAILED DESCRIPTION

[0056] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0057] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail, but should be considered part of the specification when appropriate.

[0058] Applicants have found that:

[0059] The prior art has the following methods:

[0060] (1) Data-driven method

[0061] Data-driven speaker synthesis technology aims to generate animation synchronized with facial movements based on input data such as audio. Audio-driven technology achieves synchronization through a mapping relationship between audio and facial movements. Early methods are mostly based on generative adversarial networks (GANs) and use facial landmarks such as lip shapes as constraints. In recent years, three-dimensional reconstruction technology based on NeRF has been gradually applied to audio-driven synthesis, improving the quality and realism of generated results. However, data-driven methods have high computational costs and require high-performance hardware devices. The training cost is also high, and there is still room for improvement in the naturalness and scalability of generated facial animations. There are still challenges in improving scalability and naturalness of details.

[0062] (2) 2D digital human synthesis

[0063] 2D digital human synthesis has made great progress in recent years with the development of deep learning techniques, especially the application of generative adversarial networks (GAN) and diffusion models. Early GAN methods achieved facial animation synthesis through image generation models, but due to the lack of three-dimensional structure, the generated facial expressions and movements still lack realism. In contrast, 2D video generation methods based on diffusion models rely on spatial-temporal decomposition techniques and perform well in visual generation quality and temporal consistency. Representative works such as Video Diffusion Models (VDM) use U-Net architecture to process joint training of images and videos, and ImagenVideo generates high-resolution videos through cascading diffusion models. Although diffusion model-based techniques have made significant progress in generating high-fidelity videos, slow inference speed remains a major bottleneck for this type of method; in addition, GAN methods, although providing an image-to-image generation path in the early stage, lack three-dimensional support, resulting in a lack of realism in the generated results.

[0064] (3) 3D digital human synthesis (implicit representation)

[0065] Implicit Neural Radiance Fields (NeRF) technology has rapidly become a core tool in the field of three-dimensional reconstruction and synthesis since its introduction in 2020. AD-NeRF first proposed an audio data-driven three-dimensional radiance field training method to generate high-quality dynamic facial animation. Subsequent research such as RAD-NeRF and DFRF further encodes spatial coordinates and audio features through small networks, enabling real-time synthesis and significantly reducing training costs. Although NeRF methods have achieved excellent performance in generating highly realistic three-dimensional digital humans, the complexity and slow rendering speed caused by their implicit representation make it difficult to meet large-scale application requirements. How to improve rendering efficiency and enhance model versatility remains an important direction for future research.

[0066] In view of this, with reference to Figure 1 , the present application provides a neural radiance field-based audio-driven three-dimensional digital human generation method, comprising the following steps:

[0067] S1, constructing a pose-preservation prior generation model, extracting latent representations of audio features and facial pose feature points through a convolutional neural network, and learning cross-modal interaction relationships between audio features and facial pose feature points using self-attention mechanisms and cross-attention mechanisms to generate facial pose prior information synchronized with input audio;

[0068] S2, constructing a neural radiance field rendering model, using three-plane hashing encoding technology for efficient sampling of three-dimensional space, and dynamically fusing facial pose feature points and audio features through attention mechanisms;

[0069] S3, input the pose-preservation prior generated in step S1 into the neural radiance field rendering model constructed in step S2, and introduce light ray cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with the input audio.

[0070] In this embodiment, a two-stage framework is proposed to combine the pose-preservation prior driven feature and the NeRF rendering to synthesize a high-fidelity audio-driven portrait sequence with better rendering quality:

[0071] Specifically, the facial pose feature points and the audio are used to compose the pose-preservation prior for synthesizing the portrait. The facial pose feature points are composed of the unified lip-sync feature points and the pose feature points. Since the facial pose feature points have a unique matching relationship with the facial motion, we add them to the training to increase the driving information and the stability of the synthesized sequence, and also improve the generalization problem caused by training with only a small amount of audio. In order to obtain feature points corresponding to any audio, we implement a Transformer encoder based on cross-attention mechanism, which fuses the features of the pose feature point embedding and the audio embedding of the two modalities, and finally decodes and synthesizes the lip-sync feature points.

[0072] In the radiance field network part, in order to improve the sampling accuracy and convergence speed, we use three-plane hash coding sampling to obtain spatial information. Using audio and facial landmarks as dynamic input, we also use attention mechanism to fuse audio and facial landmark driving feature information, and finally connect it with spatial information as training data input into the portrait radiance field.

[0073] As an implementation, in S1, the training process of the pose-preservation prior generation model includes:

[0074] The audio feature is extracted by two-dimensional convolution operation to obtain the audio feature representation a t ; For the pose feature points, since they are essentially a one-dimensional point set data, the latent representation of the facial pose feature points is extracted by one-dimensional convolution to obtain the feature point feature p t ;

[0075] The relationship between the audio feature and the facial pose feature point feature is learned by using the Transformer module to fuse the information of the two modalities through cross-attention mechanism;

[0076] In the decoding stage, the facial pose feature points are taken as queries, and the results output by the encoder are taken as keys and values for cross-attention fusion to generate the predicted value of the lip feature points;

[0077] The first loss function including the prediction lip feature point error and the timing consistency loss is introduced to optimize the training process of the facial feature point generator.

[0078] In this embodiment, by introducing a face feature point generator, the audio-driven pose prior information is used to guide the NeRF rendering, which enhances the synchronization control of the model on the facial action, making the generated facial motion more stable. This technology effectively avoids the inconsistency between audio and facial action, and improves the temporal consistency and naturalness of the synthesized face.

[0079] Through the Transformer module and cross-attention mechanism, the deep fusion of audio features and facial pose features is realized, and the accuracy of facial pose prior information is improved. The introduction of lip feature point prediction error and timing consistency loss optimizes the training process of the facial feature point generator, making the generated facial pose prior more synchronized and coherent with the audio.

[0080] Preferably, in S1, the Transformer module is used to learn the relationship between audio features and facial pose feature points. The audio features are represented as a t , and the facial pose feature points are represented as p t . a t and p t are encoded to obtain the initial audio Token{Z a ∈R T} and pose feature point Token{Z p ∈R T}, and the encoding process is completed by position encoding γ(·):

[0081] Z a ,Z p =γ(a t ),γ(p t )

[0082] Preferably, in S1, in order to better capture the information interaction between audio and feature points, this embodiment introduces a cross-attention mechanism, which overcomes the limitations of simple feature splicing and self-attention mechanism in learning multi-modal information. Specifically, the audio features are taken as Key (K) and Value (V), because the audio provides the basic conditions for generating facial pose feature points; the facial pose feature points are taken as Query (Q), which guides the generation or adjustment of facial feature point changes, and according to the audio input to represent the changes of the lips, so as to help the model understand how the lips should change according to the specific audio input:

[0083] CrossAttn(Z p ,Z a )=Attention(Q,K,V)

[0084] Additionally, in the decoding stage, the embodiment takes the pose feature points as queries, and the CrossAttn results output by the encoder as keys and values, and performs cross-attention fusion again to obtain the predicted values of the lip feature points

[0085]

[0086] To ensure the accuracy of the generated feature points, the embodiment adopts a first loss function landmark The loss function includes the L1 error of the predicted lip feature points and the L2 loss set to ensure the timing consistency.

[0087] Specifically, in S1, to better infer the facial feature points synchronized with the audio, the embodiment introduces a new loss value calculation for quantifying the synchronization of the facial feature points with the audio; according to the principles of the CLIP technology and the Wav2lip algorithm, a loss calculation based on 2D landmark and audio synchronization, i.e., L_sync, is proposed. In the implementation of introducing the first loss function, it includes:

[0088] Custom positive and negative samples are defined, and two convolution modules are used to downsample the facial pose feature points and the audio features respectively to obtain two feature vectors of the same dimension;

[0089] The error is calculated using the cross-entropy loss Cross-Entropy:

[0090]

[0091] Wherein, represents the real lip feature points.

[0092] The first loss function combines the generation accuracy and timing consistency, ensuring the smoothness and accuracy of the lip movements. Through this technical solution, the invention can realize accurate audio-driven lip synchronization generation and improve the timing consistency and naturalness of the synthesized portrait.

[0093] In the embodiment, the encoding method of the audio features and the facial pose feature points, the calculation process of the cross-attention mechanism, and the calculation method of the first loss function are further limited. Through position encoding and cross-attention mechanism, the timing information and interaction of the audio features and the facial pose feature points are fully utilized, and the accuracy of the pose prior information is improved. The introduction of custom positive and negative samples and cross-entropy loss effectively optimizes the prediction accuracy of the lip feature points, making the generated digital human lip movements more consistent with the input audio.

[0094] As an implementation, in S2, when constructing a neural radiation field rendering model, a NeRF-based renderer module (NERF-BASED RENDERER) is constructed in this embodiment to generate realistic speaker portrait images. This module combines the pose-preservation prior information generated from the input audio and performs portrait synthesis through a three-dimensional neural radiation field model, solving the problem of lack of three-dimensional realism in 2D portraits in the prior art, specifically:

[0095] In the NeRF rendering process, the face pose-preservation prior information and the input image sequence are jointly used as the input of NeRF, where the image sequence is parameterized by three-dimensional coordinates X and viewing direction d;

[0096] To improve the sampling efficiency of spatial information and reduce the impact of hash collision on training, a three-plane hash encoding technique is used to obtain spatial information, and the implicit function The expression is as follows:

[0097]

[0098] Where c and σ represent the color and density in the radiation field, respectively, and H is the result of three-plane hash encoding.

[0099] The audio feature and the face pose feature point information are extracted from multiple consecutive frames, and the consistency of the generated face time is determined.

[0100] By using the three-plane hash encoding technique, the sampling efficiency of three-dimensional spatial information is significantly improved, and the hash collision problem that may occur in traditional NeRF is avoided. Especially in handling dynamic scenes, the convergence speed and rendering efficiency of the model are improved, and the training and inference process is optimized

[0101] Preferably, in S2, to deal with the hash collision problem in dynamic scenes and capture more delicate dynamic information, it further includes:

[0102] The three-dimensional coordinates X of each frame are projected in three directions to obtain the spatial information H(X) of the frame, and the specific expression is:

[0103]

[0104] Where, represents the projection in the plane direction.

[0105] Preferably, to better extract the multi-modal information of audio and face feature points, an attention mechanism is used to fuse the audio and face feature points, specifically:

[0106] Audio feature and feature point feature The cross-attention calculation is performed by the following formula:

[0107]

[0108] The fused features are used to guide the volume rendering process of NeRF, and the rendering formula is:

[0109]

[0110] Where t0 and t1 are the near and far boundaries of the ray r, and T(t) is the cumulative transmittance of the ray from t0 to t, expressed as follows:

[0111]

[0112] Through the cross-modal attention mechanism, the audio and facial feature point features are deeply fused, improving the model's understanding and generation ability of multi-modal data, and ensuring that the synthesized portrait can maintain high-quality dynamic performance under various input conditions.

[0113] Preferably, the neural radiance field rendering model reduces the error between the rendered image and the real image by using a second loss function, and the second loss function is:

[0114]

[0115] Where, is the set of camera rays, C gt is the color of the real image.

[0116] In this embodiment, the construction process of the neural radiance field rendering model is further limited, including the input of pose prior information and image sequence, three-plane hash coding, cross-attention fusion of audio features and pose feature points, and volume rendering of neural radiance field. At the same time, a second loss function is introduced to optimize the training process of the neural radiance field rendering model. Through the three-plane hash coding technology, a compact spatial information representation is obtained, reducing the computational complexity of spatial sampling and improving the rendering efficiency. The cross-attention mechanism realizes the dynamic fusion of audio features and facial pose feature points, improving the spatio-temporal consistency in the rendering process. The introduction of ray cumulative transmittance calculation and the second loss function optimizes the rendering quality of the neural radiance field, generating a high-fidelity, detailed three-dimensional digital human.

[0117] In some possible embodiments, in the pose-preserving prior generation model, an expression intensity control parameter is introduced, and by adjusting the parameter, the amplitude of expression changes in the generated facial pose prior information can be controlled. At the same time, in the neural radiance field rendering model, the fusion weight of the facial feature points is dynamically adjusted according to the expression intensity parameter, so that the facial expression of the generated digital person can be adjusted between exaggeration and subtlety according to the requirement, specifically:

[0118] 1. An expression intensity control parameter is added to the input of the pose-preserving prior generation model, which is fused with the audio features and facial pose feature points through a self-attention mechanism to generate facial pose prior information matching the expression intensity.

[0119] 2. In the neural radiance field rendering model, the fusion weight of the facial feature points is set according to the expression intensity parameter. When the expression intensity parameter is large, the fusion weight of the facial feature points is increased to generate exaggerated expression changes; when the expression intensity parameter is small, the fusion weight of the facial feature points is reduced to generate subtle expression changes.

[0120] 3. An expression intensity loss function is designed to optimize the learning process of the expression intensity control parameter by calculating the difference between the generated expression and the expected expression intensity, improving the accuracy of expression control.

[0121] In this embodiment, by introducing the expression intensity control mechanism, the generated digital person can realize the regulation of facial expression from exaggeration to subtlety according to the application requirement, improving the expressiveness and applicability of the digital person and meeting the requirement of fine control of facial expression in different scenarios.

[0122] In some possible embodiments, the method can be applied to whole-body level audio-driven digital person generation. On the basis of the original audio-driven face generation, human body posture information is introduced as an additional input condition. By extending the pose-preserving prior generation model and the neural radiance field rendering model, the human body posture features are fused with the audio features and facial features to generate whole-body movements and facial expressions synchronized with the audio, realizing whole-body level audio-driven digital person generation, specifically:

[0123] 1. A human body posture estimation algorithm is used to extract human body key point information from the input video to obtain human body posture features.

[0124] 2. The pose-preserving prior generation model is extended to fuse the human body posture features with the audio features and facial pose feature points, and through the self-attention mechanism and cross-attention mechanism, the cross-modal interaction relationship among the three is learned to generate prior information of the whole-body posture and facial expression.

[0125] 3. The extended neural radiance field rendering model is introduced, the human body posture feature is introduced as an additional condition input, the three-plane hash coding is used for sampling the whole body three-dimensional space, and the attention mechanism is used for dynamically fusing the human body posture, the facial feature points and the audio features.

[0126] 4. The whole body posture consistency loss function is introduced, the difference between the generated whole body posture and the real posture is calculated, and the audio-driven digital human generation process at the whole body level is optimized.

[0127] In the embodiment, the audio-driven digital human generation is extended from the face to the whole body by fusing the human body posture information, so that the generated digital human can produce whole body actions and facial expressions according to the audio information, the realism and interactivity of the digital human are improved, and the application scenarios of the audio-driven digital human generation are expanded.

[0128] In a second aspect, with reference to Figure 2 The embodiment provides an audio-driven three-dimensional digital human generation system based on a neural radiance field, which applies the generation method of each of the above embodiments, and comprises:

[0129] A first processing module is configured to construct a posture preservation prior generation model, extract latent representations of audio features and facial posture feature points through a convolutional neural network, and learn cross-modal interaction relationships between the audio features and the facial posture feature points by using a self-attention mechanism and a cross-attention mechanism to generate facial posture prior information synchronized with input audio;

[0130] A second processing module is configured to construct a neural radiance field rendering model, perform efficient sampling on a three-dimensional space by using a three-plane hash coding technology, and dynamically fuse facial posture feature points and audio features by using an attention mechanism.

[0131] A three-dimensional generation module is configured to input the generated posture preservation prior information into the constructed neural radiance field rendering model, introduce light cumulative transmittance calculation, and generate a high-fidelity three-dimensional digital human synchronized with input audio.

[0132] The first processing module, the second processing module and the three-dimensional generation module cooperate to realize end-to-end generation from input audio to high-fidelity three-dimensional digital human. The system integrates key technologies such as audio driving, facial posture prior and neural radiance field rendering, generates realistic digital humans with coherent actions, rich expressions and lip synchronization, and provides high-quality digital human assets for virtual reality, augmented reality, mixed reality and other applications.

[0133] The following describes an embodiment:

[0134] Embodiment 1

[0135] To compare the results generated by different algorithms, this embodiment demonstrates the digital human inferred by each algorithm under the self-driving of 3 key frames. The LA-NeRF column is the generation result of this embodiment, which is more accurate and clear compared to other methods, and can achieve more accurate reconstruction results in lip reconstruction. This phenomenon also proves that there is a many-to-many phenomenon between audio and mouth movement, which will cause the mouth reconstruction to be blurred.

[0136] In addition, when the mouth has a large motion, such as pouting or opening the mouth, the method of this embodiment can more robustly synthesize the mouth movement and reconstruct more details. Since the pose is fixed, the real driver of the digital human is the mouth feature point in the landmark.

[0137] Furthermore, in this embodiment, data-driven synthesis is performed on the same identity test set, and the quality of portrait reconstruction is compared. The following table is the quantitative evaluation data of different methods:

[0138] Table 1. Quantitative evaluation data of different methods

[0139]

[0140] Among them, the LA-NeRF column is the evaluation data of this embodiment, and the peak signal-to-noise ratio (PSNR), image feature distance (FID), boundary distance (LMD), and learning perceptual image block similarity (LPIPS) are used to measure the similarity between the generated image and the real image; the lip synchronization is evaluated using the synchronous network confidence score (Sync).

[0141] The evaluation results are shown in Table 1, compared with the best performing method in recent years, which has:

[0142] (1) The method of this embodiment performs excellent quality in most indicators.

[0143] (2) Specifically, from the perspective of video reconstruction, the rendering effect of this embodiment maintains the best visual quality. It is observed that the wav2lip and makeittalk algorithms have lower fidelity for image reconstruction, and perform poorly in FID and LPIPS. ER-NeRF reduces the impact of high-dimensional hash conflicts in sampling point coding, has better face reconstruction results, and performs better in LMD.

[0144] In summary, compared with the prior art, the above embodiment provides a neural radiance field-based audio-driven three-dimensional digital human generation method and system, which at least makes the following technical optimizations to overcome the defects of the prior art:

[0145] 1. Improve sampling precision and convergence speed: In the high-precision three-dimensional scene reconstruction of traditional NeRF method, the problems of low sampling efficiency and slow convergence speed are faced. We introduce three-plane hash coding sampling, which greatly improves the extraction efficiency of spatial information, and combines facial feature points and audio data to realize a more efficient sampling process, shorten the training time and improve the convergence speed of the model;

[0146] 2. Optimize dynamic feature fusion: Although NeRF performs well in static scene generation, it has the problem of unstable action generation when dealing with dynamic characters. The invention uses a cross-modal attention mechanism to dynamically fuse facial feature points and audio information, making the generated actions more smooth, improving the stability of facial expressions and lip synchronization, and reducing the blur and flicker phenomenon in dynamic scenes;

[0147] 3. Enhance the generalization ability of the model: The generalization ability of existing NeRF models is weak, especially in the case of insufficient training data, which can easily lead to a decline in generation quality. The invention introduces a pose preservation prior module based on audio and facial feature points, which improves the stability and generalization ability of the model under different input conditions through joint training of multi-modal data, and reduces the impact of data scarcity on generation quality;

[0148] Therefore, the method realizes an end-to-end audio-driven digital human generation process through the organic combination of the pose preservation prior generation and the neural radiance field rendering stages, simplifies the traditional process, and improves the generation efficiency and quality. Traditional digital human generation methods usually need to capture and model facial expressions, body movements and sounds separately, and then synchronize and fuse them, which is a complex process and is prone to misalignment and unnaturalness. The invention automatically completes the mapping from audio to three-dimensional digital human through end-to-end neural network learning, avoiding complex manual processing and synchronization operations. Therefore, the invention provides a simple and efficient, fully automated audio-driven digital human generation scheme, which greatly reduces the threshold and cost of digital human production, and promotes the popularization and application of digital human technology.

[0149] The above embodiments are only preferred embodiments of the present application, and cannot be used to limit the scope of protection of the present application. Any non-essential changes and substitutions made by those skilled in the art based on the present application are within the scope of the present application.

Claims

1. A method for generating a three-dimensional digital human driven by audio based on neural radiance fields, characterized in that, The method comprises the following steps: S1, constructing a pose maintaining prior generation model, extracting audio features and latent representations of facial pose feature points through a convolutional neural network, and learning cross-modal interaction relationships between the audio features and the facial pose feature points by using a self-attention mechanism and a cross-attention mechanism to generate facial pose prior information synchronized with input audio; S2, constructing a neural radiance field rendering model, using three-plane hash coding technology to efficiently sample a three-dimensional space, and dynamically fusing facial pose feature points and audio features through an attention mechanism; S3, inputting the pose maintaining prior information generated in step S1 into the neural radiance field rendering model constructed in step S2, and introducing light cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with input audio; In S1, the training process of the pose maintaining prior generation model comprises: extracting audio features through two-dimensional convolution operation, and extracting latent representations of facial pose feature points through one-dimensional convolution; learning the relationship between the audio features and the facial pose feature points by using a Transformer module, and fusing the information of the two modalities through a cross-attention mechanism; in the decoding stage, the facial pose feature points are taken as queries, the results output by the encoder are taken as keys and values, cross-attention fusion is performed, and a prediction value of the lip feature points is generated; a first loss function comprising a prediction lip feature point error and a timing consistency loss is introduced to optimize the training process of the facial feature point generator; In S2, when constructing the neural radiance field rendering model, it comprises: In the NeRF rendering process, the face pose prior information is taken as an input of the NeRF together with the input image sequence, wherein the image sequence is parameterized by three-dimensional coordinates and a view direction . Three-plane hash coding technique is used to obtain spatial information, implicit function The expression is as follows: wherein, and respectively denote the color and density in the radiance field, is the result of the tri-plane hashing encoding; extracting information of audio features and facial pose feature points from multiple consecutive frames, and judging the consistency of the generated facial time.

2. A system for use in the method of claim 1, characterized in that It comprises: a first processing module for constructing a pose maintaining prior generation model, extracting audio features and latent representations of facial pose feature points through a convolutional neural network, and learning cross-modal interaction relationships between the audio features and the facial pose feature points by using a self-attention mechanism and a cross-attention mechanism to generate facial pose prior information synchronized with input audio; a second processing module for constructing a neural radiance field rendering model, using three-plane hash coding technology to efficiently sample a three-dimensional space, and dynamically fusing facial pose feature points and audio features through an attention mechanism; a three-dimensional generation module for inputting the generated pose maintaining prior information into the constructed neural radiance field rendering model, and introducing light cumulative transmittance calculation to generate a high-fidelity three-dimensional digital human synchronized with input audio.

Images