Audio data processing method, neural network training method, and related apparatus

Abstract

The present disclosure relates to an audio data processing method, a neural network training method, an audio data processing apparatus, an electronic device, and a computer readable storage medium. The audio data processing method comprises: determining facial prior information of a virtual object on the basis of reference video data; determining a target audio feature on the basis of target audio data; determining target expression sequence information on the basis of the facial prior information and the target audio feature; and on the basis of the target expression sequence information and a reference frame in the reference video data, determining target video data corresponding to the target audio data. According to the present disclosure, the synchronization precision between audio content and a facial component (e.g., the mouth) action of a virtual object can be significantly improved, the sense of reality of the mouth action is enhanced, an expression difference between individuals is captured and reproduced more accurately, and the operation efficiency and stability of the whole system are improved while the system performance is improved.

Description

Methods for processing audio data, methods for training neural networks, and related devices

[0001] This application claims priority to Chinese Patent Application No. 2024102321570, filed on February 29, 2024, entitled “Method and Apparatus for Processing Audio Data”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of artificial intelligence services, and more specifically to the processing of audio data. Background Technology

[0003] In recent years, technologies for generating video data from audio data have developed rapidly, especially audio-driven face generation technology, which has broad application prospects. This technology can analyze information such as emotion and tone of voice from the input audio signal and control the expressions and movements of virtual characters in real time to achieve highly synchronized performance. In the entertainment industry, game development, human-computer interaction interfaces, and virtual reality conferencing, this technology has demonstrated wide application value, and with the continuous innovation of deep learning and neural network technologies, its expressiveness and realism are constantly improving.

[0004] Despite this, current audio-driven virtual face generation technology still faces a series of technical challenges. The primary challenge is the difficulty in achieving precise synchronization between audio data and mouth movements, resulting in facial animations that often fail to perfectly match the spoken content. Secondly, current technology cannot yet achieve ideal realistic visual effects when processing complex facial micro-details and textures, leading to low realism in virtual faces. Furthermore, ensuring that different virtual objects exhibit facial expressions consistent with their unique personalities, thus realistically reflecting individual differences, is also a pressing issue. Finally, improving system efficiency and overall stability to adapt to the needs of more real-time application scenarios is also an important direction for future improvements to this technology.

[0005] Therefore, audio-driven virtual face generation technology still needs further optimization and improvement. Summary of the Invention

[0006] This disclosure provides a method for processing audio data, a method for training a neural network, an apparatus for processing audio data, an electronic device, and a computer-readable storage medium.

[0007] This disclosure provides a method for processing audio data, the method comprising: determining facial prior information of a virtual object based on reference video data; determining target audio features based on target audio data; determining target facial expression sequence information based on the facial prior information and the target audio features; and determining target video data corresponding to the target audio data based on the target facial expression sequence information and reference frames in the reference video data.

[0008] This disclosure provides a method for training a neural network, wherein the neural network includes an audio-to-expression converter and an expression-to-face renderer. The method includes: determining sample facial prior information of a virtual object based on sample video data; determining sample target audio features based on sample audio data corresponding to the sample video data; determining predicted values ​​of target sample expression sequence information using the audio-to-expression converter based on the sample facial prior information and the sample target audio features; training the audio-to-expression converter using a first loss function based on the predicted values ​​of the target sample expression sequence information; determining sample reference frames and sample source video frames based on the sample video data; and training the expression-to-face renderer using a second loss function based on the sample facial prior information, the sample reference frames, and the sample source video frames.

[0009] This disclosure provides an apparatus for processing audio data, the apparatus comprising: a first preprocessing module configured to: determine facial prior information of a virtual object based on reference video data; a second preprocessing module configured to: determine target audio features based on target audio data; an audio-to-expression converter module configured to: determine target expression sequence information based on the facial prior information and the target audio features; and an expression-to-face renderer configured to: determine target video data corresponding to the target audio data based on the target expression sequence information and reference frames in the reference video data.

[0010] In another aspect, embodiments of this application provide a computer device, including:

[0011] Processor, communication interface, memory, and communication bus;

[0012] The processor, the communication interface, and the memory communicate with each other via the communication bus; the communication interface is the interface of the communication module.

[0013] The memory is used to store computer programs and transfer the computer programs to the processor; the processor is used to invoke the computer programs in the memory to execute the methods described above.

[0014] In another aspect, embodiments of this application provide a storage medium for storing a computer program for executing the methods described above.

[0015] In another aspect, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described above.

[0016] This embodiment of the disclosure accurately identifies the facial features of a virtual object by determining its prior facial information in reference video data. This allows for the determination of a target expression sequence that conforms to and naturally integrates with the virtual object's facial features based on the prior facial information and target audio features. When using the target expression sequence information and reference frames to determine the target video data, the synchronization accuracy between the audio content in the target video data and the facial components (e.g., mouth) movements of the virtual object on the screen is significantly improved, enhancing the realism of mouth movements. It also accurately reflects the identity characteristics of the virtual object, more accurately capturing and reproducing individual differences in expression. Furthermore, while improving system performance, it ensures overall operational efficiency and stability, making it particularly suitable for real-time video generation. Attached Figure Description

[0017] Figure 1 is an example schematic diagram illustrating a scenario according to an embodiment of the present disclosure;

[0018] Figure 2 shows a schematic diagram according to an embodiment of the present disclosure;

[0019] Figure 3 shows a flowchart of a method for processing audio data according to an embodiment of the present disclosure;

[0020] Figure 4A shows a schematic diagram of an improved neural network model according to an embodiment of the present disclosure;

[0021] Figure 4B illustrates a schematic diagram of the audio-to-facial expression conversion stage according to an embodiment of the present disclosure;

[0022] Figure 4C shows a schematic diagram of an audio-to-facial expression converter according to an embodiment of the present disclosure;

[0023] Figure 4D shows a schematic diagram of an expression-to-face renderer according to an embodiment of the present disclosure;

[0024] Figure 5A shows a flowchart of a method for training a neural network according to an embodiment of the present disclosure;

[0025] Figure 5B shows a schematic diagram of training an audio-to-expression converter according to an embodiment of the present disclosure;

[0026] Figure 6 shows a comparison of test results according to embodiments of the present disclosure;

[0027] Figure 7 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure;

[0028] Figure 8 shows a schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure;

[0029] Figure 9 shows a schematic diagram of a storage medium according to an embodiment of the present disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0031] In this specification and accompanying drawings, operations and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these operations and elements are omitted. Furthermore, in the description of this disclosure, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0032] The models that can be used in embodiments of this disclosure can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs, where neurons are arranged in different layers. Generally, a neural network model includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received from the input layer into a representation useful for generating the output in the output layer. Nodes are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at nodes in the input layer of the neural network is propagated to nodes in the output layer via any of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and this disclosure does not limit this.

[0033] First, with reference to FIG1, an application scenario of the method for processing audio data according to an embodiment of the present disclosure and a corresponding apparatus will be described. FIG1 shows a schematic diagram of an application scenario 100 according to an embodiment of the present disclosure, wherein a server 110 and a plurality of terminals 120 are schematically shown.

[0034] The neural network model of this disclosure can be integrated into various electronic devices, such as any electronic device among the server 110 and multiple terminals 120 in Figure 1. For example, the neural network model can be integrated into terminal 120. Terminal 120 can be a mobile phone, tablet computer, laptop computer, desktop computer, personal computer (PC), smart speaker, or smartwatch, but is not limited to these. Alternatively, the neural network model can also be integrated into server 110. Server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations.

[0035] It is understood that the computer device used for inference using the neural network model of this disclosure can be a terminal device, a server, or a system composed of a terminal device and a server. The method for processing audio data according to this disclosure can be executed on a terminal, on a server, or jointly by a terminal device and a server.

[0036] It is worth noting that both the terminal 110 and server 120 according to the embodiments of this disclosure adhere to data protection principles, respect users' data rights, and safeguard users' data security and privacy. The terminal 110 and server 120 according to the embodiments of this disclosure will clearly inform users of the purpose, method, and scope of collecting, using, storing, transmitting, and deleting user data, and obtain users' consent. The terminal 110 and server 120 according to the embodiments of this disclosure will take reasonable technical and management measures to prevent user data from being leaked, tampered with, damaged, or lost. The providers of the terminal 110 and server 120 according to the embodiments of this disclosure will regularly review and update user data, and promptly delete expired or useless data. Furthermore, cloud service providers using the embodiments of this disclosure respect users' rights to data access, correction, deletion, withdrawal of consent, complaints, and claims, and provide convenient channels and procedures to enable users to effectively exercise these rights.

[0037] Furthermore, the process of data analysis using artificial intelligence technology in terminal 120 or server 110 is conducted based on the principles of legality, rationality, and transparency. The data collected and processed by the artificial intelligence model according to embodiments of this disclosure is relevant, necessary, and appropriate for the predictive purpose, and does not contain any personally identifiable or sensitive information. The neural network model according to embodiments of this disclosure employs appropriate techniques and organizational measures to protect the security and integrity of data, preventing unauthorized access, use, or disclosure.

[0038] The artificial intelligence-based neural network model according to embodiments of this disclosure will comply with relevant data protection regulations and ethical principles. This neural network model is trained on a large amount of anonymized and de-identified data, and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable, and will not cause any misleading or discriminatory behavior. The artificial intelligence model is designed solely to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. Furthermore, the neural network model will be regularly reviewed and updated to adapt to changes in the data environment and legal regulations.

[0039] Traditionally, audio-driven virtual face generation methods can be broadly categorized into two types: person-specific methods and person-generic methods. Person-specific methods use neural network models that require training or fine-tuning on specific individuals before prediction, while person-generic methods can directly generate speaking facial videos of unseen virtual characters.

[0040] To address the audio-visual synchronization issues in the two methods mentioned above, some solutions introduce mouth synchronization discriminators or lip-reading networks to enhance the intelligibility of the mouth region. However, such solutions still cannot solve the audio-visual synchronization problem. To improve video quality, some solutions employ diffusion models or adversarial generative models to attempt to improve image fidelity. However, such solutions struggle to achieve a balance between efficiency, visual quality, and mouth movement accuracy.

[0041] Furthermore, when constructing the correspondence between audio data and facial expressions, some solutions employ fusion modeling methods to capture mouth movement features more efficiently and accurately. Specifically, they can predict region labels of the target face or extract key point information in an unsupervised environment based on audio data. Although facial region labels and key points are relatively easy to obtain, due to the inherent sparsity of this information, these solutions cannot fully express complex facial movements such as pursing or closing the lips.

[0042] This disclosure provides an architectural improvement to the neural network model. Compared to the single-module structure of traditional neural network models suitable for audio-driven virtual face generation, the improved neural network model consists of two decoupled modules: an audio-to-expression transformer and an expression-to-face renderer.

[0043] Therefore, the improved neural network model can process the input data in three separate stages to achieve higher performance. These three stages are: preprocessing, audio-to-facial expression conversion, and facial expression-to-video frame reconstruction.

[0044] Specifically, firstly, in the preprocessing stage, the preprocessing module can perform necessary feature extraction on the reference video data and target audio data to obtain the facial prior information and target audio features of the virtual object. Next, in the audio-to-expression conversion stage, the audio-to-expression converter maps the received facial prior information and target audio features into corresponding target expression sequence information, particularly focusing on subtle changes in mouth movements. Finally, in the expression-to-video frame reconstruction stage, the expression-to-face renderer, based on the target expression sequence information, reference frames in the reference video data, and the reference head pose feature vector corresponding to the reference frames, accurately generates target video data corresponding to the target audio data. This target video data is precisely synchronized with the target audio data and possesses highly realistic mouth movements and facial expression details.

[0045] Compared to the traditional single-stage approach of neural network models, which directly inputs audio and video data to generate updated video footage, the improved multi-stage neural network model according to embodiments of this disclosure can significantly improve the synchronization accuracy between audio content and facial components (e.g., mouth) movements of virtual objects on the screen, enhance the realism of mouth movements, more accurately capture and reproduce facial expression differences between individuals, and ensure overall operational efficiency and stability while improving system performance.

[0046] This disclosure provides a method for processing audio data to utilize the improved neural network model for inference. The method includes: determining facial prior information of a virtual object based on reference video data; determining target audio features based on target audio data; determining target facial expression sequence information based on the facial prior information and the target audio features; and determining target video data corresponding to the target audio data based on the target facial expression sequence information, a reference frame in the reference video data, and a reference head pose feature vector corresponding to the reference frame.

[0047] By utilizing an improved neural network for inference, the synchronization between audio content and the facial movements of virtual objects on screen is significantly enhanced by first determining the target facial expression sequence and then predicting video frames based on it. This method not only enhances the realism of facial movements but also accurately reflects the identity characteristics of virtual objects. While improving performance, it also ensures the efficiency and stability of the neural network, making it particularly suitable for real-time video generation.

[0048] Similarly, for the improved neural network model, this disclosure also provides a method for training the neural network to train the improved neural network model. The neural network includes an audio-to-expression converter and an expression-to-face renderer. The method includes: training the audio-to-expression converter based on sample video data and corresponding sample audio data; and training the expression-to-face renderer based on sample facial prior information, sample reference frames, and sample source video frames. The audio-to-expression converter and the expression-to-face renderer can be trained separately and independently, and their training processes do not interfere with each other.

[0049] In the training process of the improved neural network model, an improved first loss function and a second loss function are optionally introduced. These two loss functions optimize for different characteristics, improving the performance of the trained neural network by adjusting its parameters. The first and second loss functions enable the neural network to learn and adapt to the features of the input data more effectively and optimize from multiple aspects during training. This comprehensive design of loss functions, considering different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby improving its performance in practical applications.

[0050] The method for processing audio data, the method for training a neural network, and the apparatus according to embodiments of the present disclosure are described below with reference to Figures 2 to 9.

[0051] Figure 2 shows a schematic diagram according to an embodiment of the present disclosure.

[0052] As shown in Figure 2, a specific application scenario for audio-driven virtual face generation technology is as follows: There is already a video clip (hereinafter referred to as reference video data), which corresponds to an audio clip. The video content shows the facial expressions of a virtual object speaking to the camera, especially the details of its mouth movements. In this video clip, the virtual object says a sentence, such as "The weather is so nice today." To produce more videos of this virtual object more quickly, it is necessary to synchronize the lip movements of the virtual object in this video to another different audio content (hereinafter referred to as target audio data), such as "I love you." That is, through technical means, the virtual character who was originally saying "The weather is so nice today" generates target video data based on another audio data. In this target video data, the lip movements of the virtual object can accurately match the shape and rhythm that should be present when saying "I love you."

[0053] As shown in Figure 2, the neural network model according to the embodiments of this disclosure mainly consists of two parts: an audio-to-expression converter and an expression-to-face renderer.

[0054] During preprocessing, embodiments of this disclosure extract facial prior information from the reference video data, including but not limited to: reference facial shape information, reference expression information, and reference head pose information. Simultaneously, audio features are extracted from the target audio data during processing. Optionally, a 3D Morphable Face Model (3DMM) can be used as a 3D morphological model to extract more accurate facial shape, pose, and expression information (such as the reference facial shape information, reference expression information, and reference head pose information mentioned later) from the reference video data, thereby achieving accurate mapping from video to expression sequences. Compared to traditional methods that require marking key points in facial regions, the 3D morphological model according to embodiments of this disclosure helps neural networks learn the movements and changes of facial components (e.g., the mouth) more effectively and accurately.

[0055] In the audio-to-facial conversion stage, the facial prior information extracted in the preprocessing is input into the audio-to-facial converter to generate target facial expression sequence information, so as to better preserve and reflect the identity characteristics and subtle changes in emotional expression of the virtual human body.

[0056] In the expression-to-video frame reconstruction stage, the target expression sequence information is used as input to the expression-to-face renderer, enabling the renderer to accurately generate motion video footage of target facial components (e.g., the mouth, eyes, ears, eyebrows, etc.) using an improved facial identity alignment module. Optionally, the expression-to-face renderer can use a facial individual alignment (FIA) module, which requires only one reference video frame to obtain a target video frame that matches the facial characteristics of the virtual character during pronunciation. Compared to traditional methods, the expression-to-face renderer according to embodiments of this disclosure improves the personalized expression of the target video frame, reduces computational costs, and helps to improve inference speed.

[0057] The embodiments of this disclosure can generate detailed facial components (used to represent virtual organs) in real time without relying on complex and inefficient computing modules, significantly improving the detail of video images and the efficiency of video data generation. Although this disclosure mainly uses the mouth as an example, those skilled in the art should understand that the solution of this disclosure can be applied to the generation of any facial organs related to expression and emotion. Of course, this disclosure is not limited thereto.

[0058] Figure 3 shows a flowchart of a method 30 for processing audio data according to an embodiment of the present disclosure.

[0059] Method 30 can be executed at a terminal device or a server (such as terminal 120 or server 110 as shown in FIG1). Method 30 includes the following operations S301 to S304. Of course, method 30 may also include more or fewer operations, and this disclosure is not limited thereto.

[0060] Optionally, operations S301 and S302 are operations in the preprocessing stage, which can be performed by any facial information extraction model. Operation S303 can be performed in the audio information to expression information conversion stage, while operation S304 can be performed in the expression to video frame reconstruction stage.

[0061] In operation S301, the prior facial information of the virtual object is determined based on the reference video data.

[0062] Optionally, the reference video data may include virtual objects in the video content. These virtual objects are objects with faces and capable of speech, such as real people or animals, or cartoon characters, virtual 3D models, etc. The reference video data may be obtained by capturing real scenes using video capture equipment or generated by software.

[0063] The reference video data can be the input video data shown in Figure 2, in which the virtual object says, "The weather is so nice today." The video data includes multiple reference video frames, each of which is an image. The image shows the head of the virtual object.

[0064] The facial prior information of a virtual object refers to accurate facial information used to identify the face of the virtual object and to represent its facial features. Optionally, the facial prior information of a virtual object includes one or more of the following: reference facial shape information, reference expression information, and reference head pose information. Of course, this disclosure is not limited thereto.

[0065] For example, reference facial shape information can be information used to characterize the facial shape features of a virtual object. Optionally, the reference facial shape information can be extracted from any reference video frame and may include the texture and 3D shape information of the virtual object's face. Optionally, based on the reference facial shape information, a 3D facial portrait of the virtual object without expression can be obtained by directly performing an affine transformation on a general 3D standard virtual object. A 3D standard virtual object, also known as a white model virtual object, is a standardized model used to simulate human facial expressions and movements. These models typically have a uniform three-dimensional shape and structure to represent the basic appearance of a virtual object. Of course, this disclosure is not limited to this.

[0066] For example, reference facial expression information can be any information used to characterize the facial expression characteristics of a virtual object, reflecting the virtual object's facial expression characteristics under different linguistic emotional expressions. Reference facial expression information can be extracted from each reference video frame, thus obtaining the reference facial expression feature vector β corresponding to each reference video frame i. i The reference facial expression information can refer to the set of reference facial expression feature vectors corresponding to all reference video frames. Optionally, based on the reference facial expression feature vectors and reference facial shape information, a 3D facial avatar of a virtual object with a specific expression in a certain frame can be obtained by directly performing an affine transformation on a general 3D standard virtual object. Of course, this disclosure is not limited to this.

[0067] For example, reference head pose information is used to characterize arbitrary information about the pose characteristics of a virtual object's head. Reference head pose information is extracted from each reference video frame, resulting in a reference head pose feature vector corresponding to each reference video frame i. The reference head pose information can refer to the set of reference head pose feature vectors corresponding to all reference video frames. Optionally, based on the reference head pose feature vector, a general 3D standard virtual object can be directly adjusted to achieve the head pose corresponding to the reference head pose feature vector through rotation. Of course, this disclosure is not limited to this.

[0068] In operation S302, the target audio features are determined based on the target audio data.

[0069] The audio identified by the target audio data is the audio that needs to be synthesized into a virtual object, that is, the audio emitted by the virtual object in the target video data determined in subsequent step S304. Optionally, the target audio data can be the target audio data shown in Figure 2, where the audio corresponding to the input video data is "I love you". The target audio data can include multiple audio frames, and the target audio features refer to the sound-related features extracted from these audio frames. These features may include the spectral information, time-domain information, and sound intensity of the audio signal. Of course, this disclosure is not limited to this.

[0070] In operation S303, target facial expression sequence information is determined based on the facial prior information and the target audio features.

[0071] Optionally, the target facial expression sequence information can be reference facial expression information corresponding to the target audio features. The target facial expression sequence information is a set of target facial expression feature vectors arranged in temporal order, with each target facial expression feature vector corresponding to a target audio frame, used to reflect the facial expression characteristics of the virtual object when uttering specific words or phrases. This mapping relationship can be many-to-one, one-to-one, or one-to-many. For example, one target facial expression feature vector can correspond to one target audio frame, or multiple target audio frames, or even multiple target facial expression feature vectors corresponding to a single target audio frame. Of course, this disclosure is not limited to this.

[0072] Optionally, the target facial expression sequence information can be directly determined based solely on the reference facial shape information and reference expression information in the facial prior information, combined with the target audio features. Of course, this disclosure is not limited to this.

[0073] In operation S304, target video data corresponding to the target audio data is determined based on the target facial expression sequence information and the reference frames in the reference video data.

[0074] The virtual object involved in the target video data and the virtual object involved in the reference video data of S301 are the same virtual object. However, the virtual object in the target video data will emit the audio corresponding to the target audio data.

[0075] Since the target facial expression sequence information can accurately identify the facial features of the virtual object when it emits the audio corresponding to the target audio feature, and the reference frame can identify the relevant information of the virtual object, the facial features of the virtual object when it emits the audio corresponding to the target audio data can be accurately simulated by using the target facial expression sequence information and guided by the reference frame in the reference video data, thereby obtaining the target video data.

[0076] Optionally, the number of target video frames in the target video data can be matched with the number of audio frames in the target audio data. Each target video frame displays a specific facial expression of the virtual object when it speaks the corresponding audio frame. Optionally, the mouth movements (i.e., lip shape) of the virtual object when speaking the corresponding audio frame should correspond to the pronunciation of that audio frame. The consistency between the facial expressions of the virtual object presented in the target video frames and its pronunciation makes the generated video more realistic and lifelike.

[0077] Optionally, reference frame I in the reference video data r This can be any video frame from the reference video data, used to provide the basic shape and texture of the virtual object as reference information during the expression-to-video frame reconstruction stage. Specifically, in the generation process of each target video frame in the target video data, the same reference frame I can be used. r Different reference frames may also be used, but this disclosure is not limited thereto.

[0078] Optionally, in S304, a source video frame can be extracted from the reference video data; the source video frame can be masked based on the target expression sequence to obtain a first video frame, wherein in the first video frame, some facial components of the virtual object are replaced by black pixel blocks; and based on the first video frame and the reference frame, the target video data corresponding to the target audio data can be determined using the facial prior information.

[0079] This application does not limit the relationship between reference frames and source video frames. For example, the source video frame can be one or more reference frames. This application also does not limit the number of source video frames used when determining the target video data. For example, it can be one source video frame or multiple source video frames.

[0080] Optionally, masking the source video frame based on the target expression sequence to obtain the first video frame includes: determining the pixel region corresponding to a portion of the facial components of the virtual object in the source video frame based on the target expression sequence and the reference facial shape information; and modifying the pixel values ​​in the pixel region to preset values.

[0081] Optionally, determining the target video data corresponding to the target audio data based on the first video frame and the reference frame, using the facial prior information, includes: encoding the first video frame and the reference frame using a weighted encoder to obtain the encoding features of the first video frame and the encoding features of the reference frame; and decoding the encoding features of the first video frame and the encoding features of the reference frame using the facial prior information and the target expression sequence, using a decoder to obtain the target video data corresponding to the target audio data. The target video data includes a target video frame, and the target video frame includes predicted values ​​of partial facial components of the virtual object replaced by black pixel blocks.

[0082] For example, a first video frame can be generated based on the target facial expression sequence information. (wherein, the first video frame) The facial components in the video frame are masked (i.e., the facial components in that video frame are replaced with black pixel blocks), and then based on the source video frame I... s and reference frame I r Determine the target video frame The following explanation uses a mouth component mask (also known as a mouth mask) as an example. Those skilled in the art will understand that this solution can also be applied to any facial component.

[0083] Specifically, for each video frame, facial structures related to mouth movements (e.g., the lower half of the face) are masked using a mouth mask, while retaining variable facial structures unrelated to expression changes or the target audio frame. Then, reference frame I from the reference video data is used... r The method involves using a face identity alignment module to predict pixel values ​​at the mouth mask location and filling these predicted pixel values ​​into the target video frame to obtain the target video frame. This disclosure is not limited thereto.

[0084] Optionally, operation S304 also includes a facial identity alignment operation, which utilizes a facial identity alignment module with a cross-attention structure to establish an effective connection between target expression-related parameters (e.g., target expression feature vector) and the effective facial texture of a single reference frame. This structure enables the module to achieve more precise lip shape control. Specifically, the cross-attention structure allows the module to simultaneously focus on the target expression and the facial texture information extracted from a single reference frame. This attention mechanism allows the module to precisely adjust the shape and movement of the lips based on the target expression feature vector and the facial texture of the reference frame. By combining these two types of information, the module can better understand the features and structure of the face and precisely control lip movements.

[0085] The method according to the embodiments of this disclosure can improve the accuracy and credibility of the lip movements of virtual objects, and enhance the communication effect and visual expressiveness of videos. The embodiments of this disclosure can generate fine details of facial organs in real time without relying on complex and inefficient computing modules, which greatly improves the fineness of video images and the efficiency of generating video data.

[0086] Figure 4A illustrates a schematic diagram of an improved neural network model according to an embodiment of the present disclosure. Figure 4B illustrates a schematic diagram of an audio-to-facial expression conversion stage according to an embodiment of the present disclosure. Figure 4C illustrates a schematic diagram of an audio-to-facial expression converter according to an embodiment of the present disclosure. Figure 4D illustrates a schematic diagram of an facial expression to face renderer according to an embodiment of the present disclosure.

[0087] As shown in Figure 4A, in the preprocessing stage, operation S301 can be performed using a deep 3D facial reconstruction model to determine the facial prior information of the virtual object based on the reference video data. The facial prior information includes reference facial shape information (α1), reference expression information (β1,…,β1…), and reference facial expression information (β1,…,β1…). N ), and reference head position pose information (ρ1,…,ρ N Given a series of video frames including the face of a virtual object, a deep 3D face reconstruction model can extract 3D coefficients, including shape coefficients, expression coefficients, and pose coefficients. Optionally, the deep 3D face reconstruction model can utilize a hybrid loss function for weakly supervised learning, which considers both low-level and perceptual information for supervision. Simultaneously, the deep 3D face reconstruction model can also utilize complementary information from different images for multi-image face reconstruction, achieving shape aggregation. The deep 3D face reconstruction model is fast, accurate, and robust, capable of handling occlusion and large pose conditions. However, embodiments of this disclosure can also use other neural network models to extract prior information about the face of virtual objects.

[0088] Optionally, in operation S302, self-supervised speech representation learning can be performed using a BERT model of hidden units (hereinafter also referred to as the HuBert model). The target audio features (w1,…,w) are determined based on the target audio data by predicting the mask of the hidden units. l The Hubert model employs a self-supervised learning method, predicting hidden units through masks to learn high-quality speech representations. However, embodiments of this disclosure can also utilize other neural network models to determine target audio features.

[0089] In the audio-to-facial expression conversion stage, in operation S303, the audio-to-facial expression converter can utilize one reference facial shape information (α1) and N reference facial expression information (β1,…,β1) from the 3D coefficients. N ), combined with l target audio features (w1,…,w l The target facial expression sequence information is obtained, which includes T target facial expression features. N, l, and T can be the same or different values, and this disclosure is not limited thereto.

[0090] For example, each virtual object has a unique individual facial and mouth structure, and each virtual object also has different speaking habits. In this case, it includes reference facial shape information (α1) and N reference facial expression information (β1,…,β). N The facial prior information can reflect the personality or identity characteristics of a virtual object. Specifically, reference facial shape information (α1) characterizes the identity of the virtual object through the texture and shape properties of the face. Optionally, this reference facial shape information can be correlated with natural facial dimensions and oral proportions. On the other hand, reference facial expression information captures an individual's unique speaking style, which provides a difference between the personalized virtual object and the standard shape. Reference facial expression information includes factors such as the amplitude of mouth movements and emotional expression. By comprehensively utilizing this information, facial expression sequences that match the target audio can be generated more accurately, making the virtual object's expressions more vivid and natural.

[0091] As shown in Figure 4A, the audio-to-expression converter is based on a transformer architecture, which includes an encoder and a decoder. In this architecture, when the facial prior information includes reference facial shape information and reference expression information, the step of determining the target expression sequence information may optionally include: processing the reference facial shape information (α1) and the reference expression information (β1,…,β…). N ) and the target audio features (w1,…,w l The audio-to-expression converter encodes the target facial expression (Z) to obtain a hidden vector (Z); and decodes the hidden vector (Z) to determine the target facial expression sequence information. Figure 4B further illustrates some details of the audio-to-expression converter.

[0092] By encoding reference facial shape information and reference expression information in conjunction with target audio features, relevant information such as facial shape and expression associated with virtual objects can be accurately associated with different audios involved in the target audio features. This allows the obtained target expression sequence information to more naturally identify the facial expressions of virtual objects when they emit target audio features.

[0093] Specifically, the 3D coefficients include reference facial shape information α1 (which will serve as the default reference facial shape information), and N reference facial expression information (β1,…,β). N ) and l target audio features (w1,…,w l Embedding vectors can be obtained separately through a fully connected network. These embedding vectors are then concatenated in time order to obtain a total of 1+N+l tokens. These tokens are then input into a Multi-Head Self-Attention (MHSA) (also known as a transformer encoder) for encoding to obtain the hidden vector Z. Z = MHSA(α1,β1,…,β) N ,w1,…,w l (1)

[0094] Expression query vector q t (Initially set to zero) is combined with the hidden vector Z and fed into a Multi-Head Cross-Attention (MHCA) network (i.e., a converter decoder) for decoding to obtain T target expression features.

[0095] In the stage of reconstructing facial expression information into video data, during operation S304, the facial expression to face renderer first obtains any one reference video frame from the reference video data as the source video frame I. s For example, suppose we want to generate the m-th target video frame, then we can extract the m-th reference video frame from the reference video data as the source video frame. Of course, this disclosure is not limited to this.

[0096] Next, the expression-to-face renderer can use a target expression feature vector from the target expression sequence information. and source video frame I s Generate the first video frame The first video frame The facial components in the image are masked. Optionally, the region corresponding to the masked facial components is learnable, meaning that the size and shape of the region can be associated with the expression and identity information of the virtual object.

[0097] Optionally, unlike traditional methods, a first video frame is generated. The process is learnable. As shown in Figure 4B, the source video frame I can be selectively processed. sMasking allows for more precise control over modifying facial component areas. First, 3D vertices are estimated based on predicted expression feature vectors and then projected onto the image. Next, based on these 3D vertices, the mouth and neck areas can be drawn and filled with black pixel blocks (or pixel blocks with arbitrary preset pixel values). In other words, the first video frame contains a masked region relative to the source video frame; this masked region of the first video frame can be black pixel blocks, and this region of the source video frame includes part of the virtual object's facial components.

[0098] The process of generating the mask is as follows:

[0099] Where P is the projection function, and τ t The translation matrix indicating t. C is the matrix used to solve for vertices V in the facial and neck regions of interest. xy The function of the contour, point V xy The set is S.

[0100] By masking the source video frames, the facial region that needs to be predicted can be accurately identified, and this region is directly related to the target audio features emitted by the virtual object, thereby improving the targeting and accuracy of the prediction.

[0101] First video frame and the reference frame I in the reference video data r The two images are fed into a shared-weight encoder to process them and extract multi-scale features. Specifically, the shared-weight encoder processes the first video frame... Encoding as feature F s and reference frame I r Encoding as feature F r .

[0102] Alternatively, as shown in Figure 4A, the weighted encoder consists of d stacked residual blocks, hence it is also called a d-level encoder. At each level (residual block), the image resolution is halved, while the dimension of the extracted features increases. This setup allows for efficient extraction of multi-scale features in a weighted encoder. Because the encoder weights are shared, the first video frame can be processed in parallel. and reference frame I r Encoding is then performed. Specifically, the encoder extracts features of various scales from the input image through multiple levels of processing. For the i-th level in the d-level encoder, its multi-scale features can be calculated as follows.

[0103] Among them, SE iThis represents the i-th level of the encoder. At the end of each level, a representation containing features of different scales and levels of abstraction is obtained. These features can capture different details and structural information of the input image. Thus, for the first video frame... and reference frame I r We obtain a set of eigenvectors of size d, that is... and Due to the first video frame and reference frame I r By sharing encoder weights, source and reference video frames can be processed more efficiently to extract their shared feature representations. This parallel computing approach not only speeds up processing but also ensures that the feature representations of source and reference video frames are compared in the same space, which is beneficial for subsequent matching and fusion.

[0104] Next, the expression-to-face renderer will use a decoder, including a facial identity alignment module, to process the first video frame. Corresponding feature F s and reference frame I r Corresponding feature F r Decode the video to obtain the target video frame.

[0105] As can be seen, by first determining the target facial expression sequence and then predicting video frames based on it during the inference process using the improved neural network, the synchronization between the audio content and the facial components of the virtual object on the screen is significantly improved. This method not only enhances the realism of facial movements but also accurately reflects the identity characteristics of the virtual object. While improving performance, it also ensures the operational efficiency and stability of the neural network, making it particularly suitable for real-time video generation.

[0106] Figure 4D illustrates more example details of the decoder. Specifically, the decoder consists of d stacked decoding layers, hence it is also called a d-level decoder. For example, the decoder includes d decoding layers, each layer's input including the output of the previous decoding layer, the face prior information, and the encoded features of the reference frame, and each decoding layer's output is a predicted value for the target video frame. Assume the input features for each level of the decoder are... Output features are

[0107] As shown in Figure 4D, the 3D coefficients Before being fused to the decoder via the adaptor (AdalN) module, dimensionality mapping is performed through a three-layer MLP. This process aims to map the 3D coefficients to a feature space compatible with the decoder network for subsequent feature fusion and generation processes. Thus, the input features of each level of the decoder... First, upsampling is performed to increase the resolution in order to obtain features. feature Then after 3D coefficients Modulation is applied to match the shape of specific facial components, thereby improving control over facial component movement and facial expression generation. The modulated features are compared with those from reference frame I. r Features Aligned at the same resolution, the two are blended to control face generation and texture aggregation. The above process can be described as follows.

[0108] FIA i This is a first-level (or first-layer) decoding layer in a decoder that includes a facial identity alignment module.

[0109] Furthermore, to avoid generating unnecessary background in the target video frame and to ensure a more realistic and accurate final result, a fusion strategy can be further employed. Specifically, in some embodiments of this disclosure, the output of the last decoding layer of the decoder can be fused with the output of the first decoding layer of the decoder to obtain the final output. This fusion strategy aims to retain only the pixels corresponding to the predicted facial components while eliminating background interference, resulting in a clearer image that focuses on facial features. This process can be described as follows.

[0110] Figure 5A shows a flowchart of a method 50 for training a neural network according to an embodiment of the present disclosure, which can train the improved neural network model shown in Figures 4A to 4C. Figure 5B shows a schematic diagram of training an audio-to-expression converter according to an embodiment of the present disclosure.

[0111] Method 50 can be executed at a terminal device or a server (such as terminal 120 or server 110 as shown in FIG1). Method 50 includes the following operations S501 to S506. Of course, method 50 may also include more or fewer operations, and this disclosure is not limited thereto.

[0112] In operation S501, prior facial information of the virtual object is determined based on the sample video data. In operation S502, the audio features of the sample target are determined based on the sample audio data corresponding to the sample video data.

[0113] For example, suppose the facial prior information obtained based on sample video data includes: sample facial shape information (α1), sample target expression sequence information (e.g., sample expression information β1,…,β1). N ), and the pose information of the sample head part (ρ1,…,ρ NThese facial prior information can be obtained directly through preprocessing. Such facial prior information can be used as the true value of the facial prior information of virtual objects. The preprocessing process has been described in detail above, and will not be repeated here.

[0114] Next, referring to the preprocessing described in detail above, the sample audio data corresponding to the sample video data can be extracted to directly determine the sample audio features corresponding to the sample audio data. Here, the sample audio data corresponding to the sample video data refers to the audio data containing the phrase "The weather is nice today" in the sample video data, assuming a virtual person is saying "The weather is nice today".

[0115] Next, the training process of the audio-to-expression converter is described based on operations S503 and S504. Those skilled in the art should understand that this disclosure is not limited thereto.

[0116] In operation S503, based on the sample facial prior information and the sample target audio features, the audio-to-expression converter is used to determine the predicted value of the target sample expression sequence information. In operation S504, based on the predicted value of the target sample expression sequence information, the audio-to-expression converter is trained using a first loss function. Optionally, the sample facial prior information may include the true value of the sample target expression sequence information. Operation S503 includes: masking the true value of the virtual object's facial prior information to obtain the masked facial prior information of the virtual object; and based on the masked facial prior information of the virtual object and the sample audio features, using the audio-to-expression converter to determine the predicted value of the sample target expression sequence information's facial prior information of the virtual object. Of course, this disclosure is not limited thereto.

[0117] Optionally, to determine the predicted value of the sample target facial expression sequence information of the virtual object, it is first necessary to mask its ground truth value to obtain the masked sample target facial expression sequence information of the virtual object. This process involves hiding or masking a portion of the ground truth value, thereby generating a masked version. The masked information can be masked according to specific conditions or rules, for example, hiding specific reference facial expression information β. t Next, based on these masked sample target facial expression sequence information (i.e., excluding specific reference facial expression information β), t Based on facial prior information and sample audio features, the audio-to-expression converter trained in the training can be used to predict the masked portion of the sample target expression sequence information of the virtual object (e.g., facial prior information) and sample audio features. The training objective for the audio-to-expression converter is to maximize the true value β of the facial prior information. t Compared with the predicted value It is close enough. Therefore, the parameters of each neuron in the audio-to-expression converter can be continuously adjusted to make the first loss function reach its extreme value. Of course, this disclosure is not limited to this.

[0118] Optionally, as shown in Figure 5B, the first loss function used to train the audio-to-expression converter may include, but is not limited to, mean squared error (MSE) loss and 3D vertex loss.

[0119] For example, the mean squared error (MSE) loss is used to calculate the error between the predicted and true values ​​of an expression feature vector. Assume the predicted expression feature vector is... For, the true facial expression feature vector is β. t The MSE loss can then be expressed as:

[0120] Optionally, the 3D vertex loss is obtained by combining the predicted expression feature vectors. The predicted positions of 3D vertices are calculated using facial shape information α1 and reference head pose information ρ. These vertices are keypoints (e.g., corners of the mouth) selected from facial regions (e.g., the mouth) to evaluate the distance between the ground truth and predicted values ​​of keypoints in that facial region. Optionally, by minimizing the difference between the predicted and ground truth values ​​of keypoint positions, accurate alignment of the generated 3D vertices with the mouth keypoints can be ensured. Let V represent the 3D vertex calculated from the coefficients. 3D vertex loss L V It can be described as follows:

[0121] Therefore, the first loss function can be expressed as L a2e =L MSE +0.1*C V Here, 0.1 is a hyperparameter, which can also be set to any other arbitrary value. This disclosure is not limited thereto.

[0122] Next, the training process of the expression-to-face renderer is described based on operations S505 and S506. Those skilled in the art should understand that this disclosure is not limited thereto. The training process of the audio-to-expression converter and the training process of the expression-to-face renderer can be independent of each other. Specifically, as detailed below, the input data used in the training process of the expression-to-face renderer can be the sample facial prior information (which optionally includes the true values ​​of the sample target expression sequence information), the sample reference frame, and the sample source video frame. These input data can all be obtained in the preprocessing process (i.e., operations S501 and S502) without going through the audio-to-expression converter. Of course, in other embodiments, the audio-to-expression converter and the expression-to-face renderer can also be jointly trained, and this disclosure is not limited thereto.

[0123] Specifically, the sample facial prior information includes the true values ​​of the sample target expression sequence information. In operation S505, a sample reference frame and a sample source video frame are determined based on the sample video data. Then, in operation S506, the expression-to-face renderer is trained using a second loss function based on the sample facial prior information, the sample reference frame, and the sample source video frame.

[0124] Optionally, in operation S506, the sample source video frame is masked based on the true value of the target expression sequence to obtain a masked sample source video frame; based on the sample face prior information and the true values ​​of the sample reference frame and the sample target expression sequence information, the predicted value of the sample source video frame is determined using the expression-to-face renderer for the masked sample source video frame; and based on the predicted value and the true value of the sample source video frame, the expression-to-face renderer is trained using the second loss function; wherein the second loss function includes one or more of the following: pixel loss, perceptual loss, adversarial loss, and local pixel loss.

[0125] Optionally, the prior information of the sample face includes 3D coefficients. Referring to the method detailed above, the sample source video frames can first be masked (i.e., the masked sample source video frames have their facial components replaced by black pixel blocks). Then, based on these 3D coefficients and the sample reference frame, the predicted value of the sample source video frame is determined. The training objective of the expression-to-face renderer is to make the true value of the sample source video frame sufficiently close to the predicted value. Therefore, the parameters of each neuron in the expression-to-face renderer can be continuously adjusted to make the second loss function reach its extreme value. Of course, this disclosure is not limited to this.

[0126] Optionally, the second loss function used to train the expression-to-video renderer can employ a combination of multiple loss functions to constrain the synchronization between facial components and expressions, thereby improving the visual quality of the generated video frames.

[0127] For example, pixel loss can be used to measure the pixel-level difference between the generated image and the target image. This is achieved by minimizing the predicted values ​​of the sample source video frames. and the true value I s The pixel differences between the pixels contribute to the accuracy and sharpness of the generated video frames. As an example, the pixel loss L1 can be calculated as: Of course, this disclosure is not limited to this.

[0128] For example, perceptual loss measures the predicted value of the source video frames used to generate the sample, based on features extracted by deep learning models (such as VGG networks), i.e., perceptual information obtained from the image. and the true value I s The perceptual difference between the generated images and the target expressions. Perceptual loss helps improve the realism and naturalness of the generated images to better match the target expressions. Perceptual loss can be calculated as:

[0129] For example, adversarial loss evaluates the predictions of sample source video frames through a discriminator network in a generative adversarial network (GAN) framework. The generator network aims to deceive the discriminator, making it unable to distinguish the predicted values ​​of the sample source video frames. and the true value I s This improves the realism and fidelity of the image. The adversarial loss can be calculated as: Where D(·) represents the discriminator.

[0130] For example, local pixel loss focuses on enhancing pixel-level details in specific regions, such as the details of the mouth. By introducing an additional loss term for specific facial regions, the generated image becomes sharper and more realistic in details such as the mouth. Local pixel loss can be calculated as: Where M' represents the binary mask of the target facial region.

[0131] The second loss function is obtained by weighted summation of the above loss functions. These loss functions work together during the training of the expression-to-video renderer to ensure that the generated video accurately reflects the target expression while maintaining high levels of visual quality and detail. The second loss function can be calculated as L. e2f =λ1*L1+λ2*L2+λ3*L3+λ4*L4, where λ1=1, λ2=1, λ3=0.1, λ4=1. λ1 to λ4 are all hyperparameters, which can be set to any other value in practical applications, and this disclosure is not limited to this.

[0132] In the training process of the improved neural network model, an improved first loss function and a second loss function are optionally introduced. These two loss functions optimize for different characteristics, improving the performance of the trained neural network by adjusting its parameters. The first and second loss functions enable the neural network to learn and adapt to the features of the input data more effectively and optimize from multiple aspects during training. This comprehensive design of loss functions, considering different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby improving its performance in practical applications.

[0133] This disclosure also provides an apparatus for processing audio data, the apparatus comprising: a first preprocessing module for determining facial prior information of a virtual object based on reference video data; a second preprocessing module for determining target audio features based on target audio data; an audio-to-expression converter module for determining target expression sequence information based on the facial prior information and the target audio features; and an expression-to-face renderer for determining target video data corresponding to the target audio data based on the target expression sequence information and reference frames in the reference video data.

[0134] In one possible implementation, the facial prior information includes one or more of the following: reference facial shape information, reference expression information, and reference head pose information.

[0135] In one possible implementation, when the facial prior information includes the reference facial shape information and the reference expression information, the audio-to-expression converter module is further configured to:

[0136] The reference facial shape information, the reference expression information, and the target audio features are encoded to obtain a hidden vector; and

[0137] The hidden vector is decoded to determine the target facial expression sequence information.

[0138] In one possible implementation, the expression-to-face renderer is further used for:

[0139] Extract source video frames from the reference video data;

[0140] The source video frame is masked based on the target facial expression sequence to obtain a first video frame, wherein in the first video frame, some facial components of the virtual object are replaced by black pixel blocks; and

[0141] Based on the first video frame and the reference frame, the target video data corresponding to the target audio data is determined using the facial prior information.

[0142] In one possible implementation, the expression-to-face renderer is further used for:

[0143] Based on the target expression sequence and the reference facial shape information, determine the pixel regions corresponding to certain facial components of the virtual object in the source video frame; and

[0144] The pixel values ​​in the pixel region are modified to preset values ​​to obtain the first video frame.

[0145] In one possible implementation, the expression-to-face renderer is further used for:

[0146] A weighted encoder is used to encode the first video frame and the reference frame to obtain a first coding feature of the first video frame and a second coding feature of the reference frame.

[0147] Using the facial prior information and the target expression sequence, a decoder is used to decode the first encoded feature and the second encoded feature to obtain target video data corresponding to the target audio data. The target video data includes target video frames, and the black pixel blocks in the target video frames are replaced with predicted values ​​of some facial components of the virtual object.

[0148] In one possible implementation, the decoder includes multiple cascaded decoding layers, each decoding layer's input including the output of the previous decoding layer, the facial prior information, and the encoded features of the reference frame, and each decoding layer's output being a predicted value for the target video frame.

[0149] It is evident that by determining the prior facial information of virtual objects in the reference video data, the facial features of virtual objects can be accurately identified. This allows for the determination of target expression sequences that conform to and naturally integrate these facial features, based on the prior facial information and target audio features. When using target expression sequences and reference frames to determine target video data, the synchronization accuracy between the audio content in the target video data and the facial components (e.g., mouth) movements of virtual objects on the screen can be significantly improved, enhancing the realism of mouth movements. Furthermore, it can accurately reflect the identity characteristics of virtual objects, more accurately capture and reproduce individual differences in expression, and while improving system performance, it also ensures overall operational efficiency and stability, making it particularly suitable for real-time video generation.

[0150] This disclosure also provides an apparatus for training a neural network, the neural network including an audio-to-expression converter and an expression-to-face renderer, the apparatus comprising:

[0151] The first determining module is used to determine the sample facial prior information of the virtual object based on the sample video data;

[0152] The second determining module is used to determine the target audio features of the sample based on the sample audio data corresponding to the sample video data.

[0153] The third determining module is used to determine the predicted value of the target sample's facial expression sequence information based on the sample's prior facial information and the sample's target audio features, using the audio-to-expression converter.

[0154] The first training module is used to train the audio-to-expression converter based on the predicted values ​​of the target sample expression sequence information and using a first loss function.

[0155] The fourth determining module is used to determine the sample reference frame and the sample source video frame based on the sample video data; and

[0156] The second training module is used to train the expression-to-face renderer based on the sample facial prior information, the sample reference frame, and the sample source video frame, using a second loss function.

[0157] In one possible implementation, the sample facial prior information includes the true value of the sample target expression sequence information, and the third determining module is further configured to:

[0158] The true value of the virtual object's facial prior information is masked to obtain the masked facial prior information of the virtual object;

[0159] Based on the facial prior information of the masked virtual object and the sample audio features, the predicted value of the sample target facial expression sequence information of the virtual object is determined using the audio-to-expression converter.

[0160] In one possible implementation, the sample facial prior information includes the true values ​​of the sample target expression sequence information, and the second training module is further configured to:

[0161] Based on the true values ​​of the sample target facial expression sequence, the sample source video frame is masked to obtain the masked sample source video frame;

[0162] Based on the sample facial prior information and the sample reference frame, for the masked sample source video frame, the expression-to-face renderer is used to determine the predicted value of the sample source video frame; and

[0163] Based on the predicted and actual values ​​of the sample source video frames, the expression-to-face renderer is trained using the second loss function.

[0164] The second loss function includes one or more of the following: pixel loss, perceptual loss, adversarial loss, and local pixel loss.

[0165] As can be seen, improved first and second loss functions can be selectively introduced during the training process of the improved neural network model. These two loss functions optimize for different characteristics, improving the performance of the trained neural network by adjusting its parameters. The first and second loss functions enable the neural network to learn and adapt to the features of the input data more effectively and optimize from multiple aspects during training. This comprehensive design of loss functions, considering different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby enhancing its performance in practical applications.

[0166] As shown in Figure 6, the predicted result of the video frame obtained by the scheme according to the embodiment of this disclosure is very close to the actual result of the video frame, and its degree of restoration of detail texture is much higher than that of the traditional scheme.

[0167] According to another aspect of this disclosure, an electronic device is also provided for implementing the method according to an embodiment of this disclosure. FIG7 shows a schematic diagram of an electronic device 2000 according to an embodiment of this disclosure.

[0168] As shown in Figure 8, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code, which, when executed by the one or more processors 2010, can perform the methods described above.

[0169] The processor in this disclosure embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, operations, and logic block diagrams disclosed in this disclosure embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0170] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0171] For example, the methods or apparatus according to embodiments of this disclosure can also be implemented using the architecture of the computing device 3000 shown in FIG. 9. As shown in FIG. 8, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, the architecture shown in FIG. 8 is only exemplary, and one or more components in the computing device shown in FIG. 8 may be omitted as needed when implementing different devices.

[0172] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 9 shows a schematic diagram of the storage medium 4000 according to this disclosure.

[0173] As shown in FIG9, computer-readable instructions 4010 are stored on the computer storage medium 4020. When the computer-readable instructions 4010 are executed by a processor, the methods according to embodiments of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0174] In addition, this application embodiment also provides a storage medium for storing a computer program for executing the method provided in the above embodiment.

[0175] This application also provides a computer program product including a computer program, which, when run on a computer, causes the computer to perform the method provided in the above embodiments.

[0176] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0177] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0178] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A method for processing audio data, the method being performed by a computer device, the method comprising: Based on reference video data, determine the prior facial information of the virtual object; Based on the target audio data, determine the target audio features; Based on the facial prior information and the target audio features, the target expression sequence information is determined; as well as Based on the target facial expression sequence information and the reference frames in the reference video data, the target video data corresponding to the target audio data is determined.

2. The method for processing audio data as described in claim 1, wherein the facial prior information includes one or more of the following: reference facial shape information, reference expression information, and reference head pose information.

3. The method for processing audio data as described in claim 2, wherein when the facial prior information includes the reference facial shape information and the reference expression information, determining the target expression sequence information based on the facial prior information and the target audio features includes: The reference facial shape information, the reference expression information, and the target audio features are encoded to obtain a hidden vector; as well as The hidden vector is decoded to determine the target facial expression sequence information.

4. The method for processing audio data as described in claim 2, wherein determining the target video data corresponding to the target audio data based on the target facial expression sequence information and reference frames in the reference video data includes: Extract source video frames from the reference video data; The source video frame is masked based on the target expression sequence to obtain a first video frame, wherein in the first video frame, some facial components of the virtual object are replaced by black pixel blocks; as well as Based on the first video frame and the reference frame, the target video data corresponding to the target audio data is determined using the facial prior information.

5. The method for processing audio data as described in claim 4, wherein masking the source video frame based on the target facial expression sequence to obtain the first video frame comprises: Based on the target expression sequence and the reference facial shape information, the pixel regions corresponding to some facial components of the virtual object in the source video frame are determined. as well as The pixel values ​​in the pixel region are modified to preset values ​​to obtain the first video frame.

6. The method for processing audio data as described in claim 4, wherein determining the target video data corresponding to the target audio data based on the first video frame and the reference frame, using the facial prior information, comprises: A weighted encoder is used to encode the first video frame and the reference frame to obtain a first coding feature of the first video frame and a second coding feature of the reference frame. Using the facial prior information and the target expression sequence, a decoder is used to decode the first encoded feature and the second encoded feature to obtain target video data corresponding to the target audio data. The target video data includes target video frames, and the black pixel blocks in the target video frames are replaced with predicted values ​​of some facial components of the virtual object.

7. The method for processing audio data as described in claim 6, wherein the decoder comprises a plurality of cascaded decoding layers, the input of each decoding layer includes the output of the previous decoding layer, the facial prior information and the encoded features of the reference frame, and the output of each decoding layer is a predicted value of the target video frame.

8. A method for training a neural network, the neural network including an audio-to-expression converter and an expression-to-face renderer, the method being performed by a computer device, the method comprising: Based on sample video data, determine the sample facial prior information of virtual objects; Based on the sample audio data corresponding to the sample video data, determine the target audio features of the sample; Based on the prior information of the sample face and the audio features of the sample target, the predicted value of the target sample expression sequence information is determined using the audio-to-expression converter. Based on the predicted values ​​of the target sample facial expression sequence information, the audio-to-facial expression converter is trained using a first loss function; Based on the sample video data, the sample reference frame and the sample source video frame are determined; as well as Based on the sample facial prior information, the sample reference frame, and the sample source video frame, the expression-to-face renderer is trained using a second loss function.

9. The method for training a neural network as described in claim 8, wherein the sample facial prior information includes the true value of the sample target expression sequence information, and the step of determining the predicted value of the sample target expression sequence information using the audio-to-expression converter includes: The true value of the virtual object's facial prior information is masked to obtain the masked facial prior information of the virtual object; Based on the facial prior information of the masked virtual object and the sample audio features, the predicted value of the sample target facial expression sequence information of the virtual object is determined using the audio-to-expression converter.

10. The method for training a neural network as described in claim 8, wherein the sample facial prior information includes the true values ​​of the sample target expression sequence information, and the training of the expression-to-face renderer includes: Based on the true values ​​of the sample target facial expression sequence, the sample source video frame is masked to obtain the masked sample source video frame; Based on the sample facial prior information and the sample reference frame, for the masked sample source video frame, the expression-to-face renderer is used to determine the predicted value of the sample source video frame. as well as Based on the predicted and actual values ​​of the sample source video frames, the expression-to-face renderer is trained using the second loss function. The second loss function includes one or more of the following: pixel loss, perceptual loss, adversarial loss, and local pixel loss.

11. An apparatus for processing audio data, the apparatus comprising: The first preprocessing module is used to determine the facial prior information of the virtual object based on the reference video data: The second preprocessing module is used to determine the target audio features based on the target audio data; An audio-to-expression converter module is used to determine target expression sequence information based on the facial prior information and the target audio features; An expression-to-face renderer is used to determine target video data corresponding to the target audio data based on the target expression sequence information and reference frames in the reference video data.

12. An apparatus for training a neural network, the neural network including an audio-to-expression converter and an expression-to-face renderer, the apparatus comprising: The first determining module is used to determine the sample facial prior information of the virtual object based on the sample video data; The second determining module is used to determine the target audio features of the sample based on the sample audio data corresponding to the sample video data. The third determining module is used to determine the predicted value of the target sample's facial expression sequence information based on the sample's prior facial information and the sample's target audio features, using the audio-to-expression converter. The first training module is used to train the audio-to-expression converter based on the predicted values ​​of the target sample expression sequence information and using a first loss function. The fourth determining module is used to determine the sample reference frame and the sample source video frame based on the sample video data; as well as The second training module is used to train the expression-to-face renderer based on the sample facial prior information, the sample reference frame, and the sample source video frame, using a second loss function.

13. A computer device, the computer device comprising: Processor, communication interface, memory, and communication bus; The processor, the communication interface, and the memory communicate with each other via the communication bus; the communication interface is the interface of the communication module. The memory is used to store computer programs and transmit the computer programs to the processor; The processor is configured to invoke a computer program stored in memory to execute the method described in any one of claims 1-7, or to execute the method described in any one of claims 8-10.

14. A storage medium for storing a computer program for performing the method of any one of claims 1-7, or the method of any one of claims 8-10.

15. A computer program product comprising a computer program, which, when run on a computer, causes the computer to perform the method of any one of claims 1-7, or the method of any one of claims 8-10.

Images