Audio data processing method, neural network training method, and related apparatus

Abstract

The present disclosure relates to an audio data processing method, a neural network training method, an audio data processing apparatus, an electronic device, and a computer readable storage medium. The audio data processing method comprises: determining facial prior information of a virtual object on the basis of reference video data; determining a target audio feature on the basis of target audio data; determining target expression sequence information on the basis of the facial prior information and the target audio feature; and on the basis of the target expression sequence information and a reference frame in the reference video data, determining target video data corresponding to the target audio data. According to the present disclosure, the synchronization precision between audio content and a facial component (e.g., the mouth) action of a virtual object can be significantly improved, the sense of reality of the mouth action is enhanced, an expression difference between individuals is captured and reproduced more accurately, and the operation efficiency and stability of the whole system are improved while the system performance is improved.

Description

Method for processing audio data, method for training neural network and related devices

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on February 29, 2024, with application number 2024102321570 and application name “Method and Apparatus for Processing Audio Data”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of artificial intelligence services, and more specifically to processing audio data. Background Art

[0003] In recent years, technologies for generating video data from audio data have rapidly developed. Audio-driven face generation, in particular, holds broad application prospects. This technology parses input audio signals to extract information such as emotion and intonation, and uses this information to control the expressions and movements of virtual characters in real time, achieving highly synchronized performance. This technology has demonstrated widespread application value in the entertainment industry, game development, human-computer interaction interfaces, and virtual reality conferencing. With the continued advancement of deep learning and neural network technologies, its expressiveness and realism are continuously improving.

[0004] Despite this, current audio-driven virtual face generation technology still faces a series of technical challenges. The primary challenge is the difficulty in achieving precise synchronization between audio data and mouth movements, which results in the generated facial animation often not being able to perfectly match the speech content. Secondly, when it comes to processing complex facial micro-details and textures, current technology is still unable to achieve the ideal realistic visual effect, resulting in low realism of virtual faces. In addition, how to ensure that different virtual objects present facial expressions consistent with their unique personalities, thereby truly reflecting individual differences, is also a problem that needs to be solved urgently. Finally, improving the system's operating efficiency and overall stability to meet the needs of more real-time application scenarios is also an important direction for future improvements to this technology.

[0005] Therefore, audio-driven virtual face generation technology still needs further optimization and improvement. Summary of the Invention

[0006] Embodiments of the present disclosure provide a method for processing audio data, a method for training a neural network, an apparatus for processing audio data, an electronic device, and a computer-readable storage medium.

[0007] An embodiment of the present disclosure provides a method for processing audio data, the method comprising: determining facial prior information of a virtual object based on reference video data; determining target audio features based on target audio data; determining target expression sequence information based on the facial prior information and the target audio features; and determining target video data corresponding to the target audio data based on the target expression sequence information and a reference frame in the reference video data.

[0008] An embodiment of the present disclosure provides a method for training a neural network, wherein the neural network includes an audio-to-expression converter and an expression-to-face renderer, and the method includes: determining sample facial prior information of a virtual object based on sample video data; determining sample target audio features based on sample audio data corresponding to the sample video data; determining a predicted value of target sample expression sequence information using the audio-to-expression converter based on the sample facial prior information and the sample target audio features; training the audio-to-expression converter using a first loss function based on the predicted value of the target sample expression sequence information; determining a sample reference frame and a sample source video frame based on the sample video data; and training the expression-to-face renderer using a second loss function based on the sample facial prior information, the sample reference frame, and the sample source video frame.

[0009] An embodiment of the present disclosure provides a device for processing audio data, the device comprising: a first preprocessing module, configured to determine facial prior information of a virtual object based on reference video data; a second preprocessing module, configured to determine target audio features based on target audio data; and an audio-to-expression converter module, configured to determine target expression sequence information based on the facial prior information and the target audio features; and an expression-to-facial renderer, configured to determine target video data corresponding to the target audio data based on the target expression sequence information and a reference frame in the reference video data.

[0010] In another aspect, an embodiment of the present application provides a computer device, including:

[0011] Processor, communication interface, memory and communication bus;

[0012] Wherein, the processor, the communication interface and the memory complete communication with each other through the communication bus; the communication interface is an interface of the communication module;

[0013] The memory is used to store a computer program and transmit the computer program to the processor; the processor is used to call the computer program in the memory to execute the above method.

[0014] On the other hand, an embodiment of the present application provides a storage medium, which is used to store a computer program, and the computer program is used to execute the method of the above aspect.

[0015] On the other hand, an embodiment of the present application provides a computer program product including a computer program, which, when executed on a computer, enables the computer to execute the above method.

[0016] The disclosed embodiment can accurately identify the facial features of the virtual object by determining the facial prior information of the virtual object in the reference video data, and thus can determine the target expression sequence information that conforms to the facial features of the virtual object and naturally integrates the facial features based on the facial prior information and the target audio features. When the target expression sequence information and the reference frame are used to determine the target video data, the synchronization accuracy between the audio content in the target video data and the facial components (such as the mouth) movements of the virtual object on the screen can be significantly improved, the real texture of the mouth movement can be enhanced, and the identity characteristics of the virtual object can be accurately reflected, and the expression differences between individuals can be more accurately captured and reproduced. Moreover, while improving the performance of the system, the overall operating efficiency and stability are also ensured, which is particularly suitable for real-time video generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is an exemplary schematic diagram illustrating a scenario according to an embodiment of the present disclosure;

[0018] FIG2 shows a schematic diagram according to an embodiment of the present disclosure;

[0019] FIG3 shows a flowchart of a method for processing audio data according to an embodiment of the present disclosure;

[0020] FIG4A shows a schematic diagram of an improved neural network model according to an embodiment of the present disclosure;

[0021] FIG4B shows a schematic diagram of the audio to expression conversion stage according to an embodiment of the present disclosure;

[0022] FIG4C shows a schematic diagram of an audio to expression converter according to an embodiment of the present disclosure;

[0023] FIG4D shows a schematic diagram of an expression-to-face renderer according to an embodiment of the present disclosure;

[0024] FIG5A shows a flowchart of a method for training a neural network according to an embodiment of the present disclosure;

[0025] FIG5B shows a schematic diagram of training an audio-to-expression converter according to an embodiment of the present disclosure;

[0026] FIG6 shows a comparison diagram of test results according to an embodiment of the present disclosure;

[0027] FIG7 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure;

[0028] FIG8 shows a schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure;

[0029] FIG9 shows a schematic diagram of a storage medium according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, the following will describe in detail exemplary embodiments of the present disclosure with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0031] In this specification and the accompanying drawings, substantially the same or similar operations and elements are represented by the same or similar reference numerals, and repeated descriptions of these operations and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance or ranking.

[0032] The models that can be used in the embodiments of the present disclosure can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs in which neurons are arranged in different layers. Typically, a neural network model includes an input layer and an output layer, which are separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation that is useful for generating output in the output layer. The nodes are fully connected to the nodes in the adjacent layers via edges, and there are no edges between the nodes in each layer. The data received at the nodes of the input layer of the neural network is propagated to the nodes of the output layer via any one of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and the present disclosure does not limit this.

[0033] First, an application scenario of the method for processing audio data and corresponding apparatus according to an embodiment of the present disclosure is described with reference to Figure 1. Figure 1 shows a schematic diagram of an application scenario 100 according to an embodiment of the present disclosure, in which a server 110 and multiple terminals 120 are schematically shown.

[0034] The neural network model of the embodiment of the present disclosure can be specifically integrated into various electronic devices, for example, any electronic device in the server 110 and multiple terminals 120 in Figure 1. For example, the neural network model can be integrated into the terminal 120. The terminal 120 can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal computer (PC), a smart speaker or a smart watch, etc., but is not limited to this. For another example, the neural network model can also be integrated into the server 110. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and the present disclosure does not limit this.

[0035] It is understood that the computer device that performs inference using the neural network model of the embodiments of the present disclosure can be either a terminal device, a server, or a system consisting of a terminal device and a server. The method for processing audio data of the embodiments of the present disclosure can be executed on a terminal device, on a server, or jointly by a terminal device and a server.

[0036] It is worth noting that the terminal 110 and the server 120 according to the embodiment of the present disclosure comply with the data protection principles, respect the user's data rights, and protect the user's data security and privacy. According to the embodiment of the present disclosure, the terminal 110 and the server 120 will clearly inform the user of the purpose, method and scope of collecting, using, storing, transmitting and deleting the user's data, and obtain the user's consent. According to the embodiment of the present disclosure, the terminal 110 and the server 120 will take reasonable technical and management measures to prevent the user's data from being leaked, tampered with, damaged or lost. According to the embodiment of the present disclosure, the terminal 110 and the server 120 provider will regularly review and update the user's data and delete expired or useless data in a timely manner. In addition, the cloud service provider using the embodiment of the present disclosure respects the user's rights to data access, correction, deletion, withdrawal of consent, complaints and claims, and provides convenient channels and procedures so that users can effectively exercise these rights.

[0037] Furthermore, the data analysis process using artificial intelligence technology in the terminal 120 or server 110 is based on the principles of legality, rationality, and transparency. The data collected and processed by the artificial intelligence model according to the embodiments of the present disclosure is relevant, necessary, and appropriate for the prediction purpose and does not contain any personally identifiable information or sensitive information. The neural network model according to the embodiments of the present disclosure adopts appropriate technical and organizational measures to protect the security and integrity of the data and prevent unauthorized access, use, or disclosure of the data.

[0038] The artificial intelligence-based neural network model according to the embodiments of the present disclosure will comply with relevant data protection regulations and ethical principles. The neural network model is trained based on a large amount of anonymized and de-identified data and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable and will not cause any misleading or discriminatory effects. The artificial intelligence model is designed only to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. In addition, the neural network model will be reviewed and updated regularly to adapt to changes in the data environment and legal regulations.

[0039] Traditionally, audio-driven virtual face generation methods can be divided into two categories: person-specific methods and person-generic methods. Person-specific methods require neural network models to be trained or fine-tuned on a specific individual before prediction, while person-generic methods can directly generate speaking facial videos of unseen virtual characters.

[0040] To address the audio-visual synchronization issues in the above two methods, some solutions have introduced lip synchronization discriminators or lip reading networks to enhance the intelligibility of the mouth area. However, such solutions still cannot solve the audio-visual synchronization problem. To improve video quality, some solutions have adopted diffusion models or adversarial generative models to attempt to improve image fidelity. However, such solutions struggle to strike a balance between efficiency, visual quality, and mouth movement accuracy.

[0041] Furthermore, when establishing a correspondence between audio data and facial expressions, some solutions employ fusion modeling approaches, aiming to more efficiently and accurately capture mouth movement features. Specifically, they can predict target facial region labels based on audio data or extract key point information in an unsupervised environment. Although facial region labels and key points are relatively easy to obtain, the inherent sparsity of this information prevents these solutions from fully expressing complex facial movements such as pursing or closing the lips.

[0042] The disclosed embodiments improve the neural network model architecture. Compared to the single-module structure of traditional neural network models for audio-driven virtual face generation, the improved neural network model consists of two decoupled modules: an audio-to-expression transformer and an expression-to-face renderer.

[0043] As a result, the improved neural network model can process input data in three stages to achieve higher performance: preprocessing, audio-to-expression conversion, and expression-to-video frame reconstruction.

[0044] Specifically, first, in the preprocessing stage, the preprocessing module can perform necessary feature extraction on the reference video data and target audio data to obtain the facial prior information and target audio features of the virtual object. Then, in the audio-to-expression conversion stage, the audio-to-expression converter maps the received facial prior information and target audio features into corresponding target expression sequence information, especially for subtle expression changes in mouth movements. Finally, in the expression-to-video frame reconstruction stage, the expression-to-facial renderer accurately generates target video data corresponding to the target audio data based on the target expression sequence information, the reference frame in the reference video data, and the reference head pose feature vector corresponding to the reference frame. The target video data is precisely synchronized with the target audio data and has highly realistic mouth movements and facial expression details.

[0045] Compared with the single-stage solution of the traditional neural network model, that is, directly inputting audio data and video data to generate updated video images, the improved multi-stage neural network model according to the embodiment of the present disclosure can significantly improve the synchronization accuracy between the audio content and the facial components (such as the mouth) movements of the virtual object on the screen, enhance the real texture of the mouth movement, more accurately capture and reproduce the differences in facial expressions between individuals, and, while improving the performance of the system, also ensure the overall operating efficiency and stability.

[0046] In response to the improved neural network model, embodiments of the present disclosure provide a method for processing audio data to utilize the improved neural network model for inference. The method comprises: determining facial prior information of a virtual object based on reference video data; determining target audio features based on target audio data; determining target expression sequence information based on the facial prior information and the target audio features; and determining target video data corresponding to the target audio data based on the target expression sequence information, a reference frame in the reference video data, and a reference head pose feature vector corresponding to the reference frame.

[0047] The improved neural network inference process significantly improves the synchronization between the audio content and the facial components of the virtual object on screen by first determining the target expression sequence and then predicting the video frame based on this sequence. This approach not only enhances the realism of facial movements but also accurately reflects the identity of the virtual object. While improving performance, it also ensures the efficiency and stability of the neural network, making it particularly suitable for real-time video generation.

[0048] Similarly, for the improved neural network model, the embodiment of the present disclosure also provides a method for training a neural network to train the improved neural network model. The neural network includes an audio-to-expression converter and an expression-to-face renderer. The method includes: training the audio-to-expression converter based on sample video data and sample audio data corresponding to the sample video data; and training the expression-to-face renderer based on sample facial prior information, sample reference frames, and sample source video frames. The audio-to-expression converter and the expression-to-face renderer can be trained separately and independently, and the training processes of the two do not interfere with each other.

[0049] During the training process of the improved neural network model, an improved first loss function and a modified second loss function are optionally introduced. These two loss functions optimize different aspects of the neural network's performance by adjusting the neural network parameters. The first and second loss functions enable the neural network to more effectively learn and adapt to the characteristics of the input data, optimizing multiple aspects during the training process. This loss function design, which comprehensively considers different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby enhancing its performance in practical applications.

[0050] The following describes a method for processing audio data, a method and an apparatus for training a neural network according to embodiments of the present disclosure with reference to FIG. 2 to FIG. 9 .

[0051] FIG2 shows a schematic diagram according to an embodiment of the present disclosure.

[0052] As shown in Figure 2, the specific application scenario for implementing audio-driven virtual face generation technology is as follows: there is a video clip (hereinafter referred to as reference video data) that corresponds to an audio clip. The video content shows the facial expressions of a virtual character speaking to the camera, especially the details of their mouth movements. In this video clip, the virtual character utters a sentence, such as "What a nice day today." To quickly produce more videos of this virtual character, it is necessary to synchronize the virtual character's lip movements in this video with another different audio clip (hereinafter referred to as target audio data), such as "I love you." That is, through technical means, the virtual character originally saying "What a nice day today" can generate target video data based on the other audio clip. In this target video data, the virtual character's lip movements can accurately match the form and rhythm of saying "I love you."

[0053] As shown in FIG2 , the neural network model according to an embodiment of the present disclosure is mainly composed of two parts: an audio to expression converter and an expression to facial renderer.

[0054] During the preprocessing process, the embodiment of the present disclosure extracts facial prior information from the reference video data, including but not limited to: reference facial shape information, reference expression information, and reference head posture information. At the same time, during the processing, audio features are also extracted from the target audio data. Optionally, a three-dimensional deformable face model (3D Morphable Face Model, 3DMM) can be used as a three-dimensional morphological model to extract more accurate facial shape information, posture information and expression information (such as the reference facial shape information, reference expression information and reference head posture information and other facial prior information mentioned later) from the reference video data to achieve accurate mapping from video to expression sequence. Compared with the traditional method that requires marking key points in the facial area, the three-dimensional morphological model according to the embodiment of the present disclosure helps the neural network to learn the movements and changes of facial components (for example, the mouth) more effectively and accurately.

[0055] In the audio-to-expression conversion stage, the facial prior information extracted from the preprocessing will be input into the audio-to-expression converter to generate the target expression sequence information, so as to better preserve and reflect the identity characteristics of the virtual human body and subtle changes in emotional expression.

[0056] In the expression-to-video-frame reconstruction stage, the target expression sequence information will be used as the input of the expression-to-face renderer, so that the renderer can use the improved facial identity alignment module to accurately generate the motion video images of the target facial components (for example, the mouth, eyes, ears, eyebrows and other components of the target face). Optionally, the expression-to-face renderer can use a facial individual alignment module (FIA), which only requires one reference video frame to obtain the target video frame that conforms to the facial personality characteristics of the virtual character when pronouncing the pronunciation. Compared with the traditional method, the expression-to-face renderer according to the embodiment of the present disclosure improves the personalized expression of the target video frame, reduces the computing cost, and helps to improve the reasoning speed.

[0057] The disclosed embodiments can generate fine details of facial components (used to represent virtual organs) in real time without relying on complex and inefficient computing modules, significantly improving the fineness of video images and the efficiency of generating video data. Although the following description of this disclosure mainly uses the mouth as an example, those skilled in the art should understand that the disclosed solutions can be applied to the generation of any facial organs related to the expression and emotional communication. Of course, this disclosure is not limited to this.

[0058] FIG3 shows a flowchart of a method 30 for processing audio data according to an embodiment of the present disclosure.

[0059] Method 30 may be performed at a terminal device or a server (such as the terminal 120 or the server 110 described in FIG1 ). Method 30 includes the following operations S301 to S304 . Of course, method 30 may also include more or fewer operations, and the present disclosure is not limited thereto.

[0060] Optionally, operations S301 and S302 are operations in the pre-processing stage, which can be performed by any facial information extraction model. Operation S303 can be performed in the conversion stage of audio information to expression information, and operation S304 can be performed in the expression to video frame reconstruction stage.

[0061] In operation S301 , facial prior information of a virtual object is determined based on reference video data.

[0062] Optionally, the video content involved in the reference video data includes a virtual object, which is an object with a face and can pronounce words, such as a real person or animal, a cartoon character, a virtual three-dimensional model, etc. The reference video data can be obtained by capturing a real scene using a video capture device, or can be generated by software.

[0063] The reference video data may be input video data such as that shown in FIG2 , in which a virtual object says, "It's a nice day today." The video data includes multiple reference video frames, each of which is an image. The image shows the head of the virtual object.

[0064] The virtual object's facial prior information refers to accurate facial information used to identify the virtual object's face and is used to reflect the virtual object's facial features. Optionally, the virtual object's facial prior information includes one or more of the following: reference facial shape information, reference expression information, and reference head pose information. Of course, the present disclosure is not limited to this.

[0065] For example, the reference facial shape information can be information used to characterize the facial shape features of a virtual object. Alternatively, the reference facial shape information can be extracted from any reference video frame, which may include the texture and 3D shape information of the virtual object's face. Alternatively, based on the reference facial shape information, an expressionless 3D facial avatar of the virtual object can be obtained by directly performing an affine transformation on a universal 3D standard virtual object. A 3D standard virtual object, also known as a white model virtual object, is a standardized model used to simulate human facial expressions and movements. These models typically have a uniform three-dimensional shape and structure and are used to represent the basic appearance of a virtual object. Of course, the present disclosure is not limited to this.

[0066] For example, the reference expression information can be any information used to characterize the expression characteristics of the virtual object, which is used to reflect the expression characteristics of the virtual object under different language expressions. The reference expression information can be extracted from each reference video frame to obtain the reference expression feature vector β corresponding to each reference video frame i. i The reference expression information may refer to a set of reference expression feature vectors corresponding to all reference video frames. Alternatively, based on the reference expression feature vectors and the reference facial shape information, a 3D facial avatar of the virtual object with a specific expression in a certain frame may be obtained by directly performing an affine transformation on the universal 3D standard virtual object. Of course, the present disclosure is not limited to this.

[0067] For example, the reference head pose information is any information used to characterize the posture characteristics of the head of a virtual object. The reference head pose information is extracted from each reference video frame to obtain a reference head pose feature vector corresponding to each reference video frame i, and the reference head pose information can refer to a set of reference head pose feature vectors corresponding to all reference video frames. Optionally, based on the reference head pose feature vector, the universal 3D standard virtual object can be directly adjusted by rotation to achieve the head pose corresponding to the reference head pose feature vector. Of course, the present disclosure is not limited to this.

[0068] In operation S302 , target audio features are determined based on target audio data.

[0069] The audio identified by the target audio data is the audio that needs to be synthesized as an audio-virtual object, that is, the audio that needs to be emitted by the virtual object in the target video data determined in the subsequent S304. Optionally, the target audio data can be the target audio data as shown in Figure 2, where the audio corresponding to the input video data is "I love you". The target audio data may include multiple audio frames, and the target audio features refer to the sound-related features extracted from these audio frames. These features may include spectral information, time domain information, sound intensity, etc. of the audio signal. Of course, the present disclosure is not limited to this.

[0070] In operation S303 , target expression sequence information is determined based on the facial prior information and the target audio features.

[0071] Optionally, the target expression sequence information can be reference expression information corresponding to the target audio feature. The target expression sequence information is a collection of target expression feature vectors arranged in time sequence, and each target expression feature vector corresponds to a target audio frame, which is used to reflect the expression characteristics of the virtual object when speaking a specific sentence. This mapping relationship can be in the form of many-to-one, one-to-one, or one-to-many. For example, a target expression feature vector can correspond to one frame of target audio frame, or it can correspond to multiple frames of target audio frames, or even multiple target expression feature vectors can correspond to a single target audio frame. Of course, the present disclosure is not limited to this.

[0072] Alternatively, the target expression sequence information may be directly determined based solely on the reference facial shape information and reference expression information in the facial prior information, combined with the target audio features.

[0073] In operation S304 , target video data corresponding to the target audio data is determined based on the target expression sequence information and a reference frame in the reference video data.

[0074] The virtual object involved in the target video data and the virtual object involved in the reference video data in S301 are the same virtual object, except that the virtual object in the target video data emits audio corresponding to the target audio data.

[0075] Since the target expression sequence information can accurately identify the expression characteristics of the virtual object when the virtual object emits the audio corresponding to the target audio characteristics, and the reference frame can identify the relevant information of the virtual object, the target expression sequence information can be used to accurately simulate the facial condition of the virtual object emitting the audio corresponding to the target audio data based on the guidance of the reference frame in the reference video data, thereby obtaining the target video data.

[0076] Optionally, the number of target video frames in the target video data can match the number of audio frames in the target audio data. Each target video frame shows a specific facial expression of the virtual object when speaking the corresponding audio frame. Optionally, the mouth movement (i.e., mouth shape) of the virtual object when speaking the corresponding audio frame should correspond to the pronunciation of the audio frame. The consistency between the facial expression of the virtual object presented in the target video frame and its pronunciation makes the generated video more realistic and lifelike.

[0077] Optionally, the reference frame I in the reference video data r It can be any frame in the reference video data, which is used to provide the basic shape and texture of the virtual object as reference information in the expression to video frame reconstruction stage. r Different reference frames may also be used, and the present disclosure is not limited thereto.

[0078] Optionally, in S304, a source video frame can be extracted from the reference video data; the source video frame can be masked based on the target expression sequence to obtain a first video frame, wherein, in the first video frame, part of the facial components of the virtual object are replaced by black pixel blocks; and based on the first video frame and the reference frame, the facial prior information is used to determine the target video data corresponding to the target audio data.

[0079] This application does not limit the relationship between the reference frame and the source video frame. For example, the source video frame can be one or more reference frames. This application also does not limit the number of source video frames used when determining the target data. For example, it can be one source video frame or multiple source video frames.

[0080] Optionally, masking the source video frame based on the target expression sequence to obtain a first video frame includes: determining a pixel area corresponding to partial facial components of the virtual object in the source video frame based on the target expression sequence and the reference facial shape information; and modifying the pixel values ​​in the pixel area to preset values.

[0081] Optionally, the method of determining the target video data corresponding to the target audio data based on the first video frame and the reference frame using the facial prior information includes: encoding the first video frame and the reference frame using a shared weight encoder to obtain the encoding features of the first video frame and the encoding features of the reference frame; using the facial prior information and the target expression sequence, and decoding the encoding features of the first video frame and the encoding features of the reference frame using a decoder to obtain the target video data corresponding to the target audio data, wherein the target video data includes a target video frame, and the target video frame includes predicted values ​​of partial facial components of the virtual object replaced by black pixel blocks.

[0082] For example, the first video frame may be generated based on the target expression sequence information. (wherein, the first video frame The facial components in the video frame are masked, that is, the facial components in the video frame are replaced by black pixel blocks), and then based on the source video frame I s and reference frame I r , determine the target video frame The following description will be made by taking the mask of the mouth component (also referred to as the mouth mask) as an example. Those skilled in the art will appreciate that this solution can also be applied to any facial component.

[0083] Specifically, for each video frame, the facial structure related to mouth movement is masked (e.g., the lower half of the face) by using the mouth mask, and the facial variable structure that is not related to expression change or target audio frame is retained. r and a video frame including a learnable mouth mask, predicting pixel values ​​at the mouth mask using a face identity alignment module and filling the predicted pixel values ​​at the mouth mask into the target video frame to obtain the target video frame. The present disclosure is not limited thereto.

[0084] Optionally, in operation S304, a facial identity alignment operation is also performed, which uses a facial identity alignment module with a cross-attention structure to establish an effective connection between target expression related parameters (such as target expression feature vectors) and the effective facial texture of a single reference frame. Through this structure, the module can achieve more precise lip shape control. Specifically, the cross-attention structure allows the module to pay attention to the target expression and the facial texture information extracted from a single reference frame at the same time. This attention mechanism enables the module to accurately adjust the shape and movement of the lips based on the target expression feature vector and the facial texture of the reference frame. By combining these two types of information, the module can better understand the features and structure of the face and accurately control the lip movement.

[0085] The method according to the embodiment of the present disclosure can improve the accuracy and credibility of the lip movements of virtual objects, enhance the communication effect and visual expressiveness of the video. The embodiment of the present disclosure can generate fine details of facial organs in real time without relying on complex and inefficient computing modules, greatly improving the fineness of the video picture and the efficiency of generating video data.

[0086] Figure 4A shows a schematic diagram of an improved neural network model according to an embodiment of the present disclosure. Figure 4B shows a schematic diagram of the audio-to-expression conversion stage according to an embodiment of the present disclosure. Figure 4C shows a schematic diagram of the audio-to-expression converter according to an embodiment of the present disclosure. Figure 4D shows a schematic diagram of the expression-to-facial renderer according to an embodiment of the present disclosure.

[0087] As shown in FIG4A , in the pre-processing stage, operation S301 can be performed using a deep 3D facial reconstruction model to determine the facial prior information of the virtual object based on the reference video data. The facial prior information includes reference facial shape information (α1), reference expression information (β1, ..., β N ), and reference head pose information (ρ1,…,ρ N ). Given a series of video frames including the face of a virtual object, the deep 3D facial reconstruction model is able to extract 3D coefficients, including shape coefficients, expression coefficients, and pose coefficients. Optionally, the deep 3D facial reconstruction model can use a mixed loss function for weakly supervised learning, which takes into account low-level and perceptual level information for supervision. At the same time, the deep 3D facial reconstruction model can also use complementary information from different images to perform multi-image facial reconstruction, achieving shape aggregation. The deep 3D facial reconstruction model is fast, accurate, and robust, and can cope with occlusion and large posture situations. Nevertheless, the various embodiments of the present disclosure can also use other neural network models to extract facial prior information of virtual objects.

[0088] Optionally, in operation S302, a Bert model of hidden units (hereinafter referred to as HuBert model) can be used to perform self-supervised speech representation learning, and target audio features (w1, ..., w l The Hubert model uses a self-supervised learning method to predict hidden units through masks, thereby learning high-quality speech representations. However, various embodiments of the present disclosure may also use other neural network models to determine target audio features.

[0089] In the conversion stage of audio information to expression information, in operation S303, the audio to expression converter can use a reference facial shape information (α1) and N reference expression information (β1, ..., β N ), combined with l target audio features (w1,…,w l ) obtain the target expression sequence information, which includes T target expression features N, l and T can be the same value or different values, and the present disclosure is not limited thereto.

[0090] For example, each virtual object has its own unique facial and mouth structure, and each virtual object also has different speaking habits. In this case, the reference facial shape information (α1) and N reference expression information (β1,…,β N ) can reflect the personality characteristics or identity characteristics of the virtual object. Specifically, the reference facial shape information (α1) characterizes the identity of the virtual object through the texture and shape characteristics of the face. Optionally, the reference facial shape information can be associated with the natural facial size and oral cavity proportions. On the other hand, the reference expression information captures the individual's unique speaking style, which provides a difference from the standard shape for the personalized virtual object. The reference expression information includes factors such as the amplitude of mouth movement and emotional expression. By comprehensively utilizing this information, a facial expression sequence that matches the target audio can be generated more accurately, making the expression of the virtual object more vivid and natural.

[0091] As shown in FIG4A , the audio to expression converter is based on a transformer architecture, which includes an encoder and a decoder. Under such an architecture, when the facial prior information includes reference facial shape information and reference expression information, the step of determining the target expression sequence information optionally includes: performing a multi-step process on the reference facial shape information (α1), the reference expression information (β1,…,β N ) and the target audio features (w1,…,w l ,) is encoded to obtain a hidden vector (Z); and the hidden vector (Z) is decoded to determine the target expression sequence information. FIG4B further shows some details of the audio to expression converter.

[0092] By encoding reference facial shape information and reference expression information in combination with target audio features, relevant information such as facial shape and expression related to the virtual object can be accurately associated with the different audios involved in the target audio features, so that the obtained target expression sequence information can more naturally identify the facial expression of the virtual object when emitting the target audio features.

[0093] Specifically, the reference facial shape information α1 in the 3D coefficient (which will be used as the default reference facial shape information), N reference expression information (β1, ..., β N ) and l target audio features (w1,…,w l ) can be passed through a fully connected network to obtain embedding vectors. These embedding vectors are then concatenated in time order to obtain a total of 1+N+l tokens, which are input into the Multi-Head Self-Attention (MHSA) (also known as the transformer encoder) for encoding to obtain the hidden vector Z. Z=MHSA(α1,β1,…,β N ,w1,…,w l ) (1)

[0094] Expression query vector q t (initially set to zero) is combined with the hidden vector Z and fed into the Multi-Head Cross-Attention (MHCA) network (also known as the transformer decoder) for decoding to obtain T target expression features

[0095] In the reconstruction phase of expression information to video data, in operation S304, the expression to face renderer first obtains any reference video frame from the reference video data as the source video frame I. s For example, assuming that the mth target video frame is to be generated, the mth reference video frame can be extracted from the reference video data as the source video frame. Of course, the present disclosure is not limited thereto.

[0096] Then, the expression to face renderer can be based on a target expression feature vector in the target expression sequence information and source video frame I s , generate the first video frame The first video frame The facial components in are masked. Optionally, the area corresponding to the masked facial components is learnable, that is, the size and shape of the area can be associated with both the expression information and the identity information of the virtual object.

[0097] Optionally, unlike conventional methods, the first video frame is generated The process is learnable. As shown in FIG4B , the source video frame I can be selectively sMasking allows for more accurate control over the modification of facial component areas. First, based on the predicted expression feature vector, 3D vertices are estimated and then projected onto the image. Next, based on these 3D vertices, the mouth and neck areas can be drawn and filled with black pixel blocks (or pixel blocks with pixel values ​​of any preset values). That is, relative to the source video frame, the first video frame has a mask area, which belongs to the source video frame that is masked. The mask area of ​​the first video frame can be a black pixel block. In this area of ​​the source video frame, part of the facial components of the virtual object are included.

[0098] The process of generating the mask is as follows:

[0099] Where P is the projection function and τ t Indicates the translation matrix of t. C is solved for the vertices V in the face and neck region of interest xy The contour function of point V xy The set of is S.

[0100] By masking the source video frame, the facial area that needs to be predicted can be accurately identified, and this area is directly related to the target audio features emitted by the virtual object, thereby improving the pertinence and accuracy of the prediction.

[0101] First video frame and the reference frame I in the reference video data r The two images are jointly input to a shared weight encoder to process the two images to extract multi-scale features. Specifically, the shared weight encoder converts the first video frame Encoded as feature F s , and reference frame I r Encoded as feature F r .

[0102] Alternatively, as shown in FIG4A , the shared weight encoder is composed of d stacked residual blocks, so the encoder is also called a d-level encoder. At each level (residual block), the image resolution is halved while the feature dimension of the extracted is increased. This setting allows for efficient extraction of multi-scale features in the shared weight encoder. Since the weights of the encoder are shared, the first video frame can be processed in parallel. and reference frame I r Specifically, the encoder extracts features of various scales from the input image through multiple levels of processing. For the i-th level in the d-level encoder, its multi-scale features can be calculated as follows.

[0103] Among them, SE iRepresents the i-th level of the encoder. At the end of each level of processing, a representation containing features of different scales and abstract levels is obtained. These features can capture different details and structural information of the input image. Thus, for the first video frame and reference frame I r Get a set of d-size feature vectors, that is, and Since the first video frame and reference frame I r The shared encoder weights enable more efficient processing of source and reference frames, extracting their shared feature representations. This parallel computing approach not only speeds up processing but also ensures that the feature representations of the source and reference frames are compared in the same space, facilitating subsequent matching and fusion.

[0104] Next, the expression to face renderer will use the decoder including the face identity alignment module to transform the first video frame The corresponding feature F s and reference frame I r Corresponding feature F r Decode to obtain the target video frame.

[0105] As can be seen, the improved neural network inference process, by first determining the target expression sequence and then predicting the video frame based on this sequence, significantly improves the synchronization between the audio content and the facial components of the virtual object on the screen. This approach not only enhances the realism of facial movements but also accurately reflects the identity of the virtual object. While improving performance, it also ensures the efficiency and stability of the neural network, making it particularly suitable for real-time video generation.

[0106] FIG4D shows more example details of the decoder. Specifically, the decoder is composed of d stacked decoding layers, so the decoder is also called a d-level decoder. For example, the decoder includes d decoding layers, and the input of each decoding layer includes the output of the previous decoding layer, the facial prior information, and the encoded features of the reference frame. The output of each decoding layer is the predicted value of the target video frame. Assume that the input features of each level of the decoder are The output features are

[0107] As shown in Figure 4D, the 3D coefficient Before being fused to the decoder through the adapter (AdalN) module, dimension mapping is performed through a three-layer MLP. This process aims to map the 3D coefficients into a feature space compatible with the decoder network for subsequent feature fusion and generation processes. As a result, the input features of each level of the decoder are First upsample to increase the resolution to get the features feature Then through the 3D coefficient Modulation to match the shape of specific facial components, thereby better controlling the movement of facial components and the generation of facial expressions. The modulated features are compared with the features from the reference frame I r Features Aligned at the same resolution, the two are fused to control face generation and aggregate texture. The above process can be expressed as follows.

[0108] FIA i It is a level (or layer) decoding layer in the decoder that includes the face identity alignment module.

[0109] In addition, in order to avoid unnecessary background in the target video frame and ensure that the final result is more realistic and accurate, a fusion strategy can be further adopted. Specifically, in some embodiments of the present disclosure, the output of the last decoding layer of the decoder can be fused with the output of the first decoding layer of the decoder to obtain the final output. This fusion strategy aims to retain only the pixels corresponding to the predicted facial components while eliminating background interference, making the generated image clearer and focused on facial features. The process can be expressed as follows.

[0110] Figure 5A shows a flow chart of a method 50 for training a neural network according to an embodiment of the present disclosure, which can be used to train the improved neural network model shown in Figures 4A to 4C. Figure 5B shows a schematic diagram of training an audio to expression converter according to an embodiment of the present disclosure.

[0111] Method 50 may be performed at a terminal device or a server (such as the terminal 120 or the server 110 described in FIG1 ). Method 50 includes the following operations S501 to S506 . Of course, method 50 may also include more or fewer operations, and the present disclosure is not limited thereto.

[0112] In operation S501, sample facial prior information of a virtual object is determined based on sample video data. In operation S502, sample target audio features are determined based on sample audio data corresponding to the sample video data.

[0113] For example, assuming that the facial prior information obtained based on the sample video data includes: sample facial shape information (α1), sample target expression sequence information (e.g., sample expression information β1, ..., β N ), and sample head pose information (ρ1,…,ρ N). These facial prior information can be directly obtained through preprocessing. Such facial prior information can be used as the true value of the facial prior information of the virtual object. The preprocessing process has been described in detail above and will not be repeated in this disclosure.

[0114] Next, referring to the preprocessing process described in detail above, the sample audio data corresponding to the sample video data can be extracted to directly determine the sample audio features corresponding to the sample audio data. Here, the sample audio data corresponding to the sample video data refers to, for example, the audio data containing the speech "The weather is so nice today" when a virtual person in the sample video data is saying "The weather is so nice today."

[0115] Next, the training process of the audio-to-expression converter is described based on operations S503 and S504. Those skilled in the art should understand that the present disclosure is not limited thereto.

[0116] In operation S503, based on the sample facial prior information and the sample target audio features, the audio-to-expression converter is used to determine the predicted value of the target sample expression sequence information. In operation S504, based on the predicted value of the target sample expression sequence information, the audio-to-expression converter is trained using a first loss function. Optionally, the sample facial prior information may include the true value of the sample target expression sequence information, and operation S503 includes: masking the true value of the facial prior information of the virtual object to obtain the masked facial prior information of the virtual object; based on the masked facial prior information of the virtual object and the sample audio features, using the audio-to-expression converter to determine the predicted value of the facial prior information of the sample target expression sequence information of the virtual object. Of course, the present disclosure is not limited to this.

[0117] Optionally, in order to determine the predicted value of the sample target expression sequence information of the virtual object, it is first necessary to mask its true value to obtain the masked sample target expression sequence information of the virtual object. This process involves hiding or covering part of the information in the true value to generate a masked version. The masked information can be covered according to specific conditions or rules, for example, hiding specific reference expression information β t Then, based on these masked sample target expression sequence information (that is, excluding the specific reference expression information β t The facial prior information) and sample audio features can be used to predict the predicted value of the mask part of the sample target expression sequence information of the virtual object using the audio-to-expression converter in training (for example, ). The training goal of the audio-to-expression converter is to make the true value of the facial prior information β t and predicted value Therefore, the parameters of each neuron in the audio-to-expression converter can be continuously adjusted to make the first loss function reach an extreme value. Of course, the present disclosure is not limited to this.

[0118] Optionally, as shown in FIG5B , the first loss function for training the audio-to-expression converter may include, but is not limited to, mean square error (MSE) loss and 3D vertex loss.

[0119] For example, the mean square error (MSE) loss is used to calculate the error between the predicted value and the true value of the expression feature vector. Assume that the predicted expression feature vector is The true expression feature vector is β t , then the MSE loss can be expressed as:

[0120] Optionally, the 3D vertex loss is calculated by combining the predicted expression feature vectors The predicted positions of 3D vertices are calculated using the facial shape information α1 and the reference head pose information ρ. These vertices are key points (e.g., mouth corners) selected from a facial region (e.g., mouth) to evaluate the distance between the true and predicted values ​​of the key points of that facial region. By minimizing the difference between the predicted and true values ​​of the key point positions, the generated 3D vertices can be optionally aligned with the mouth key points. Let V represent the 3D vertices calculated from the coefficients. 3D vertex loss L V It can be described as follows:

[0121] Thus, the first loss function can be expressed as L a2e =L MSE +0.1*C V 0.1 is a hyperparameter, which can also be set to any other value. The present disclosure is not limited to this.

[0122] Next, the training process of the expression-to-face renderer is described based on operations S505 and S506. Those skilled in the art should understand that the present disclosure is not limited to this. The training process of the audio-to-expression converter and the training process of the expression-to-face renderer can be independent of each other. Specifically, as described in detail below, the input data used in the training process of the expression-to-face renderer can be the sample facial prior information (which optionally includes the true value of the sample target expression sequence information), the sample reference frame, and the sample source video frame. These input data can all be obtained in the preprocessing process (i.e., operations S501 and S502) without being processed by the audio-to-expression converter. Of course, in other embodiments, the audio-to-expression converter and the expression-to-face renderer can also be trained jointly, and the present disclosure is not limited to this.

[0123] Specifically, the sample facial prior information includes the true value of the sample target expression sequence information. In operation S505, a sample reference frame and a sample source video frame are determined based on the sample video data. Next, in operation S506, the expression-to-face renderer is trained using a second loss function based on the sample facial prior information, the sample reference frame, and the sample source video frame.

[0124] Optionally, in operation S506, the sample source video frame is masked based on the true value of the target expression sequence to obtain a masked sample source video frame; based on the sample facial prior information and the true values ​​of the sample reference frame and the sample target expression sequence information, the expression-to-face renderer is used to determine the predicted value of the sample source video frame for the masked sample source video frame; and based on the predicted value and the true value of the sample source video frame, the expression-to-face renderer is trained using the second loss function; wherein the second loss function includes a combination of one or more of the following items: pixel loss, perceptual loss, adversarial loss and local pixel loss.

[0125] Optionally, the sample face prior information includes 3D coefficients Referring to the method detailed above, the sample source video frame can first be masked (i.e., a masked sample source video frame is obtained, in which facial components are replaced by black pixel blocks). The predicted value of the sample source video frame is then determined based on these 3D coefficients and the sample reference frame. The training goal of the expression-to-face renderer is to ensure that the true value of the sample source video frame is sufficiently close to the predicted value. Therefore, the parameters of each neuron in the expression-to-face renderer can be continuously adjusted to maximize the second loss function. Of course, the present disclosure is not limited to this.

[0126] Optionally, the second loss function used to train the expression-to-video renderer may adopt a combination of multiple loss functions to constrain the synchronization between facial components and expressions to improve the visual quality of the generated video frames.

[0127] For example, pixel loss can be used to measure the pixel-level difference between the generated image and the target image. and the true value I s The pixel difference between the two improves the accuracy and clarity of the generated video frames. As an example, the pixel loss L1 can be calculated as: Of course, the present disclosure is not limited thereto.

[0128] For example, perceptual loss is based on the features extracted by deep learning models (such as VGG network) (i.e., perceptual information obtained from images) to measure the prediction value of generating sample source video frames. and the true value I s The perceptual loss helps improve the realism and naturalness of the generated image to better match the target expression. The perceptual loss can be calculated as:

[0129] For example, the adversarial loss evaluates the predicted value of a sample source video frame via a discriminator network in a generative adversarial network (GAN) framework. The generator network aims to fool the discriminator into being unable to distinguish the predicted values ​​of the sample source video frames. and the true value I s , thereby improving the authenticity and realism of the image. The adversarial loss can be calculated as: where D(·) represents the discriminator.

[0130] For example, local pixel loss focuses on enhancing pixel-level details in specific areas, such as the details of the mouth. By introducing additional loss terms for specific facial areas, the generated image is made clearer and more realistic in details such as the mouth. The local pixel loss can be calculated as: Where M' represents the binary mask of the target facial region.

[0131] The second loss function is obtained by weighted summing the above loss functions. The above loss functions act together in the training process of the expression to video renderer to ensure that the generated video can accurately reflect the target expression while maintaining a high level of visual quality and details. The second loss function can be calculated as L e2f =λ1*L1+λ2*L2+λ3*L3+λ4*L4, where λ1=1, λ2=1, λ3=0.1, and λ4=1. λ1 to λ4 are all hyperparameters, which can be set to any other values ​​in practical applications, and the present disclosure is not limited thereto.

[0132] During the training process of the improved neural network model, an improved first loss function and a modified second loss function are optionally introduced. These two loss functions optimize different aspects of the neural network's performance by adjusting the neural network parameters. The first and second loss functions enable the neural network to more effectively learn and adapt to the characteristics of the input data, optimizing multiple aspects during the training process. This loss function design, which comprehensively considers different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby enhancing its performance in practical applications.

[0133] An embodiment of the present disclosure also provides a device for processing audio data, which includes: a first preprocessing module for determining facial prior information of a virtual object based on reference video data; a second preprocessing module for determining target audio features based on target audio data; and an audio-to-expression converter module for determining target expression sequence information based on the facial prior information and the target audio features; and an expression-to-facial renderer for determining target video data corresponding to the target audio data based on the target expression sequence information and a reference frame in the reference video data.

[0134] In a possible implementation, the facial prior information includes one or more of the following: reference facial shape information, reference expression information, and reference head pose information.

[0135] In a possible implementation, when the facial prior information includes the reference facial shape information and the reference expression information, the audio-to-expression converter module is further configured to:

[0136] Encoding the reference facial shape information, the reference expression information, and the target audio feature to obtain a latent vector; and

[0137] The hidden vector is decoded to determine the target expression sequence information.

[0138] In one possible implementation, the expression-to-face renderer is further configured to:

[0139] extracting source video frames from the reference video data;

[0140] Masking the source video frame based on the target expression sequence to obtain a first video frame, wherein in the first video frame, part of the facial components of the virtual object are replaced by black pixel blocks; and

[0141] Based on the first video frame and the reference frame, target video data corresponding to the target audio data is determined using the facial prior information.

[0142] In one possible implementation, the expression-to-face renderer is further configured to:

[0143] Determining, based on the target expression sequence and the reference facial shape information, a pixel area corresponding to a partial facial component of the virtual object in the source video frame; and

[0144] The pixel values ​​in the pixel area are modified to preset values ​​to obtain the first video frame.

[0145] In one possible implementation, the expression-to-face renderer is further configured to:

[0146] Encoding the first video frame and the reference frame using a weight-sharing encoder to obtain a first encoding feature of the first video frame and a second encoding feature of the reference frame;

[0147] Utilizing the facial prior information and the target expression sequence, a decoder is used to decode the first encoding feature and the second encoding feature to obtain target video data corresponding to the target audio data, wherein the target video data includes a target video frame, and an area of ​​black pixel blocks in the target video frame is replaced by a predicted value of a partial facial component of the virtual object.

[0148] In one possible implementation, the decoder includes multiple cascaded decoding layers, the input of each decoding layer includes the output of the previous decoding layer, the facial prior information and the encoding features of the reference frame, and the output of each decoding layer is the predicted value of the target video frame.

[0149] It can be seen that by determining the facial prior information of the virtual object in the reference video data, the facial features of the virtual object can be accurately identified, and thus the target expression sequence information that conforms to the facial features of the virtual object and naturally integrates the facial features can be determined based on the facial prior information and the target audio features. When the target expression sequence information and the reference frame are used to determine the target video data, the synchronization accuracy between the audio content in the target video data and the facial components (such as the mouth) of the virtual object on the screen can be significantly improved, the real texture of the mouth movement can be enhanced, and the identity characteristics of the virtual object can be accurately reflected. The expression differences between individuals can be more accurately captured and reproduced. Moreover, while improving the performance of the system, the overall operating efficiency and stability are also ensured, making it particularly suitable for real-time video generation.

[0150] The present disclosure also provides a device for training a neural network, wherein the neural network includes an audio-to-expression converter and an expression-to-face renderer, and the device includes:

[0151] A first determining module is used to determine sample facial prior information of a virtual object based on sample video data;

[0152] a second determining module, configured to determine a sample target audio feature based on the sample audio data corresponding to the sample video data;

[0153] A third determination module is configured to determine a predicted value of target sample expression sequence information using the audio-to-expression converter based on sample facial prior information and sample target audio features;

[0154] A first training module is configured to train the audio-to-expression converter using a first loss function based on a predicted value of the target sample expression sequence information;

[0155] a fourth determining module, configured to determine a sample reference frame and a sample source video frame based on the sample video data; and

[0156] The second training module is used to train the expression-to-face renderer based on the sample facial prior information, the sample reference frame, and the sample source video frame using a second loss function.

[0157] In a possible implementation, the sample facial prior information includes a true value of sample target expression sequence information, and the third determination module is further configured to:

[0158] Masking the true value of the facial prior information of the virtual object to obtain the masked facial prior information of the virtual object;

[0159] Based on the masked facial prior information of the virtual object and the sample audio features, the audio-to-expression converter is used to determine a predicted value of the sample target expression sequence information of the virtual object.

[0160] In a possible implementation, the sample facial prior information includes a true value of sample target expression sequence information, and the second training module is further configured to:

[0161] masking the sample source video frame based on a true value of the sample target expression sequence to obtain a masked sample source video frame;

[0162] Determining a prediction value of the sample source video frame using the expression-to-face renderer for the masked sample source video frame based on the sample facial prior information and the sample reference frame; and

[0163] Based on the predicted value and the true value of the sample source video frame, the expression to face renderer is trained using the second loss function;

[0164] The second loss function includes one or more of the following: pixel loss, perceptual loss, adversarial loss, and local pixel loss.

[0165] As can be seen, during the training process of the improved neural network model, an improved first loss function and a modified second loss function are optionally introduced. These two loss functions optimize different aspects of the neural network's characteristics, respectively. By adjusting the neural network parameters, the performance of the trained neural network is improved. The first and second loss functions enable the neural network to more effectively learn and adapt to the characteristics of the input data, and optimize multiple aspects during the training process. This loss function design, which comprehensively considers different characteristics, helps improve the generalization ability and training efficiency of the neural network, thereby improving its performance in practical applications.

[0166] As shown in FIG6 , the predicted result of the video frame obtained according to the solution of the embodiment of the present disclosure is very close to the real result of the video frame, and the degree of restoration of the detailed texture is much higher than that of the traditional solution.

[0167] According to another aspect of the present disclosure, an electronic device is provided for implementing the method according to an embodiment of the present disclosure. FIG7 shows a schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure.

[0168] As shown in Figure 8, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores computer-readable code, which, when executed by the one or more processors 2010, may execute the method described above.

[0169] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The methods, operations, and logic block diagrams disclosed in the embodiments of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor, and may be an X86 architecture or an ARM architecture.

[0170] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0171] For example, the method or apparatus according to an embodiment of the present disclosure may also be implemented with the aid of the architecture of the computing device 3000 shown in FIG9 . As shown in FIG8 , the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, and the like. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method provided by the present disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, the architecture shown in FIG8 is merely exemplary. When implementing different devices, one or more components in the computing device shown in FIG8 may be omitted according to actual needs.

[0172] According to another aspect of the present disclosure, a computer-readable storage medium is further provided. FIG9 shows a schematic diagram of a storage medium 4000 according to the present disclosure.

[0173] As shown in FIG9 , the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0174] In addition, an embodiment of the present application further provides a storage medium, which is used to store a computer program, and the computer program is used to execute the method provided by the above embodiment.

[0175] An embodiment of the present application further provides a computer program product including a computer program, which, when executed on a computer, enables the computer to execute the method provided in the above embodiment.

[0176] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0177] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0178] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will appreciate that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for processing audio data, the method being performed by a computer device, the method comprising: Determining facial prior information of a virtual object based on reference video data; determining target audio features based on the target audio data; Determining target expression sequence information based on the facial prior information and the target audio features; as well as Target video data corresponding to the target audio data is determined based on the target expression sequence information and a reference frame in the reference video data.

2. The method for processing audio data according to claim 1, wherein the facial prior information comprises one or more of the following: reference facial shape information, reference expression information, and reference head pose information.

3. The method for processing audio data according to claim 2, wherein when the facial prior information includes the reference facial shape information and the reference expression information, determining the target expression sequence information based on the facial prior information and the target audio features comprises: Encoding the reference facial shape information, the reference expression information, and the target audio feature to obtain a latent vector; as well as The hidden vector is decoded to determine the target expression sequence information.

4. The method for processing audio data according to claim 2, wherein determining the target video data corresponding to the target audio data based on the target expression sequence information and the reference frame in the reference video data comprises: extracting source video frames from the reference video data; Masking the source video frame based on the target expression sequence to obtain a first video frame, wherein in the first video frame, part of the facial components of the virtual object are replaced by black pixel blocks; as well as Based on the first video frame and the reference frame, target video data corresponding to the target audio data is determined using the facial prior information.

5. The method for processing audio data according to claim 4, wherein masking the source video frame based on the target expression sequence to obtain the first video frame comprises: Determining, based on the target expression sequence and the reference facial shape information, a pixel area corresponding to a partial facial component of the virtual object in the source video frame; as well as The pixel values ​​in the pixel area are modified to preset values ​​to obtain the first video frame.

6. The method for processing audio data according to claim 4, wherein determining the target video data corresponding to the target audio data based on the first video frame and the reference frame and utilizing the facial prior information comprises: Encoding the first video frame and the reference frame using a weight-sharing encoder to obtain a first encoding feature of the first video frame and a second encoding feature of the reference frame; Utilizing the facial prior information and the target expression sequence, a decoder is used to decode the first encoding feature and the second encoding feature to obtain target video data corresponding to the target audio data, wherein the target video data includes a target video frame, and an area of ​​black pixel blocks in the target video frame is replaced by a predicted value of a partial facial component of the virtual object.

7. The method for processing audio data as described in claim 6, wherein the decoder includes multiple cascaded decoding layers, the input of each decoding layer includes the output of the previous decoding layer, the facial prior information and the encoded features of the reference frame, and the output of each decoding layer is a predicted value of the target video frame.

8. A method for training a neural network, the neural network comprising an audio-to-expression converter and an expression-to-face renderer, the method being performed by a computer device, the method comprising: Determining sample facial prior information of a virtual object based on the sample video data; determining a sample target audio feature based on the sample audio data corresponding to the sample video data; Determining a predicted value of target sample expression sequence information using the audio-to-expression converter based on sample facial prior information and sample target audio features; Based on the predicted value of the target sample expression sequence information, the audio to expression converter is trained using a first loss function; Determining a sample reference frame and a sample source video frame based on the sample video data; as well as The expression-to-face renderer is trained based on the sample facial prior information, the sample reference frame, and the sample source video frame using a second loss function.

9. The method for training a neural network according to claim 8, wherein the sample facial prior information includes a true value of the sample target expression sequence information, and the determining the predicted value of the sample target expression sequence information using the audio-to-expression converter comprises: Masking the true value of the facial prior information of the virtual object to obtain the masked facial prior information of the virtual object; Based on the masked facial prior information of the virtual object and the sample audio features, the audio-to-expression converter is used to determine a predicted value of the sample target expression sequence information of the virtual object.

10. The method for training a neural network according to claim 8, wherein the sample facial prior information comprises a true value of sample target expression sequence information, and the training of the expression-to-face renderer comprises: masking the sample source video frame based on a true value of the sample target expression sequence to obtain a masked sample source video frame; determining, for the masked sample source video frame, a prediction value of the sample source video frame using the expression-to-face renderer based on the sample facial prior information and the sample reference frame; as well as Based on the predicted value and the true value of the sample source video frame, the expression to face renderer is trained using the second loss function; The second loss function includes one or more of the following: pixel loss, perceptual loss, adversarial loss, and local pixel loss.

11. A device for processing audio data, the device comprising: The first pre-processing module is used to determine the facial prior information of the virtual object based on the reference video data: A second preprocessing module is used to determine target audio features based on the target audio data; an audio-to-expression converter module, configured to determine target expression sequence information based on the facial prior information and the target audio features; An expression-to-face renderer is configured to determine target video data corresponding to the target audio data based on the target expression sequence information and a reference frame in the reference video data.

12. A device for training a neural network, the neural network comprising an audio-to-expression converter and an expression-to-face renderer, the device comprising: A first determining module is used to determine sample facial prior information of a virtual object based on sample video data; a second determining module, configured to determine a sample target audio feature based on the sample audio data corresponding to the sample video data; A third determination module is configured to determine a predicted value of target sample expression sequence information using the audio-to-expression converter based on sample facial prior information and sample target audio features; A first training module is configured to train the audio-to-expression converter using a first loss function based on a predicted value of the target sample expression sequence information; a fourth determining module, configured to determine a sample reference frame and a sample source video frame based on the sample video data; as well as The second training module is used to train the expression-to-face renderer based on the sample facial prior information, the sample reference frame, and the sample source video frame using a second loss function.

13. A computer device, comprising: Processor, communication interface, memory and communication bus; Wherein, the processor, the communication interface and the memory complete communication with each other through the communication bus; the communication interface is an interface of the communication module; The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to call a computer program in a memory to execute the method according to any one of claims 1 to 7, or to execute the method according to any one of claims 8 to 10.

14. A storage medium for storing a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 7, or execute the method according to any one of claims 8 to 10.

15. A computer program product comprising a computer program, which, when run on a computer, enables the computer to execute the method according to any one of claims 1 to 7, or the method according to any one of claims 8 to 10.

Images