Data processing method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides a data processing method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: performing first coding on a first audio to obtain a first content feature, and performing second coding on the first audio to obtain a first emotion feature; performing fusion processing on the first content feature and the first emotion feature to obtain a first fusion feature; performing first decoding on the first fusion feature and the first emotional feature to obtain first emotional data and first mouth shape data, and performing second decoding on the first emotional feature to obtain second emotional data; and generating first facial animation data matched with the first audio based on the first emotion data, the second emotion data and the first mouth shape data. According to the method and the device, the naturalness and the strong emotional property of the generated facial animation under different emotional expressions can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] In recent years, with the rapid development of virtual reality, digital humans, and human-computer interaction technologies, 3D facial animation generation technology has been widely used in various fields such as virtual anchors, game characters, and intelligent customer service. Among related technologies, 3D facial animation generation technology mainly relies on audio-driven methods, and its technical approaches include phoneme recognition-based driving methods, lip-shape mapping-based expression reconstruction methods, and end-to-end methods that utilize deep learning to map speech to facial movements. Summary of the Invention

[0003] This application provides a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product that can improve the naturalness and strong emotionality of generated facial animations under different emotional expressions.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a data processing method, including:

[0006] The first audio is encoded in a first way to obtain a first content feature, and the first audio is encoded in a second way to obtain a first emotion feature;

[0007] The first content feature and the first emotion feature are fused together to obtain the first fused feature;

[0008] The first fusion feature and the first emotion feature are first decoded to obtain first emotion data and first lip shape data, and the first emotion feature is second decoded to obtain second emotion data.

[0009] Based on the first emotion data, the second emotion data, and the first lip shape data, first facial animation data matching the first audio is generated.

[0010] This application provides a data processing apparatus, including:

[0011] The encoding module is used to perform a first encoding on the first audio to obtain a first content feature, and to perform a second encoding on the first audio to obtain a first emotion feature;

[0012] The fusion module is used to fuse the first content feature and the first emotion feature to obtain a first fused feature;

[0013] The decoding module is used to perform a first decoding on the first fused feature and the first emotion feature to obtain first emotion data and first lip shape data, and to perform a second decoding on the first emotion feature to obtain second emotion data.

[0014] The generation module is used to generate first facial animation data that matches the first audio based on the first emotion data, the second emotion data, and the first lip shape data.

[0015] In the above scheme, the first fusion feature is a first matrix, the first emotion feature is a second matrix, and the decoding module is further configured to: for each first element in the first matrix, determine the similarity between the second element in the second matrix that has the same position as the first element and the first element; for each second element in the second matrix, multiply the similarity corresponding to the second element by the second element to obtain the multiplication result corresponding to the second element, and generate a third matrix based on the multiplication result corresponding to each second element; merge the first matrix and the third matrix to obtain a fourth matrix, and segment the fourth matrix to obtain the first emotion data and the first lip shape data.

[0016] In the above scheme, the generation module is further configured to perform full connection processing on the first emotion data and the second emotion data to obtain the third emotion data; and to perform fusion processing on the third emotion data and the first lip shape data to obtain the first facial animation data.

[0017] In the above scheme, the data processing method is executed by calling the first model. The device further includes a training module, which is used to forward propagate the second audio in the first model to obtain the second facial animation data; determine the first loss based on the second facial animation data and the reference animation data corresponding to the second audio; and update the parameters of the first model based on the first loss to obtain the trained first model.

[0018] In the above scheme, the training module is further configured to: encode the second audio using the first model to obtain a second content feature; encode the second audio using the first model to obtain a second emotion feature; fuse the second content feature and the second emotion feature using the first model to obtain a second fused feature; decode the second fused feature and the second emotion feature using the first model to obtain third emotion data and second lip-sync data; decode the second emotion feature using the first model to obtain fourth emotion data; and generate second facial animation data using the first model based on the third emotion data, the fourth emotion data, and the second lip-sync data.

[0019] In the above scheme, the training module is further used to perform type prediction on the fourth emotion data to obtain a first predicted emotion type; and to determine a second loss based on the first predicted emotion type and the reference emotion type corresponding to the second audio.

[0020] In the above scheme, the training module is further used to fuse the first loss and the second loss to obtain a third loss; based on the third loss, the parameters of the first model are updated to obtain a trained first model.

[0021] This application provides an electronic device, including:

[0022] Memory is used to store executable instructions or computer programs.

[0023] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method provided in the embodiments of this application.

[0024] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the data processing method provided in this application.

[0025] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application.

[0026] The embodiments of this application have the following beneficial effects:

[0027] First, the first audio is encoded by mapping it to a content space to obtain a first content feature. Simultaneously, the first audio is encoded by mapping it to an emotion space to obtain a first emotion feature, thus acquiring the audio's content and emotional information. Next, the first content feature and the first emotion feature are fused to obtain a first fused feature, providing a more comprehensive audio feature representation. Then, the first fused feature and the first emotion feature are decoded to obtain first emotion data indicating the overall emotional expression and first lip-sync data indicating the corresponding audio content. Afterward, the first emotion feature is decoded a second time to obtain second emotion data indicating the specific emotional expression. Finally, the first emotion data, second emotion data, and first lip-sync data are used together to generate facial animation data matching the first audio, ensuring that the animation maintains the content and emotion of the first audio, thereby ensuring that the generated facial animation is more natural and emotionally resonant under different emotional expressions. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the architecture of the data processing system 100 provided in an embodiment of this application;

[0029] Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application;

[0030] Figure 3 This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0031] Figure 4 This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0032] Figure 5 These are schematic diagrams illustrating different dimensions of facial animation effects provided in the embodiments of this application;

[0033] Figure 6 This is a specific network architecture diagram provided in the embodiments of this application;

[0034] Figure 7 This is a model architecture diagram of the classifier provided in this application. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0038] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0039] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0041] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0042] 1) BlendShape Animation: BlendShape is a technique in computer graphics used to control and animate 3D models. It enables dynamic deformation by subtracting a model (usually a static model) from multiple target shapes, allowing the model to smoothly transition to the target shapes.

[0043] 2) 3D face animation: 3D face animation is a computer graphics technology that uses 3D modeling and animation techniques to simulate human facial expressions and movements. It is commonly used in film, games, virtual reality (VR), augmented reality (AR) and animation production, enabling virtual characters to display expressions and movements similar to those of real humans.

[0044] In recent years, with the rapid development of virtual reality, digital humans, and human-computer interaction technologies, 3D facial animation generation technology has been widely used in various fields such as virtual anchors, game characters, and intelligent customer service. Among related technologies, 3D facial animation generation technology mainly relies on audio-driven methods, and the main technical routes cover phoneme recognition-based driving methods, lip-shape mapping-based expression reconstruction methods, and end-to-end methods that utilize deep learning to achieve speech-to-facial motion mapping.

[0045] Phoneme-driven methods convert speech signals into phoneme sequences and match them with predefined lip-shape correspondence tables to control lip-shape changes in facial models. For example, Hidden Markov Models (HMMs) or Recurrent Neural Networks (RNNs) can be used to achieve automatic speech-to-lip-shape sequence conversion. However, most of these methods focus on mouth movements, neglecting facial expression changes closely related to the emotional state of speech, such as the overall coordinated movements of eye contact, eyebrows, and facial muscles. This results in facial animations that are significantly lacking in emotional expression, exhibit strong regularity, and lack naturalness overall.

[0046] Furthermore, to enhance the realism of facial expressions, some scholars have proposed multimodal methods that jointly train speech signals and video images, such as neural network models that combine visual facial landmark regression with speech feature embedding, thereby improving lip-sync. However, this method still has the following prominent problems:

[0047] (1) Lack of emotional driving mechanism. Related technologies only focus on the correspondence between speech and lip movements, failing to fully explore and utilize the rich emotional information contained in speech, resulting in a lack of emotional dimension in the generated facial animation, making the expressions appear stiff and unnatural, and failing to meet users' expectations for emotional expression.

[0048] (2) Insufficient animation expressiveness. Due to the lack of effective participation of emotional components, the facial expressions of the characters cannot accurately convey real emotions such as "happiness, sadness, and anger" in terms of semantic expression. This greatly weakens the user's immersive experience and interactive effect, and limits its application value in emotional interaction scenarios.

[0049] (3) Lack of versatility and adaptability. In related technologies, the models usually have high adaptability to specific 3D models, but when faced with different model inputs, problems such as misaligned mouth shapes or unnatural performance are easy to occur, exposing their lack of robustness.

[0050] In view of this, embodiments of this application provide a data processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the naturalness and strong emotionality of generated facial animations under different emotional expressions. The electronic device provided in embodiments of this application can be implemented as a server, or implemented collaboratively by a server and a terminal. The following description uses an example of a data processing method provided in embodiments of this application implemented collaboratively by a server and a terminal.

[0051] For example, see Figure 1 , Figure 1 This is a schematic diagram of the architecture of the data processing system 100 provided in this application embodiment. To support a data processing application, such as... Figure 1As shown, the data processing system 100 includes: server 200, network 300, and terminal 400. Terminal 400 is connected to server 200 through network 300. Network 300 can be a local area network (LAN), a wide area network (WAN), or a combination of both.

[0052] In some embodiments, a user inputs a first audio signal through a terminal 400 and transmits the first audio signal to a server 200 via a network 300; then, the server 200 generates facial animation data matching the first audio signal; finally, the server 200 transmits the facial animation data matching the first audio signal to the terminal 400 via the network 300 and displays it on the terminal 400.

[0053] The data processing method provided in this application can be applied to various scenarios that require facial animation generation, such as the following scenarios:

[0054] 1) In film or animation production, an audio clip (e.g., a character's dialogue) undergoes a first encoding to extract content features, such as rhythm, pitch, and volume. Simultaneously, the same audio clip undergoes a second encoding to analyze its emotional characteristics, such as happiness, sadness, and anger. Next, the extracted content and emotional features are fused using relevant algorithms (e.g., neural networks) to obtain a more comprehensive first fused feature. Subsequently, the first fused feature is first decoded to obtain first emotion data and first lip-sync data. The first emotion data guides the emotional expression of facial expressions, while the first lip-sync data synchronizes with the character's mouth movements. Then, the first emotion feature is second decoded to obtain more detailed second emotion data, which can represent subtle changes in emotion, such as the transition from happiness to surprise. Finally, based on the first emotion data, second emotion data, and first lip-sync data, facial animation data matching the original audio is generated, ensuring a more natural and realistic facial animation.

[0055] 2) In the interactive feedback scenario of the virtual assistant, the user makes a request to the virtual assistant and inputs the audio of the request into the encoding module for first and second encoding, analyzing the user's voice content and emotional state; then, through feature fusion, the user's needs and emotions are parsed and understood, and corresponding feedback is generated based on the fused features, such as answering the user's question, taking the user's emotions into account when generating the answer; after that, emotion data and lip-sync data are decoded, and facial animation data matching the audio is generated based on the animation, so that the virtual assistant can display the corresponding facial animation when giving an answer, thereby providing a richer and more natural interactive experience.

[0056] 3) In the game scene, game characters can simulate dialogue interactions in specific scenarios according to the needs of the plot. The system encodes the characters' speech to obtain the content features and emotional features corresponding to the audio. Then, the content features and emotional features are fused to guide the generation of the character's facial movements and expressions. After that, the fused features and emotional features are decoded to obtain emotional data and lip-sync data. Finally, based on the emotional data and lip-sync data, facial animation data matching the audio is generated to ensure that the character's emotional changes and lip-sync are synchronized with the audio, thereby enhancing the player's immersion and emotional resonance.

[0057] Example, Figure 1 The server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0058] The structure of the electronic device provided in the embodiments of this application will be further described below. Taking the electronic device as a terminal as an example, see... Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Figure 2 The illustrated electronic device 500 includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 540.

[0059] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0061] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 550 may optionally include one or more storage devices physically located away from the processor 510.

[0062] The memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0063] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0064] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0065] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0066] Presentation module 553 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., a display screen, a speaker, etc.) associated with user interface 530;

[0067] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0068] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A data processing device 555 stored in memory 550 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: encoding module 5551, fusion module 5552, decoding module 5553, generation module 5554, and training module 5555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to the functions implemented. It should be noted that... Figure 2 For ease of explanation, all the above modules are shown at once, but this should not be interpreted as excluding the implementation of the data processing device 555, which may only include the encoding module 5551, the fusion module 5552, the decoding module 5553, and the generation module 5554. The functions of each module will be described below.

[0069] The data processing method provided in this application will be specifically described below with reference to exemplary applications and implementations of the terminal provided in the embodiments of this application.

[0070] See Figure 3 , Figure 3 This is a flowchart illustrating the data processing method provided in the embodiments of this application, which will be combined with... Figure 3 The steps shown are explained.

[0071] It should be noted that, Figure 3 The methods illustrated can be executed by various forms of computer programs running on the terminal, and are not limited to clients. For example, they can also be the operating systems, software modules, scripts, and applets mentioned above. Therefore, the client-side examples used below should not be considered as limiting the embodiments of this application. Furthermore, for ease of description, no specific distinction will be made between the terminal and the client running on the terminal below.

[0072] In step 101, the first audio is encoded to obtain the first content feature.

[0073] It should be noted that the first content feature can be obtained through traditional acoustic feature encoding methods, such as using acoustic tools to extract acoustic features like Mel Frequency Cepstral Coefficient (MFCC) and Perceptual Linear Prediction (PLP), and using these acoustic features as the first content feature; it can also be obtained through deep learning encoding methods, such as inputting audio data into a deep learning model, like a Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), or Transformers, where the model automatically learns high-level features from the audio data and generates content-encoded features; it can also be obtained through spectral analysis encoding methods, such as converting the audio signal into a spectrogram, extracting features from the spectrogram, and using the extracted features as the first content feature; and it can also be obtained through speech recognition technology encoding methods, such as using speech recognition technology to convert audio into text, analyzing the converted text, extracting semantic features such as keywords and grammatical structures, and using these semantic features as the first content feature; no specific limitations are made here.

[0074] As an example, suppose we need to encode an audio segment. First, the continuous analog waveform of the audio signal is converted into discrete digital samples, and the continuous audio signal is divided into a series of short frames, typically about 20-30ms each. A window function (such as a Hamming window or Hanning window) is applied to each audio frame to reduce boundary effects. Next, a Fourier transform is performed on each windowed audio frame to obtain a frequency domain representation. This frequency domain representation is then passed through a set of Mel frequency filters to simulate the sensitivity of the human ear to different frequencies. The logarithm of the output of each filter is then taken and subjected to a discrete cosine transform to obtain MFCC features. Finally, the extracted MFCC features are input into a pre-trained encoder model. The encoder model analyzes the features, learns deeper audio features, and outputs the audio content encoding features. These features represent the content information in the audio, such as language, musical style, and emotion. Furthermore, after extracting the content features, further processing such as feature dimensionality reduction, feature selection, and feature normalization can be performed.

[0075] In some embodiments, the data processing method described above is executed by invoking a first model, see [link to relevant documentation]. Figure 4 , Figure 4 This is a flowchart illustrating the data processing method provided in an embodiment of this application, as shown below. Figure 4 As shown, during execution Figure 3 Before step 101 shown, you can also perform... Figure 4 Steps 107 to 109 shown will combine Figure 4The steps shown are explained.

[0076] In step 107, the second audio is forward propagated in the first model to obtain the second facial animation data.

[0077] In some embodiments, step 107 described above can be implemented as follows: The second audio is first encoded using the first model to obtain a second content feature, and the second audio is second encoded to obtain a second emotion feature; the second content feature and the second emotion feature are fused using the first model to obtain a second fused feature; the second fused feature and the second emotion feature are first decoded using the first model to obtain third emotion data and second lip-sync data, and the second emotion feature is second decoded to obtain fourth emotion data; second facial animation data is generated using the first model based on the third emotion data, the fourth emotion data, and the second lip-sync data. Thus, by simultaneously learning content features and emotion features, the learning effect of each can be improved, enhancing the overall generalization ability of the model; secondly, by fusing content features and emotion features, a more comprehensive and richer feature representation can be generated, helping the model to better capture the complex information of the audio; furthermore, by using the decoded emotion data and lip-sync data, the model can generate more realistic and natural animations, enabling the model to generate personalized animation data based on the emotion and content features of different audio content.

[0078] As an example, the second audio is first input into the first model. The content encoder in the first model maps the second audio to the content feature space for first encoding processing, obtaining the second content features (including the underlying information of the audio, such as melody, rhythm, and speech features). Simultaneously, the emotion encoder in the first model maps the second audio to the emotion feature space for third encoding, obtaining the second emotion features (including the emotional content of the audio, such as happiness, sadness, and anger). Next, the first model fuses the extracted second content features and second emotion features to obtain the second fused features. Here, the fusion processing can be implemented through a specific layer or network structure, which can combine content and emotion information to more accurately express the complete information of the audio. Subsequently, the fused second fused features and second emotion features are input into the main decoder for first decoding processing, generating third emotion data and second lip-sync data. Here, the third emotion data is the interpretation of the emotion in the fused features, and the second lip-sync data is the relevant change data of the lip shape. Simultaneously, the second emotion features are input into the emotion decoder for second decoding processing, obtaining the fourth emotion feature. Finally, the first model generates second facial animation data based on the third emotion data, the fourth emotion data, and the second lip-sync data, thus completing the forward propagation process of the first model.

[0079] In step 108, a first loss is determined based on the second facial animation data and the reference animation data corresponding to the second audio.

[0080] It should be noted that the method for determining the first loss can be calculated using a specific loss function, such as the cross-entropy loss function, the mean squared error loss function, or the mean absolute error loss function, etc., and no specific limitation is made here.

[0081] As an example, taking the Wing loss function, firstly, the difference between the second facial animation data and the reference animation data is calculated, which is obtained by calculating the Euclidean distance between the two; then, the first loss is calculated according to formula (1).

[0082]

[0083] Where y refers to the reference animation data. This refers to the second facial animation data, where c and ∈ are the two parameters of the loss function. It is a measure of difference, that is, the difference between the reference animation data and the second facial animation data.

[0084] In some embodiments, after performing step 108 above, the following processing may also be performed: type prediction is performed on the fourth emotion data to obtain a first predicted emotion type; a second loss is determined based on the first predicted emotion type and the reference emotion type corresponding to the second audio. Thus, by predicting the emotion type and comparing it with the reference emotion type, the model can learn a more accurate emotion recognition capability, making the animation generated by the model more consistent with the emotional content of the audio; furthermore, accurate emotion type prediction helps generate animations that are more consistent with the emotional content of the audio, thereby improving the naturalness and credibility of the animation.

[0085] As an example, taking the cross-entropy loss function as an example, the second loss can be calculated using formula (2).

[0086]

[0087] Among them, y i Indicates the reference emotion type. This represents the probability distribution of the first predicted emotion type.

[0088] In step 109, the parameters of the first model are updated based on the first loss to obtain the trained first model.

[0089] In practical applications, when the first loss reaches the parameter threshold, backpropagation is performed in the first model based on the first loss, and the model parameters of the first model are updated during the propagation process.

[0090] Here's an explanation of backpropagation: The second audio input is fed into the input layer of the first model, passes through the hidden layer, and finally reaches the output layer to output the result. This is the forward propagation process of the first model. Since there is an error between the output result of the first model and the actual result, the error between the result and the actual value is calculated and backpropagated from the output layer to the hidden layer until it reaches the input layer. During the backpropagation process, the values ​​of the model parameters are adjusted according to the error. The above process is iterated until convergence.

[0091] Taking the loss function (1) of the first model in this application embodiment as an example, the server determines the first loss based on the loss function, propagates the first loss back from the output layer in the first model, propagates the first loss back layer by layer, and when the first loss reaches each layer, it combines the propagated first loss to solve the gradient (that is, the partial derivative of the loss function with respect to the parameters of each layer), and updates the corresponding gradient values ​​of the parameters of each layer.

[0092] In some embodiments, step 109 described above can also be implemented by fusing the first loss and the second loss to obtain a third loss; and updating the parameters of the first model based on the third loss to obtain a trained first model. Thus, by fusing the losses, optimization can be performed at multiple levels, thereby ensuring the quality of the generated animation and its emotional consistency with the audio content. Secondly, by simultaneously optimizing multiple performance metrics, the model can have better robustness to variations in input data and noise, thereby reducing overfitting of the model to specific datasets and improving its generalization ability under different conditions.

[0093] As an example, taking the loss function (1) and loss function (2) of the first model in this application embodiment as examples, the server determines the first loss based on the loss function (1) and the second loss based on the loss function (2), and fuses the first loss and the second loss to obtain the third loss. Here, the fusion process can be carried out by weighted summation, multilayer perceptron (MLP) fusion, or multi-task learning framework, etc., without specific limitations. The first loss is propagated backward from the output layer in the first model, and then propagated backward layer by layer. When the first loss reaches each layer, the gradient (that is, the partial derivative of the loss function with respect to the parameters of each layer) is solved by combining the propagated first loss, and the gradient values ​​of the parameters of each layer are updated.

[0094] Thus, by forward propagating the second audio and acquiring the second facial animation data, the model can generate corresponding animations based on the audio content. Determining the loss based on the generated animation and the reference animation data can help the model learn how to generate facial animations that match the audio content more accurately. Secondly, using the first loss to guide the update of the model parameters helps improve the accuracy of the model's animation generation, enabling the model to generate more realistic animation data that conforms to human facial expression habits.

[0095] In step 102, the first audio is encoded a second time to obtain a first emotional feature.

[0096] Here, feature extraction can be performed manually; predefined emotion label templates can be used to analyze audio and match audio features; relevant models (such as Hidden Markov Models, Recurrent Neural Networks, Long Short-Term Memory Networks, or Gated Recurrent Units) can be used to extract and encode emotion features; convolutional neural networks or end-to-end deep learning models can also be used to extract the emotion features of audio; no specific limitations are made here.

[0097] It should be noted that the implementation method of step 102 is similar to that of step 101 above. For details, please refer to the implementation method of step 101, which will not be repeated here.

[0098] In step 103, the first content feature and the first emotion feature are fused to obtain the first fused feature.

[0099] It should be noted that features can be fused by direct concatenation, by using deep learning models (such as convolutional neural networks, recurrent neural networks, etc.) to fuse different features, by using attention mechanisms to weighted fuse different features, or by using feature selection algorithms to fuse the most useful features. No specific limitations are made here.

[0100] As an example, first, a convolutional neural network is used to extract content features (such as pitch and rhythm) from the audio, and a recurrent neural network is used to extract emotional features (such as volume, speech rate, and emotion). Next, the content features and emotional features are input into a shared neural network, which can be a multilayer perceptron (MLP) or other types of deep learning models. Then, the shared neural network fuses the content features and emotional features to obtain a more comprehensive and accurate feature representation. Finally, the fused features are used for emotion classification or to generate corresponding animated emoticons.

[0101] In step 104, the first fusion feature and the first emotion feature are first decoded to obtain first emotion data and first lip shape data.

[0102] It should be noted that the first emotion data and the first lip shape data can be obtained by designing two independent decoding networks to decode them separately; alternatively, a shared decoding network can be used to decode the first emotion data and the first lip shape data simultaneously; other decoding methods can also be used, and no specific limitations are made here.

[0103] In some embodiments, the first fusion feature is a first matrix, and the first emotion feature is a second matrix. Step 104 described above can be implemented as follows: for each first element in the first matrix, determine the similarity between the second element in the second matrix that has the same position as the first element and the first element; for each second element in the second matrix, multiply the similarity corresponding to the second element with the second element to obtain the multiplication result corresponding to the second element, and generate a third matrix based on the multiplication result corresponding to each second element; merge the first matrix and the third matrix to obtain a fourth matrix, and segment the fourth matrix to obtain the first emotion data and the first lip-sync data. Thus, through similarity calculation and weighting, the first emotion feature is enhanced, which helps improve the model's learning effect. Furthermore, merging the first and third matrices can fuse content features and emotion features, providing a more comprehensive information representation and helping to generate emotion and lip-sync data more accurately. Secondly, by training and updating the model parameters, the model's adaptability to various audio data can be enhanced, enabling it to generate appropriate emotion and lip-sync animations in different scenarios, thereby improving prediction accuracy and generating animated expressions that better match the audio content.

[0104] As an example, suppose the first fusion feature (first matrix) contains content features extracted from the audio, such as pitch, volume, rhythm, etc., and the first emotion feature (second matrix) contains emotion features extracted from the audio, such as tempo, intensity, energy, etc.; then, for each element in the first matrix (e.g., pitch), determine the similarity between the corresponding element in the second matrix (e.g., tempo) and the element in the first matrix, where the similarity can be cosine similarity or other metrics; subsequently, multiply each element in the second matrix by its corresponding similarity, and generate a new matrix (third matrix) based on the multiplication result of each second element, where the third matrix contains weighted emotion features; then, merge the first matrix (content features) and the third matrix (weighted emotion features) to generate a new matrix (fourth matrix), where the fourth matrix contains combined information of content features and emotion features; finally, segment the fourth matrix, extracting the emotion-related parts as first emotion data and extracting the lip-shape-related parts as first lip-shape data.

[0105] In step 105, the first emotion feature is decoded a second time to obtain the second emotion data.

[0106] As an example, after extracting emotion-related features from audio frame macros, a suitable decoder model is selected, such as a simple statistical model (Support Vector Machine, Random Forest, etc.) or a complex deep learning model (Recurrent Neural Network, Convolutional Neural Network, Transformers, etc.). The emotion features are then input into the trained decoder model, which decodes the input emotion features and outputs the corresponding emotion-related data.

[0107] In step 106, based on the first emotion data, the second emotion data, and the first lip-sync data, first facial animation data matching the first audio is generated.

[0108] It should be noted that emotion data and lip-sync data can be mapped to specific facial movements according to predefined rules; pre-recorded or generated facial animation data can also be used, and appropriate templates can be selected for playback or adjustment based on emotion data and lip-sync data; machine learning or deep learning models can also be used to learn the mapping relationship between emotion data, lip-sync data and facial animation, thereby generating natural and realistic facial animations through the model; the specifics can be determined according to the actual situation, and no specific limitations are made here.

[0109] In some embodiments, step 106 described above can be implemented as follows: the first emotion data and the second emotion data are processed using a fully connected layer to obtain third emotion data; the third emotion data is then fused with the first lip-sync data to obtain the first facial animation data. Thus, merging the first and second emotion data through fully connected processing enhances the model's ability to recognize emotions. The fully connected layer can learn the complex relationships between emotion features, thereby improving the accuracy of emotion recognition. Secondly, fusing the two emotion data provides more comprehensive emotional information, helping to capture and express emotions more accurately. Furthermore, the fully connected processing and fusion steps enhance the expressive power of emotion features, enabling the model to better understand and generate facial animations that match the audio content. Therefore, fusing emotion data through fully connected processing and finally fusing it with lip-sync data enhances the model's adaptability to facial animation generation tasks, improves the naturalness and personalization of the animation, simplifies the animation generation process, and helps create more realistic and diverse facial animations.

[0110] As an example, suppose the first emotion data is emotion-related data, such as pitch, volume, and speech rate; the second emotion data is speech manner, such as tone and pauses; and the first lip shape data is lip-related features, such as the degree of lip opening and closing and the position of the tongue. Next, the first and second emotion data are input into a fully connected layer for fully connected processing to obtain the fused third emotion data. Subsequently, the third emotion data is fused with the first lip shape data to generate the corresponding facial animation data. Here, the fusion processing refers to the splicing of the first emotion data and the first lip shape data.

[0111] The following describes an exemplary application of the embodiments of this application in a real-world application scenario. This exemplary application describes the specific implementation process of the data processing method in a virtual human face generation scenario.

[0112] In recent years, with the rapid development of virtual reality, digital humans, and human-computer interaction technologies, 3D facial animation generation technology has been widely used in various fields such as virtual anchors, game characters, and intelligent customer service. Among related technologies, 3D facial animation generation technology mainly relies on audio-driven methods, and the main technical routes cover phoneme recognition-based driving methods, lip-shape mapping-based expression reconstruction methods, and end-to-end methods that utilize deep learning to achieve speech-to-facial motion mapping.

[0113] Phoneme-driven methods convert speech signals into phoneme sequences and match them with predefined lip-shape correspondence tables to control lip-shape changes in facial models. For example, Hidden Markov Models (HMMs) or Recurrent Neural Networks (RNNs) can be used to achieve automatic speech-to-lip-shape sequence conversion. However, most of these methods focus on mouth movements, neglecting facial expression changes closely related to the emotional state of speech, such as the overall coordinated movements of eye contact, eyebrows, and facial muscles. This results in facial animations that are significantly lacking in emotional expression, exhibit strong regularity, and lack naturalness overall.

[0114] Furthermore, to enhance the realism of facial expressions, some scholars have proposed multimodal methods that jointly train speech signals and video images, such as neural network models that combine visual facial landmark regression with speech feature embedding, thereby improving lip-sync. However, this method still has the following prominent problems:

[0115] (1) Lack of emotional driving mechanism. Related technologies only focus on the correspondence between speech and lip movements, failing to fully explore and utilize the rich emotional information contained in speech, resulting in a lack of emotional dimension in the generated facial animation, making the expressions appear stiff and unnatural, and failing to meet users' expectations for emotional expression.

[0116] (2) Insufficient animation expressiveness. Due to the lack of effective participation of emotional components, the facial expressions of the characters cannot accurately convey real emotions such as "happiness, sadness, and anger" in terms of semantic expression. This greatly weakens the user's immersive experience and interactive effect, and limits its application value in emotional interaction scenarios.

[0117] (3) Lack of versatility and adaptability. In related technologies, the models usually have high adaptability to specific 3D models, but when faced with different model inputs, problems such as misaligned mouth shapes or unnatural performance are easy to occur, exposing their lack of robustness.

[0118] Therefore, this application proposes a scheme using an emotion dimension discriminator to help generate more vivid and realistic facial expressions in 3D facial animation tasks. This application focuses on the task of generating faces from speech, deeply mining and utilizing speech emotion features to generate accurate emotion representation dimensions. Based on this, an efficient emotion dimension discriminator is designed and integrated into the model architecture to constrain and guide the final facial effect generated by the model. Through this mechanism, precise control of the generated facial expressions is achieved, ensuring not only accurate lip movements but also a natural and fluent presentation of an emotional state highly consistent with the current emotion label. Based on this, this application uses BlandShapes to describe the output and introduces a speech emotion discriminator to constrain the generation of facial expressions. By classifying the emotion dimension generated by the model, the model is constrained to generate facial expressions that conform to the given emotion.

[0119] In some embodiments, to enable the generated facial animation effects to be widely applied to various 3D models, rather than being limited to a specific face model, this application uses BlendShapes to represent the generated effects. BlendShapes is a linear facial model that represents various facial expressions through a linear combination of multiple basic vectors. In this application, a 52-dimensional BlendShapes model is used, with each dimension corresponding to a specific basic unit of facial movement. For example, dimension 1 represents blinking the left eye, dimension 2 represents blinking the right eye, etc. Each dimension also has a threshold x∈[0,1], the magnitude of which represents the degree of the current movement. For example, see [link to relevant documentation]. Figure 5 , Figure 5 These are schematic diagrams illustrating different dimensions of facial animation effects provided in the embodiments of this application. Figure 5 When BS1 = 0.5 on the left, the left eye is half-closed; Figure 5 When BS1 = 1 on the right, the left eye is completely closed.

[0120] It should be noted that in practical applications, the 52-dimensional BlendShape parameters are divided into mouth-related groups ( ) and emotion-related groups ( Two groups are used, where n1 + n2 = 52. The main improvement of this application is to generate facial expressions that better reflect emotions; therefore, this application mainly focuses on the emotion-related Emo dimension.

[0121] In some embodiments, see Figure 6 , Figure 6 This is a specific network architecture diagram provided in the embodiments of this application, such as... Figure 6 As shown, a new emotion decoder is added to the network architecture. The emotion decoder decodes Emojis. BS The mapping result, that is, mapping the input audio to the emotion space using an emotion feature encoder to obtain emotion features, is used as input to the emotion decoder. The emotion decoder then decodes the emotion BS representation corresponding to the current frame audio. aux Simultaneously, the Emo results from the emotion decoder and the main decoder are fused through a fully connected layer (as shown in Equation (3)). This is done in conjunction with the main decoder's shape... main By splicing them together, we get the final output BS. final As shown in formula (4).

[0122] BS main =[Shpae main Emo main (3)

[0123]

[0124] in, Represents feature connections, FC is a fully connected layer, Shape main The lip-sync decoding data of the main decoder, Emo main Emotion decoding data for the main decoder, Emo aux This is the emotion decoding data for the emotion decoder.

[0125] In some embodiments, see Figure 6 The specific implementation process of model training corresponding to the data processing method provided in this application is as follows.

[0126] First, the original audio is input into the model. A content encoder maps the original audio to a content space for encoding, obtaining content-encoded features. Simultaneously, a sentiment encoder maps the original audio to a sentiment space for encoding, obtaining sentiment-encoded features. Next, the content-encoded features and sentiment-encoded features are concatenated to obtain concatenated features. These features are then fused with the concatenated features using a cross-attention mechanism, leveraging the sentiment-encoded features to guide the generation of the effect, generating a full 52-dimensional face feature set (i.e., the BlendShapes model, including lip shape and emotion). Simultaneously, the sentiment-encoded features are input into the sentiment encoder to generate an emotion-related feature set, which is then input into an emotion classification network to predict the emotion category and classification loss. Next, a fully connected layer fuses the emotion-related feature set with the emotion feature set generated by the master decoder to generate the final emotion feature set. This final emotion feature set is then fused with the lip shape feature set to obtain an emotion-enhanced feature set. A Wing loss is calculated based on the emotion-enhanced feature set and a reference feature set. Finally, backpropagation is performed using the Wing loss and classification loss to update the parameters of the entire model.

[0127] In some embodiments, see Figure 7 , Figure 7 This is a model architecture diagram of the classifier provided in this application, such as... Figure 7 As shown, in order to further constrain the accuracy of emotion expression, the network also performs emotion classification prediction on the output of the emotion encoder while learning the BS output. The emotion classification network adopts the BERT network structure, and the content after the categorical label of the BERT output is input into the fully connected layer for classification processing to obtain the predicted emotion classification result.

[0128] In some embodiments, the total loss function of the above model is the sum of the classification loss and the Wing loss, which can be calculated by formula (5).

[0129] Loss = Loss BS +αLoss CLS (5)

[0130] Among them, Loss BS Using Wing loss, Loss CLS Using cross-entropy loss, where α is the classification loss weight, the model can achieve the optimal balance between emotional naturalness and parameter accuracy when the weight coefficient α = 0.1.

[0131] In summary, this application, while maintaining the natural lip shape of the generated virtual human, enhances the naturalness and strong emotionality of the generated virtual human under different emotional expressions by introducing emotion-based B / S classification constraints, thus providing a brand-new solution for emotion computing and intelligent human-computer interaction.

[0132] The following description continues to illustrate the exemplary structure of the data processing apparatus 555 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the data processing device 555 of the memory 550 may include: an encoding module 5551, a fusion module 5552, a decoding module 5553, and a generation module 5554.

[0133] Encoding module 5551 is used to perform a first encoding on the first audio to obtain a first content feature, and to perform a second encoding on the first audio to obtain a first emotion feature; fusion module 5552 is used to fuse the first content feature and the first emotion feature to obtain a first fusion feature; decoding module 5553 is used to perform a first decoding on the first fusion feature and the first emotion feature to obtain first emotion data and first lip-sync data, and to perform a second decoding on the first emotion feature to obtain second emotion data; generation module 5554 is used to generate first facial animation data matching the first audio based on the first emotion data, the second emotion data, and the first lip-sync data.

[0134] In some embodiments, the first fusion feature is a first matrix, the first emotion feature is a second matrix, and the decoding module 5553 is further configured to: for each first element in the first matrix, determine the similarity between the second element in the second matrix that has the same position as the first element and the first element; for each second element in the second matrix, multiply the similarity corresponding to the second element by the second element to obtain the multiplication result corresponding to the second element, and generate a third matrix based on the multiplication result corresponding to each second element; merge the first matrix and the third matrix to obtain a fourth matrix, and segment the fourth matrix to obtain the first emotion data and the first lip shape data.

[0135] In some embodiments, the generation module 5554 is further configured to perform full connection processing on the first emotion data and the second emotion data to obtain third emotion data; and to perform fusion processing on the third emotion data and the first lip shape data to obtain the first facial animation data.

[0136] In some embodiments, the data processing method is performed by invoking a first model, and the data processing device 555 further includes a training module 5555.

[0137] The training module 5555 is used to forward propagate the second audio in the first model to obtain the second facial animation data; determine the first loss based on the second facial animation data and the reference animation data corresponding to the second audio; and update the parameters of the first model based on the first loss to obtain the trained first model.

[0138] In some embodiments, the training module 5555 is further configured to: encode the second audio using the first model to obtain a second content feature; encode the second audio using the first model to obtain a second emotion feature; fuse the second content feature and the second emotion feature using the first model to obtain a second fused feature; decode the second fused feature and the second emotion feature using the first model to obtain third emotion data and second lip-sync data; decode the second emotion feature using the first model to obtain fourth emotion data; and generate second facial animation data using the first model based on the third emotion data, the fourth emotion data, and the second lip-sync data.

[0139] In some embodiments, the training module 5555 is further configured to perform type prediction on the fourth emotion data to obtain a first predicted emotion type; and determine a second loss based on the first predicted emotion type and the reference emotion type corresponding to the second audio.

[0140] In some embodiments, the training module 5555 is further configured to fuse the first loss and the second loss to obtain a third loss; and update the parameters of the first model based on the third loss to obtain a trained first model.

[0141] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For any technical details not covered in the data processing apparatus provided in this application embodiment, please refer to... Figure 3 or Figure 4 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0142] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the data processing method described in this application.

[0143] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the data processing method provided in this application. For example, ... Figure 3 ,or Figure 4 The data processing method is shown.

[0144] In some embodiments, the computer-readable storage medium may be a memory such as ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0145] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0146] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0147] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0148] In summary, the embodiments of this application firstly encode the first audio file by mapping it to a content space to obtain a first content feature, and then encode the first audio file by mapping it to an emotion space to obtain a first emotion feature, thereby obtaining the content and emotion information of the audio. Next, the first content feature and the first emotion feature are fused to obtain a first fused feature, which can provide a more comprehensive audio feature representation. Subsequently, the first fused feature and the first emotion feature are first decoded to obtain first emotion data indicating the overall emotional expression of the facial expression and first lip-shape data indicating the corresponding audio content. Then, the first emotion feature is second decoded to obtain second emotion data indicating the specific emotional expression of the facial expression. Finally, the first emotion data, the second emotion data, and the first lip-shape data are used together to generate facial animation data that matches the first audio file, ensuring that the animation and the first audio file are consistent in content and emotion, thereby ensuring that the generated facial animation is more natural and emotionally resonant under different emotional expressions.

[0149] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: The first audio is encoded in a first way to obtain a first content feature, and the first audio is encoded in a second way to obtain a first emotion feature; The first content feature and the first emotion feature are fused together to obtain the first fused feature; The first fusion feature and the first emotion feature are first decoded to obtain first emotion data and first lip shape data, and the first emotion feature is second decoded to obtain second emotion data. Based on the first emotion data, the second emotion data, and the first lip shape data, first facial animation data matching the first audio is generated.

2. The method according to claim 1, characterized in that, The first fusion feature is a first matrix, and the first emotion feature is a second matrix; The first decoding of the first fused feature and the first emotion feature to obtain the first emotion data and the first lip-shape data includes: For each first element in the first matrix, determine the similarity between the second element in the second matrix that has the same position as the first element and the first element; For each second element in the second matrix, the similarity corresponding to the second element is multiplied by the second element to obtain the multiplication result of the corresponding second element, and a third matrix is ​​generated based on the multiplication result of each corresponding second element; The first matrix and the third matrix are merged to obtain a fourth matrix, and the fourth matrix is ​​then segmented to obtain the first emotion data and the first lip shape data.

3. The method according to claim 1, characterized in that, The step of generating first facial animation data matching the first audio based on the first emotion data, the second emotion data, and the first lip-sync data includes: The first and second emotion data are processed by a full connection to obtain the third emotion data; The third emotion data is fused with the first lip shape data to obtain the first facial animation data.

4. The method according to claim 1, characterized in that, The data processing method is executed by calling a first model. Before the first audio is content-encoded to obtain the first content feature, the method further includes: The second audio is propagated forward through the first model to obtain the second facial animation data; Based on the second facial animation data and the reference animation data corresponding to the second audio, a first loss is determined; Based on the first loss, the parameters of the first model are updated to obtain the trained first model.

5. The method according to claim 4, characterized in that, The step of forward propagating the second audio in the first model to obtain the second facial animation data includes: The second audio is encoded using the first model to obtain a second content feature, and then encoded using the second model to obtain a second emotion feature. The second content feature and the second emotion feature are fused together using the first model to obtain the second fused feature; The first model is used to perform a first decoding on the second fusion feature and the second emotion feature to obtain the third emotion data and the second lip shape data, and the second emotion feature is then performed a second decoding to obtain the fourth emotion data. The first model generates second facial animation data based on third emotion data, fourth emotion data, and second lip shape data.

6. The method according to claim 5, characterized in that, After performing a second decoding on the second emotional feature to obtain the fourth emotional data, the method further includes: The fourth emotion data is used to predict the type of emotion, and a first predicted emotion type is obtained. The second loss is determined based on the first predicted sentiment type and the reference sentiment type corresponding to the second audio.

7. The method according to claim 6, characterized in that, The step of updating the parameters of the first model based on the first loss to obtain the trained first model includes: The first loss and the second loss are combined to obtain the third loss; Based on the third loss, the parameters of the first model are updated to obtain the trained first model.

8. A data processing apparatus, characterized in that, The device includes: The encoding module is used to perform a first encoding on the first audio to obtain a first content feature, and to perform a second encoding on the first audio to obtain a first emotion feature; The fusion module is used to fuse the first content feature and the first emotion feature to obtain a first fused feature; The decoding module is used to perform a first decoding on the first fused feature and the first emotion feature to obtain first emotion data and first lip shape data, and to perform a second decoding on the first emotion feature to obtain second emotion data. The generation module is used to generate first facial animation data that matches the first audio based on the first emotion data, the second emotion data, and the first lip shape data.

9. An electronic device, characterized in that, include: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images