Data processing method and apparatus based on artificial intelligence, electronic device, computer program product, and computer readable storage medium

Abstract

Embodiments of the present application relate to speech processing technology. The embodiments of the present application provide a data processing method and apparatus based on artificial intelligence, an electronic device, a computer program product, and a computer readable storage medium. The method comprises: acquiring speech, and acquiring emotion data corresponding to the speech; performing first speech content feature extraction processing on the speech to obtain a first speech content feature corresponding to the speech, and performing emotion feature extraction processing on the emotion data to obtain emotion features corresponding to the speech; performing fusion processing on the first speech content feature and the emotion features to obtain a fusion feature corresponding to the speech; and performing animation parameter mapping processing on the fusion feature to obtain controller parameters corresponding to the speech, wherein the controller parameters are used for controlling a virtual object model to be presented in a target image, and the target image matches the content of the speech and the emotion data.

Description

Artificial intelligence-based data processing method, device, electronic device, computer program product, and computer-readable storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410147225.3 and application date of February 2, 2024, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to the field of speech processing, and in particular to an artificial intelligence-based data processing method, device, electronic device, computer program product, and computer-readable storage medium. Background Art

[0004] Artificial Intelligence (AI) is a comprehensive field of computer science. By studying the design principles and implementation methods of various intelligent machines, AI enables them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in even more areas and play an increasingly important role.

[0005] In order to realize facial expression driving of virtual objects in related technologies, it is necessary to prepare expression bases corresponding to different phonemes in advance, then convert the voice into corresponding phonemes, and combine different expression bases into complete animations through complex rules. Related technologies can only realize facial expression driving based on phonemes, and the process of combining expression bases is relatively complicated, the facial expression driving efficiency is low, and the driving accuracy is low. Summary of the Invention

[0006] The embodiments of the present application provide an artificial intelligence-based data processing method, device, electronic device, computer program product, and computer-readable storage medium, which can improve the target image driving efficiency and driving accuracy for a virtual object model.

[0007] The technical solution of the embodiment of the present application is implemented as follows:

[0008] An embodiment of the present application provides a data processing method based on artificial intelligence, the method being executed by an electronic device, the method comprising:

[0009] Acquire speech and obtain emotion data corresponding to the speech;

[0010] Performing a first voice content feature extraction process on the voice to obtain a first voice content feature corresponding to the voice, and performing an emotion feature extraction process on the emotion data to obtain an emotion feature corresponding to the voice;

[0011] fusing the first speech content feature and the emotion feature to obtain a fusion feature corresponding to the speech;

[0012] Performing animation parameter mapping processing on the fusion feature to obtain controller parameters corresponding to the voice;

[0013] The controller parameters are used to control the virtual object model to be presented in a target image, and the target image matches the content of the speech and the emotion data.

[0014] The present invention provides an artificial intelligence-based data processing device, comprising:

[0015] an acquisition module configured to acquire speech and acquire emotion data corresponding to the speech;

[0016] a feature extraction module configured to perform a first voice content feature extraction process on the speech to obtain a first voice content feature corresponding to the speech, and perform an emotion feature extraction process on the emotion data to obtain an emotion feature corresponding to the speech;

[0017] a superposition processing module configured to perform a fusion process on the first speech content feature and the emotion feature to obtain a fusion feature corresponding to the speech;

[0018] A mapping processing module is configured to perform animation parameter mapping processing on the fusion feature to obtain controller parameters corresponding to the voice, wherein the controller parameters are used to control the virtual object model to present a target image, and the target image matches the content of the voice and the emotional data.

[0019] An embodiment of the present application provides an electronic device, comprising:

[0020] a memory for storing computer-executable instructions;

[0021] The processor is used to implement the artificial intelligence-based data processing method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0022] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the artificial intelligence-based data processing method provided in an embodiment of the present application when executed by a processor.

[0023] An embodiment of the present application provides a computer program product, including computer-executable instructions. When the computer-executable instructions are executed by a processor, the artificial intelligence-based data processing method provided in the embodiment of the present application is implemented.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] By acquiring speech and acquiring the emotion data corresponding to the speech, the content of the speech and the emotion of the corresponding speech are obtained, a first speech content feature extraction process is performed on the speech to obtain the first speech content feature of the corresponding speech, which is used to characterize the speech itself, and an emotion feature extraction process is performed on the emotion data to obtain the emotion feature of the corresponding speech, which is used to characterize the speech emotion, and the speech content feature and the emotion feature are fused to obtain the fusion feature of the corresponding speech, that is, the characteristics of the speech itself and the emotion characteristics can be characterized at the same time, and the fusion feature is animated parameter mapping processed to obtain the controller parameters of the corresponding speech, and the controller parameters are used to control the virtual object model to present the target image, and the target image matches the content and emotion data of the speech. Here, a mapping process from features to controller parameters is directly established, so that the controller parameters used to drive the virtual object model to present the target image can be directly obtained, which improves the driving efficiency while also improving the driving efficiency of the target image of the virtual object model from the two dimensions of speech and emotion. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a schematic diagram of the architecture of a data processing system provided in an embodiment of the present application;

[0027] FIG2 is a schematic diagram of the structure of a server provided in an embodiment of the present application;

[0028] FIG3A is a schematic diagram of a flow chart of an artificial intelligence-based data processing method provided in an embodiment of the present application;

[0029] FIG3B is an optional flow chart of an artificial intelligence-based data processing method provided in an embodiment of the present application;

[0030] FIG4 is a schematic diagram of the training process of the emotion extraction network provided in an embodiment of the present application;

[0031] FIG5A is a flow chart of a parameter-driven model training process according to an embodiment of the present application;

[0032] FIG5B is a flowchart of an optional training of a parameter-driven model provided in an embodiment of the present application;

[0033] FIG6 is a background service flow chart of the voice and face driving service provided in an embodiment of the present application;

[0034] FIG7 is a flowchart of a background service of a virtual person live broadcast provided by an embodiment of the present application;

[0035] FIG8 is a schematic diagram of the architecture of a parameter-driven model provided in an embodiment of the present application;

[0036] FIG9 is a schematic diagram of a character facial animation production page provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0041] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0042] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0043] 1) 4D Scanning: Based on 3D scanning, with the addition of a timeline, each frame is a scanned 3D model. 4D scanning lies somewhere between 3D scanning and motion capture. The typical animation production process involves 3D scanning, model cleaning, skeletal rigging, and manual debugging or animation using motion capture data.

[0044] 2) Artificial Intelligence Generative Content (AIGC): AI-generated content, such as images, videos, music, and text, is generated automatically by AI. AI leverages its understanding, imagination, and creativity to create content based on specific needs and styles, including articles, short stories, reports, music, images, and even videos.

[0045] 3) Waveform-to-Vec2 (wav2vec2) model: This pre-trained model for automatic speech recognition (ASR) performs speech recognition by predicting speech units from masked portions of speech. It first uses a multi-layer convolutional neural network to process the raw waveform of the speech audio. This raw waveform is then fed into a quantizer and a self-attention model (transformer). The quantizer selects a speech unit from a learned list of units as a latent audio representation vector. The self-attention model then adds information from the entire audio sequence, and the output is used to calculate the loss function. The waveform-to-Vec2 model identifies the correct quantized speech unit based on the masked position.

[0046] 4) Voice Operated Character Animation (VOCA): This allows users to input text or voice, and generates the corresponding virtual object facial expression coefficients through certain rules or deep learning algorithms, thereby accurately driving the virtual object's lip shape and facial expressions, and producing virtual object facial animation.

[0047] In recent years, technologies such as virtual human live streaming and generative AI have exploded in popularity. These technologies have also found numerous applications in game development, such as speech synthesis, motion generation, and 3D facial animation. Voice-driven facial solutions convert input speech signals into corresponding facial expressions and lip movements. This technology can be applied to voice assistants, virtual characters, games, and other fields, making human-computer interaction more intelligent and humane. By using voice-driven lip movements, the lip movements of virtual or animated characters can be synchronized with the speech, adding interest and realism to human-computer interaction.

[0048] Related speech-driven lip-syncing solutions are categorized into two types: expert system-based solutions based on phonemes and deep learning-based solutions, which have become more popular in recent years. Expert system-based solutions require professional animators to pre-create expression bases corresponding to different phonemes, then convert the speech into corresponding phonemes and combine the different expression bases into a complete animation using complex rules. Deep learning-based solutions obtain audio vectors from speech signals, input these audio vectors into a deep learning model, and output the changes in the 3D vertex coordinates of a 3D facial head model.

[0049] When implementing the embodiments of this application, the applicant discovered that the related technology had the following problems:

[0050] 1) The expert system method based on phonetic and visual elements requires animation experts to make detailed mapping rules between speech phonemes and three-dimensional lip shapes according to the actual laws of facial movement. This is time-consuming and labor-intensive, and is limited by the professional level of animation experts.

[0051] 2) The deep learning method directly maps the signal to the vertex cloud of the 3D model. As a result, when the animator modifies the 3D animation, the vertices of the 3D model need to be modified, which is not conducive to secondary modification of the 3D animation. In addition, this method does not perform additional processing on the emotions in the speech, resulting in the inability to synthesize expressions similar to the speech emotions.

[0052] The embodiments of the present application provide an artificial intelligence-based data processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the target image driving efficiency and driving accuracy for a virtual object model.

[0053] The artificial intelligence-based data processing method provided in the embodiments of the present application can be implemented by the terminal / server alone; it can also be implemented by the terminal and the server in collaboration, for example, the server alone undertakes the artificial intelligence-based data processing method below, or, after the terminal obtains the voice and emotion data, it sends the voice and emotion data to the server, and the server executes the artificial intelligence-based data processing method based on the received voice and emotion data.

[0054] Refer to Figure 1, which is a schematic diagram of the architecture of the data processing system 100 provided in an embodiment of the present application. In order to support an artificial intelligence-based data processing application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0055] In some embodiments, the terminal 400 is used to generate a data processing request. For example, the user inputs a voice through the graphical interface 410 of the terminal 400, and the terminal 400 generates a data processing request corresponding to the voice. The server 200 is used to obtain the voice and the emotion data of the corresponding voice according to the data processing request corresponding to the voice, perform voice content feature extraction on the voice to obtain the voice content feature of the corresponding voice, and perform emotion feature extraction on the emotion data to obtain the emotion feature of the corresponding voice, superimpose the voice content feature and the emotion feature to obtain the superimposed feature of the corresponding voice, perform animation parameter mapping on the superimposed feature to obtain the controller parameters of the corresponding voice, wherein the controller parameters are used to control the virtual object model to present the target image, and the target image matches the content and emotion data of the voice. The server 200 feeds back the controller parameters of the corresponding voice to the terminal 400, and the terminal 400 controls the virtual object model to present the target image according to the controller parameters of the corresponding voice.

[0056] In some embodiments, the terminal 400 is used to generate a data processing request. For example, the user inputs voice and emotion data through the graphical interface 410 of the terminal 400. The terminal 400 generates a data processing request carrying the voice and emotion data and sends it to the server 200. The server 200 obtains the voice and the emotion data corresponding to the voice, performs voice content feature extraction on the voice to obtain the voice content feature of the corresponding voice, and performs emotion feature extraction on the emotion data to obtain the emotion feature of the corresponding voice. The voice content feature and the emotion feature are superimposed to obtain the superimposed feature of the corresponding voice, and the superimposed feature is animated parameter mapped to obtain the controller parameters of the corresponding voice. The controller parameters are used to control the virtual object model to present the target image. The target image matches the content and emotion data of the voice. The server 200 feeds back the controller parameters of the corresponding voice to the terminal 400. The terminal 400 controls the virtual object model to present the target image according to the controller parameters of the corresponding voice.

[0057] In some embodiments, the data processing method provided by the embodiment of the present application can be applied to the field of games. For example, in the game development stage, the data processing method provided by the embodiment of the present application can directly input voice to control the virtual object model to present with a target image. Since the display image of the virtual object in the game changes rapidly (for example, the process of game character A reciting his classic lines is the dynamic display image of the game character A), the data processing method provided by the embodiment of the present application can be accompanied by the adjustment of the voice, and the virtual object model can be controlled at any time to present with different target images, thereby improving the iteration efficiency and iteration accuracy of the display image of the virtual object. For example, the data processing method provided by the embodiment of the present application can also be applied to the user-generated content scenario (User Generated Content, UGC) in the game. For example, in the UGC scenario, the data processing method provided by the embodiment of the present application can obtain the voice and emotion data input by the user, and can automatically control the virtual object model to present with the target image. The target image here matches the content of the voice and the emotion data, so that the user can customize the target image of the virtual object model independently, improving the efficiency of human-computer interaction. For example, the data processing method provided in the embodiment of the present application can also be applied to user intelligent interaction scenarios in games. For example, in the user intelligent interaction scenario, the dialogue voice input by the user is obtained, and a reply voice and emotional data for replying to the dialogue voice are generated. Through the data processing method provided in the embodiment of the present application, based on the reply voice and emotional data, the virtual object model of the intelligent non-player character (NPC) in the game can be automatically controlled to present the target image and output the reply voice at the same time. The target image here matches the content of the reply voice and the emotional data, that is, the lip shape of the intelligent non-player character conforms to the content of the reply voice, and the expression and action of the intelligent non-player character match the emotional data, thereby forming a game effect of an intelligent dialogue between the NPC and the user, which can effectively enrich the human-computer interaction mode and improve the efficiency of human-computer interaction.

[0058] The data processing method provided in the embodiments of the present application can also be applied to live broadcast scenarios. For example, in a news live broadcast scenario, the live broadcast is carried out by a virtual anchor. The virtual object model of the virtual anchor can be automatically controlled by inputting voice, so that the virtual anchor is presented with a target image, thereby realizing the construction of an anchor image that takes into account both voice and emotion, and improving the driving accuracy and driving efficiency of the virtual object model during the live broadcast process.

[0059] The electronic device for executing the data processing method provided in the embodiments of the present application can be various types of terminal devices or servers. In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0060] Referring to Figure 2, Figure 2 is a schematic diagram of the structure of a server 200 provided in an embodiment of the present application. The server 200 shown in Figure 2 includes: at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the server 200 are coupled together via a bus system 240. It will be understood that the bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 240 in Figure 2.

[0061] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0064] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0066] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0067] A network communication module 252 for reaching other electronic devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0068] In some embodiments, the apparatus provided in the embodiments of the present application can be implemented using software. FIG2 shows an artificial intelligence-based data processing apparatus 253 stored in memory 250. This apparatus 253 can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 2531, a feature extraction module 2532, an overlay processing module 2533, and a mapping processing module 2534. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0069] In some embodiments, the terminal or server can implement the data processing method based on artificial intelligence provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as a live broadcast APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0070] Below, the artificial intelligence-based data processing method provided by the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the server provided in the embodiment of the present application. As mentioned above, the electronic device that implements the artificial intelligence-based data processing method of the embodiment of the present application can be a terminal, a server, or a combination of the two. Therefore, the execution entity of each step will not be repeated below.

[0071] It should be noted that the examples of the artificial intelligence-based data processing method below are explained using facial animation as an example. Those skilled in the art, based on their understanding of the following, can apply the artificial intelligence-based data processing method provided in the embodiments of the present application to animation processing including other types of objects.

[0072] See Figure 3A, which is a flow chart of the artificial intelligence-based data processing method provided in an embodiment of the present application, and will be explained in conjunction with steps 101 to 104 shown in Figure 3A.

[0073] In step 101, speech is acquired, and emotion data corresponding to the speech is acquired.

[0074] For example, a speech can be an audio file containing speech or a live speech clip. In the context of virtual object animation, the speech is typically processed by a pre-processing module. The pre-processing module can be a text-to-speech processing module or another processing module that outputs speech. The emotion data corresponding to the speech is a matrix that represents the various emotions expressed in the speech content.

[0075] As an example, emotion data can be an emotion intensity matrix. The emotion intensity matrix is ​​a one-dimensional matrix where each position corresponds to a different emotion type, and the value at each position represents the intensity of the corresponding emotion type. For example, the emotion intensity matrix is ​​[1, 0, 0, 0, 0, 0], where the first position represents joy, the second position represents anger, the third position represents sadness, the fourth position represents surprise, the fifth position represents fear, the sixth position represents disgust, and the seventh position represents no emotion. This is equivalent to the only emotion being joy, and the intensity is 1. A higher intensity value represents a stronger emotion of that type.

[0076] In some embodiments, obtaining the emotion data of the corresponding speech in step 101 can be achieved by performing any one of the following processes: obtaining the emotion data input by the object; performing a second speech content feature extraction process on the speech to obtain the second speech content feature of the corresponding speech; performing an emotion data mapping process on the second speech content feature to obtain the emotion data of the corresponding speech.

[0077] As an example, the emotional data corresponding to the speech can be the emotional data input by the object. For example, the user customizes the emotional data corresponding to the speech according to the emotion corresponding to the speech content. It can also be the emotional data obtained by extracting emotions from the speech. It can also be the emotional data obtained by the user after adjustment based on the emotional data obtained by extracting emotions from the speech. The specific method is selected according to actual needs, and this application does not limit it here.

[0078] As an example, when obtaining emotion data by performing emotion extraction on speech, first, the second speech content feature extraction process is performed on the speech through the second speech content feature extraction network in the emotion extraction network to obtain the second speech content feature of the corresponding speech, and then the emotion recognition network in the emotion extraction network performs emotion data mapping process on the second speech content feature to obtain the emotion data of the corresponding speech. The second speech content feature extraction network here can be a wav2vec2 model or other models that can be used to extract speech content features. For example, for speech A1 "Why am I here?", according to formula (1), the second speech content feature extraction process is performed on speech A1 through the wav2vec2 model to obtain the second speech content feature H corresponding to speech A1. A1 : H A1 =wav2vec2(A1) (1)

[0079] Among them, H A1 is the output of the wav2vec2 model. Then the emotion recognition network is used to identify the second speech content feature H A1 Perform emotional data mapping to obtain the emotional data e corresponding to voice A1 A1 , where the emotional data e A1 ∈{joy, anger, sadness, surprise, fear, disgust}, corresponding to the emotional data e of speech A1 A1 ={e 喜 , e 怒 , e 哀 , e 惊 , e 恐 , e 厌}={0, 0, 0, 0.8, 0.2, 0}, where e 喜 is the emotional intensity corresponding to the emotion “joy”, e 怒 is the emotional intensity corresponding to the emotion “anger”, e 哀 is the emotional intensity corresponding to the emotion “sadness”, e 惊 is the emotional intensity corresponding to the emotion “shock”, e 恐 is the emotional intensity corresponding to the emotion “fear”, e 厌 The intensity of the emotion corresponding to “disgust”.

[0080] For example, Wav2Vec2 is a self-supervised pre-trained model for speech recognition tasks. It does not require labeled speech data for training, but instead uses an unlabeled speech corpus for pre-training. This allows the model to be applicable to languages ​​for which labeled data is scarce. It employs a contrastive learning strategy, which compares speech segments from different time periods to learn useful representations and predict the content of future audio segments. During model training, various enhancements are performed on the original audio, such as time stretching and noise addition, to help the model learn more robust and general speech representations. It uses autoregressive prediction to predict the next token in an audio sequence during pre-training. This enables the model to learn long-range dependencies within audio sequences. The model leverages temporal context at different scales to capture details and structure in the audio, processing it through different convolutional layers. After pre-training, the model can be fine-tuned on specific downstream tasks using a small amount of labeled data to further improve performance.

[0081] By obtaining the emotional data corresponding to the speech through user customization or speech emotion extraction, on the one hand, the emotional data corresponding to the speech can be obtained manually or automatically according to actual needs, and on the other hand, the emotions represented by the emotional data can be made to meet user expectations, thereby making the controller parameters obtained based on the emotional data and speech meet actual needs, so that the final generated facial animation matches the speech and speech emotions.

[0082] In some embodiments, the emotion data mapping processing of the second speech content feature in step 101 to obtain the emotion data of the corresponding speech can be achieved by: performing full connection processing on the second speech content feature to obtain the first linear mapping result; performing maximum likelihood processing on the first linear mapping result to obtain the emotion data of the corresponding speech.

[0083] As an example, the emotion recognition network is used to identify the second speech content feature H A1 Emotion data mapping is performed. The emotion recognition network here consists of a fully connected layer and a normalization layer. According to formula (2), the second speech content feature H is mapped to A1 Mapped to the emotion data e corresponding to speech A1 A1 : e A1 =softmax(linear(H A1 )) (2)

[0084] Among them, linear(H A1 ) is the fully connected layer for the second speech content feature H A1 The first linear mapping result obtained by full connection processing, softmax(linear(H A1)) is the normalization layer (softmax layer) for the first linear mapping result linear(H A1 ) is the output result obtained by maximum likelihood processing, that is, the emotional data e corresponding to speech A1 A1 .

[0085] By performing full connection processing and maximum likelihood processing on the second speech content features, the emotional data corresponding to the speech is output as a prediction result for the emotional data used to predict the controller parameters, so as to ensure that the predicted emotional data can be adjusted according to actual needs, and then the controller parameters obtained according to the emotional data and speech can meet the actual needs, so that the final generated facial animation matches the speech and the speech emotion.

[0086] In step 102, a first speech content feature extraction process is performed on the speech to obtain a first speech content feature of the corresponding speech, and an emotion feature extraction process is performed on the emotion data to obtain an emotion feature of the corresponding speech.

[0087] As an example, the first speech content feature extraction network in the parameter-driven model is used to extract the first speech content feature of speech A1. The first speech content feature extraction network here can be a wav2vec2 model or other models that can be used to extract speech content features. Here, the first speech content feature extraction network is taken as an example of the wav2vec2 model. According to formula (3), the first speech content feature extraction process is performed on speech A1 to obtain the first speech content feature H corresponding to speech A1. c : H c ={wav2vec2(A1)} (3)

[0088] Among them, wav2vec2(A1) is the output obtained by the wav2vec2 model performing the first speech content feature extraction process on the speech A1.

[0089] As an example, the emotion feature extraction network in the parameter-driven model is used to extract the emotion data e corresponding to speech A1 according to formula (4). A1 Perform emotional feature extraction to obtain the emotional feature H of the corresponding speech e :

[0090] Among them, e i For sentiment data A1 The i-th dimension emotion data in the q-dimensional emotion data, EMB(e i ) is the emotion feature extraction network corresponding to e i After fusing the q emotion feature extraction results, we get the emotion feature H e.

[0091] In step 103, the first speech content feature and the emotion feature are fused to obtain a fused feature of the corresponding speech.

[0092] As an example, the first speech content feature H corresponding to speech A1 c and emotional characteristics H e Perform fusion processing to obtain the fusion feature H corresponding to speech A1 c +H e The fusion processing here may be a superposition processing, a splicing processing, or other processing methods that can fuse different features, and this application does not limit this.

[0093] In step 104, animation parameter mapping processing is performed on the fusion features to obtain controller parameters corresponding to the speech.

[0094] As an example, controller parameters are used to control a virtual object model to be presented with a target image that matches the content of the speech and the emotion data.

[0095] As an example, the target image here includes but is not limited to facial image, body image, etc.

[0096] As an example, matching with emotion data involves matching the target image to the emotional intensity of each emotion type represented by the emotion data. For example, if the intensity of anger and sadness in the emotion data is 0.5 and 0.7, respectively, and the intensity of other emotion types is 0, the target image will reflect moderate anger and a strong dominant emotion. Matching with speech can occur in two ways: the target image's lip shape can match the speech, meaning the target image appears to be reading the speech, or the target image as a whole can match the meaning of the instruction represented by the speech, meaning the target image responds to the speech's action or expression. For example, if the speech is "raise your hand," the target image will show the action of raising its hand.

[0097] As an example, in the fields of virtual reality (VR), augmented reality (AR), and digital humans, controller parameters are used to ensure that the presentation of virtual objects or virtual characters matches the user's input (such as voice content, emotional data, etc.), thereby creating a more natural and immersive user experience. Controller parameters refer to a series of variables or settings that can be used to manipulate and control the presentation of virtual objects or models. These parameters may include: position parameters, which control the position of the object in the virtual space; rotation parameters, which control the direction or orientation of the object; scaling parameters, which control the size of the object; morphological parameters, which control details such as the shape or facial expression of the object; texture parameters, which control the surface texture and color of the object; animation parameters, which control the movement and animation of the object.

[0098] For example, the target image refers to the appearance and state that a virtual object or character should present. It is determined based on the following factors: Voice content: The virtual character needs to adjust its performance based on the user's speech content. For example, if the user says "I am happy", the virtual character's mouth shape will match "I am happy"; Emotional data: By analyzing the user's voice or other biometrics (such as heart rate, skin conductance, etc.), the user's emotional state can be inferred, and the virtual character's emotional expression can be adjusted. The virtual character may show a smiling expression to achieve emotional synchronization.

[0099] The embodiments of this application use adjusted controller parameters to render virtual objects in a virtual environment, rendering them as target images. This can be applied to the following scenarios: virtual assistant scenarios: When responding to user queries, the virtual assistant will display body language and expressions that match the answers; virtual social scenarios: In a virtual social environment, the user's virtual image will display corresponding behaviors based on their emotions and conversations; game character scenarios: In-game characters can display corresponding reactions based on the player's behavior and emotions. By precisely controlling the presentation of virtual objects, the quality of the user experience can be greatly improved, making interactions more natural and realistic.

[0100] 3B , which is an optional flow chart of an artificial intelligence-based data processing method according to an embodiment of the present application. In some embodiments, step 104 shown in FIG3A can be implemented by steps 1041 and 1042 shown in FIG3B , which are described in detail below.

[0101] In step 1041, the fusion features are subjected to feedforward processing based on the attention mechanism to obtain animation parameter features corresponding to the speech.

[0102] In some embodiments, step 1041 can be implemented in the following manner: through the nth feedforward network in the N cascaded feedforward networks, the input of the nth feedforward network is mapped based on the attention mechanism to obtain the nth mapping result, and the nth mapping result output by the nth feedforward network is transmitted to the n+1th feedforward network to continue the mapping process based on the attention mechanism to obtain the n+1th mapping result corresponding to the n+1th feedforward network; the Nth mapping result output by the Nth feedforward network is used as the animation parameter feature of the corresponding speech.

[0103] As an example, the feedforward processing based on the attention mechanism is implemented through an attention feedforward network, which includes N cascaded feedforward networks, where N is an integer greater than 1, n is an integer variable whose value increases from 1, n is not less than 1 and less than N, when n is 1, the input of the nth feedforward network is the fusion feature, and when n is not less than 2 and less than N, the input of the nth feedforward network is the n-1th mapping result output by the n-1th feedforward network.

[0104] As an example, for the first feedforward network, the input of the first feedforward network (for the first feedforward network, its input is the fusion feature H corresponding to speech A1) c +H e ) performs feedforward processing based on the attention mechanism to obtain a first feedforward result, and transmits the first feedforward result output by the first feedforward network to the second feedforward network to continue the mapping processing based on the attention mechanism to obtain a second feedforward result corresponding to the second feedforward network. And so on, the Nth mapping result output by the Nth feedforward network is used as the animation parameter feature corresponding to the voice A1. The feedforward network here can be a feedforward self-attention (Feed-Forward Transformer, FFT) network.

[0105] By using N cascaded attention feedforward networks, the fused features are subjected to multiple feedforward processing based on the attention mechanism to obtain the animation parameter features corresponding to the speech, making the speech and emotional information contained in the animation parameter features richer and more accurate. In turn, the controller parameters obtained based on the emotional data and speech can meet actual needs, so that the final generated facial animation matches the speech and speech emotion.

[0106] In some embodiments, when the value of n is not less than 2 and less than N, the above-mentioned mapping processing based on the attention mechanism on the input of the nth feedforward network to obtain the nth mapping result can be achieved in the following ways: performing attention processing on the n-1th mapping result to obtain the nth attention feature; performing addition processing on the n-1th mapping result and the nth attention feature to obtain the nth intermediate feature; performing convolution processing on the nth intermediate feature to obtain the nth convolution feature; performing addition processing on the nth convolution feature and the nth intermediate feature to obtain the nth mapping result.

[0107] As an example, for the first feedforward network, the fusion feature H c +H e Perform attention processing to obtain the first attention feature and perform the fusion feature H c +H eThe first intermediate feature is added to the first attention feature to obtain the first convolution feature; the first convolution feature and the first intermediate feature are added to obtain the first feedforward result; the first feedforward result is input into the second feedforward network, and the second feedforward network performs the following processing: attention processing is performed on the first feedforward result to obtain the second attention feature, the first feedforward result and the second attention feature are added to obtain the second intermediate feature, convolution processing is performed on the second intermediate feature to obtain the second convolution feature; the second convolution feature and the second intermediate feature are added to obtain the second feedforward result.

[0108] By performing residual processing on the n-1th mapping result and the nth attention feature after attention processing in the feedforward network, and performing residual processing on the nth intermediate feature and the nth convolution feature after convolution processing, the gradient vanishing problem is avoided, the expressive power of the feedforward network is improved, and the voice information and emotional information contained in the final animation parameter features are made more accurate, so that the controller parameters obtained according to the emotional data and voice can meet the actual requirements, so that the final generated facial animation matches the voice and voice emotion.

[0109] In some embodiments, when n takes a value not less than 2 and less than N, the above-mentioned attention processing is performed on the n-1th mapping result to obtain the nth attention feature, which can be achieved in the following way: obtaining the query matrix, key matrix and value matrix of each attention network; performing the following processing through each attention network: based on the query matrix, key matrix and value matrix of the attention network, single-head attention processing is performed on the n-1th mapping result to obtain the single-head attention result corresponding to the attention network; the single-head attention results corresponding to multiple attention networks are fused to obtain the nth attention feature.

[0110] In some embodiments, the nth feedforward network includes multiple attention networks. For each attention network, the query weight matrix, key weight matrix and value weight matrix of the attention matrix are obtained, and the query weight matrix, key weight matrix and value weight matrix are respectively multiplied with the n-1th mapping result to obtain the query matrix, key matrix and value matrix of the attention network. First, the dot product of the query matrix and the key matrix is ​​calculated, and then the dot product of the query matrix and the key matrix is ​​divided by the square root of the dimension of the key vector in the key matrix to obtain a similarity score matrix. The similarity score matrix is ​​normalized to obtain a normalized matrix, and the normalized matrix is ​​matrix multiplied with the value matrix to obtain the single-head attention result corresponding to the attention network. The single-head attention results corresponding to the multiple attention networks in the nth feedforward network are fused to obtain the nth attention feature of the nth feedforward network.

[0111] By setting multiple attention networks in the feedforward network, performing multi-head attention processing on the feedforward results, calculating multiple attention matrices, and fusing the attention results corresponding to multiple attention networks, the accuracy of the attention features is improved, and the voice and emotional information contained in the final animation parameter features are made more accurate, thereby making the controller parameters obtained according to the emotional data and voice meet the actual requirements, so that the final generated facial animation matches the voice and voice emotion.

[0112] The attention score is calculated by comparing the query matrix with the key matrix, typically through methods such as dot products or additive models. The calculated attention score is normalized to generate an attention distribution, typically implemented using a softmax function. The final output is generated based on the attention distribution and the value matrix. The output is a weighted sum of the input and the attention distribution, thereby improving the feature representation capability of the attention feature.

[0113] In step 1042, the animation parameter features are fully connected to obtain controller parameters corresponding to the voice.

[0114] As an example, the controller parameters are used to control the virtual object model to present the target image, which matches the content and emotional data of the speech. Through the fully connected layer of the parameter-driven model, according to formula (5), the animation parameter features corresponding to speech A1 are fully connected to obtain the controller parameters corresponding to speech A1.

[0115] Among them, H am is the animation parameter feature corresponding to speech A1, Linear(H am ) is the fully connected layer for the animation parameter feature H am The output result obtained by performing full connection processing.

[0116] By performing multi-head self-attention feedforward processing on the fused features, animation parameter features containing accurate information about the corresponding speech content and speech emotions are obtained. The animation parameter features are then mapped to controller parameters of the corresponding speech, so that the facial expressions of the virtual object model can be directly adjusted according to the controller parameters without adjusting the vertices of the virtual object model. On the one hand, facial animation corresponding to the speech and emotion data can be generated. On the other hand, it is convenient for animation producers to operate and improve the efficiency of facial animation production.

[0117] In some embodiments, the plurality of speech frames are divided into a plurality of speech frame sequences based on a temporal sequence of the plurality of speech frames in the speech.

[0118] As an example, the speech includes multiple speech frames, the emotion data includes multiple emotion intensity sub-matrices, and the controller parameters include multiple parameter combinations. Each of the speech frame sequences corresponds to each of the emotion intensity sub-matrices, and each of the speech frame sequences corresponds to each of the control parameters. For example, for speech A={a 1 ,…,a T}, where T is the number of audio samples. Taking 16 kHz audio as an example, the number of speech frames in 1 second of speech is 16,000. x (x=1, ..., T) is the speech signal of the xth sampling point, and the real controller parameter corresponding to speech A is Y={y 1 ,…,y M}, M is the number of animation frames, generally 50 frames per second, y i (i=1,…,M) is the real controller parameter corresponding to the i-th animation frame. It can be seen that the speech frame and the real control parameter are not in a one-to-one correspondence. Since the number of speech frames is more than the number of animation frames, the speech frames need to be divided according to the time series, and the speech frames are divided into speech frame sequences corresponding to the animation frames, and each speech frame sequence has a corresponding emotion intensity submatrix for characterizing the emotion intensity. In this way, the problem of predicting the controller parameters based on the speech and the emotion data corresponding to the speech can be refined into the problem of predicting the real control parameters of the corresponding animation frame based on each speech frame sequence and the emotion intensity submatrix corresponding to each speech frame sequence.

[0119] By dividing multiple speech frames of speech into speech frame sequences corresponding to animation frames, and predicting the actual control parameters of the corresponding animation frames based on each speech frame sequence and the emotion intensity submatrix corresponding to each speech frame sequence, each animation frame can reflect the speech content and speech emotion contained in the speech frame sequence, making the facial animation of the generated virtual object image more detailed and smoother, and improving the performance of the facial animation.

[0120] In some embodiments, multiple emotion intensity values ​​corresponding to each speech frame sequence are obtained; the multiple emotion intensity values ​​are matrix-processed to obtain an emotion intensity sub-matrix corresponding to each speech frame sequence; and the multiple emotion intensity sub-matrices corresponding to the multiple speech frame sequences are combined into emotion data.

[0121] As an example, each emotion intensity submatrix includes multiple emotion intensity values ​​corresponding to multiple emotions in each speech frame sequence. For example, each emotion intensity submatrix represents six emotions corresponding to joy, anger, sadness, surprise, fear, and disgust, and then it has six emotion intensity values ​​e corresponding to these six emotions. 喜 、e 怒 、e 哀 、e 惊 、e 恐 、e 厌, the six emotion intensity values ​​are matrixed according to certain rules to obtain the emotion intensity submatrix e 子 ={e 喜 , e 怒 , e 哀 , e 惊 , e 恐 , e 厌 Then, based on the time series relationship between the speech frame sequences corresponding to the emotion intensity sub-matrices, multiple emotion intensity sub-matrices are combined to obtain the emotion data of the corresponding speech.

[0122] By representing the emotion intensity submatrix with emotion intensity values ​​corresponding to different emotions, on the one hand, complex emotions can be combined and represented in matrix form, making the emotions represented by the emotion intensity submatrix corresponding to the speech frame sequence richer and more accurate. On the other hand, representing the emotion intensity in a matrix manner is easy to understand and facilitates adjustment of the emotion data according to actual needs, thereby ensuring that the controller parameters obtained based on the emotion data and speech can meet actual needs, so that the final generated facial animation matches the speech and speech emotion.

[0123] See Figure 4, which is a schematic diagram of the training process of the emotion extraction network provided by the embodiment of the present application. In some embodiments, before executing step 101, steps 201 to 205 shown in Figure 4 may also be executed, which will be described in detail below.

[0124] In step 201, a first speech sample is obtained, and an emotion data label corresponding to the first speech sample is obtained.

[0125] As an example, the first voice sample can be an audio file containing voice content. The emotion data label corresponding to the first voice sample is used to characterize the corresponding emotion of the first voice sample. For example, for the first voice sample A2 "I don't like eating apples", the emotion expressed is disgust, so the emotion data label corresponding to the first voice sample A2 is e A2 ={e 喜 , e 怒 , e 哀 , e 惊 , e 恐 , e 厌}={0,0,0,0,0,1}, where the emotion data label is e A2 Characterizes the emotional intensity corresponding to only one emotion, namely the emotional intensity of disgust (e 厌 =1, and the emotional intensity of other emotions is 0). In actual use, it can also be used in the emotional data tag e A2 Characterizes the emotional intensity corresponding to various emotions, which is not limited in this application.

[0126] In step 202, the initialized emotion extraction network performs second speech content feature extraction processing on the first speech sample to obtain a first sample speech content feature corresponding to the first speech sample.

[0127] As an example, the second speech content feature extraction network in the initialized emotion extraction network performs the second speech content feature extraction process on the first speech sample A2 to obtain the first sample speech content feature H corresponding to the first speech sample A2. A2 The implementation of this step is the same as that of step 101 and will not be repeated here.

[0128] In step 203, the initialized emotion extraction network performs emotion data mapping processing on the first sample speech content features to obtain predicted emotion data corresponding to the first speech sample.

[0129] As an example, the emotion recognition network in the initialized emotion extraction network is used to analyze the first sample speech content feature H A2 Perform emotional data mapping processing and obtain the predicted emotional data P corresponding to the first speech sample A2 according to formula (6): P = softmax(linear(H A2 )) (6)

[0130] Among them, linear(H A2 ) is the first sample speech content feature H of the fully connected layer A2 The second linear mapping result obtained by full connection processing, softmax(linear(H A2 )) is the normalization layer (softmax layer) for the second linear mapping result linear(H A2 ) is the output result obtained by performing maximum likelihood processing, namely the predicted emotion data P corresponding to the first speech sample A2.

[0131] In step 204 , a first loss function is determined based on the emotion data label and the predicted emotion data corresponding to the first speech sample.

[0132] As an example, according to formula (7), based on the emotion data label e corresponding to the first speech sample A2 A2 And the predicted emotion data P, determine the first loss function L': L'=CE(e A2 ,P) (7)

[0133] Among them, CE(e A2 , P) is based on the emotion data label e A2 The cross entropy loss function calculated with the predicted emotion data P, the first loss function may also be other types of loss functions, which is not limited in this application.

[0134] In step 205, the initialized emotion extraction network is updated based on the first loss function to obtain a trained emotion extraction network.

[0135] As an example, based on the first loss function L′, the initialized emotion extraction network is updated to obtain a trained emotion extraction network. The trained emotion extraction network can then execute the second speech content feature extraction process on the speech in step 101 to obtain the second speech content feature of the corresponding speech, and perform emotion data mapping on the second speech content feature to obtain the emotion data of the corresponding speech. The embodiment of the present application does not limit the network structure of the emotion extraction network, that is, it can also be other network structures.

[0136] By using the first voice sample, the initialized emotion extraction network is updated so that the trained emotion extraction network can automatically extract corresponding emotion data from the voice to generate facial animation of the virtual object image that matches the voice and emotion.

[0137] 5A , which is a flow chart of the training of the parameter-driven model provided in an embodiment of the present application. In some embodiments, before executing step 101 , steps 301 to 305 shown in FIG. 5A may also be executed, which will be described in detail below.

[0138] In step 301, training samples and true controller parameters of the training samples are obtained.

[0139] As an example, the training sample includes a second speech sample and an emotion data sample corresponding to the second speech sample. The second speech sample A3 can be an audio file containing speech content, and the emotion data sample corresponding to the second speech sample A3 is used to characterize the corresponding emotion of the second speech sample A3. For example, for the second speech sample A3 "I don't like eating apples", the emotion expressed is disgust, so the emotion data sample corresponding to the second speech sample A3 is e A3 ={e 喜 , e 怒 , e 哀 , e 惊 , e 恐 , e 厌}={0, 0, 0, 0, 1}, where the emotion data sample e A3 Characterizes the emotional intensity corresponding to only one emotion, namely the emotional intensity of disgust (e 厌 =1, and the emotional intensity of other emotions is 0). In actual use, the emotional data sample e A3 Characterize the emotional intensity corresponding to multiple emotions, which is not limited in this application. Corresponding to the second voice sample A3 and the emotional data sample e A3The real controller parameters are standard controller parameters that can reflect the speech content and emotion of the second speech sample A3.

[0140] The model is driven by the initialized parameters to perform the following processing:

[0141] In step 302, the first speech content feature extraction process is performed on the second speech sample to obtain a second sample speech content feature corresponding to the second speech sample, and the emotion feature extraction process is performed on the emotion data sample to obtain a sample emotion feature corresponding to the second speech sample.

[0142] As an example, the first speech content feature extraction network in the initialized parameter-driven model is used to perform first speech content feature extraction processing on the second speech sample A3. The first speech content feature extraction network here can be a wav2vec2 model, or other models that can be used to extract speech content features. Here, the first speech content feature extraction network is taken as an example of a wav2vec2 model. By using formula (8), the first speech content feature extraction processing is performed on the second speech sample A3 to obtain the second sample speech content feature H corresponding to the second speech sample A3. c ′: H c ′={wav2vec2(A3)} (8)

[0143] Among them, wav2vec2(A3) is the output obtained by the wav2vec2 model performing the first speech content feature extraction process on the second speech sample A3.

[0144] As an example, the emotion feature extraction network in the model is driven by the initialized parameters. According to formula (9), the emotion data sample e corresponding to the second speech sample A3 is A3 Perform emotion feature extraction processing to obtain the sample emotion feature H corresponding to the second voice sample A3 e ′:

[0145] Among them, e i is the emotion data sample e A3 The i-th dimension emotion data in the q-dimensional emotion data, EMB(e i ) is the emotion feature extraction network corresponding to e i After fusing the q emotion feature extraction results, we can get the sample emotion feature H e ′.

[0146] In step 303, the second sample speech content feature and the sample emotion feature are fused to obtain a sample fusion feature corresponding to the second speech sample.

[0147] As an example, the second sample speech content feature H corresponding to the second speech sample A3 c ′ and sample emotion feature H e ' Perform fusion processing to obtain the sample fusion feature H corresponding to the second speech sample A3 c ′+H e The fusion processing here may be a superposition processing, a splicing processing, or other processing methods that can fuse different features, and this application does not limit this.

[0148] In step 304, animation parameter mapping processing is performed on the sample fusion features to obtain prediction controller parameters corresponding to the second speech sample.

[0149] As an example, the sample fusion feature H c ′+H e Perform feedforward processing based on the attention mechanism to obtain animation parameter features corresponding to the second voice sample A3, and perform full connection processing on the animation parameter features to obtain prediction controller parameters corresponding to the second voice sample A3.

[0150] In step 305 , a second loss function is determined based on the predicted controller parameters and the actual controller parameters, and the initialized parameter-driven model is updated based on the second loss function to obtain a trained parameter-driven model.

[0151] Referring to FIG. 5B , FIG. 5B is a flowchart illustrating an optional training process for a parameter-driven model according to an embodiment of the present application. In some embodiments, determining the second loss function based on the predicted controller parameters and the actual controller parameters in step 305 of FIG. 5A can be implemented through steps 3051 to 3054 shown in FIG. 5B , which are described in detail below.

[0152] In step 3051, the real control parameters corresponding to each animation frame are extracted from the real controller parameters.

[0153] As an example, the actual controller parameter is Y = {y 1 ,…,y M}, where M is the number of animation frames, y i is the real control parameter corresponding to the i-th animation frame in the real controller parameter.

[0154] In step 3052, the prediction control parameters corresponding to each animation frame are extracted from the prediction controller parameters.

[0155] As an example, the predictive controller parameters are Among them, M is the number of animation frames, is the predicted control parameter corresponding to the i-th animation frame in the predicted controller parameter.

[0156] In step 3053 , the following processing is performed for each animation frame: determining the difference between the actual control parameter of the corresponding animation frame and the predicted control parameter of the corresponding animation frame.

[0157] As an example, calculate the true control parameter y of the i-th animation frame i and predictive control parameters between By analogy, multiple differences between the actual control parameters and the predicted control parameters of multiple animation frames are obtained.

[0158] As an example, the difference here can be an L1 norm difference or an L2 norm difference. The L1 norm difference is an absolute value, and the L2 norm difference is the sum of squares of the differences of the control parameters corresponding to each type.

[0159] In step 3054, the differences corresponding to the multiple animation frames are fused to obtain a second loss function.

[0160] As an example, the difference values ​​can be directly fused to obtain the second loss function, and the square of the difference values ​​can also be fused to obtain the second loss function.

[0161] As an example, multiple differences between the actual control parameters and the predicted control parameters of multiple animation frames are put into the fusion process to obtain the second loss function L″. Since Y={y 1 ,…,y M}, will correspond to multiple animation frames The fusion process can be expressed by formula (10):

[0162] Where Y is the actual controller parameter corresponding to the second speech sample A3, is the prediction controller parameter corresponding to the second speech sample A3. The initialized parameter-driven model is updated based on the second loss function L″ to obtain a trained parameter-driven model.

[0163] By calculating the second loss function based on the difference between the actual control parameters and the predicted control parameters of each animation frame, the initialized parameter-driven model is updated and trained. This is equivalent to using multiple sub-samples to train the initialized parameter-driven model at one time, thereby improving the training efficiency and accuracy of the parameter-driven model to ensure that the controller parameters predicted by the parameter-driven model based on the speech and emotion data match the speech and emotion data, thereby generating facial animation of the virtual object image that matches the speech and emotion data.

[0164] The initialized parameter-driven model is trained by using the second voice sample, and a second loss function is determined based on the predicted controller parameters and the actual controller parameters to update the initialized parameter-driven model to obtain a parameter-driven model, so as to ensure that the controller parameters predicted by the parameter-driven model based on the voice and emotion data match the voice and emotion data, thereby generating facial animation of the virtual object image that matches the voice and emotion data.

[0165] Below, an exemplary application of the embodiment of the present application in an actual virtual human live broadcast application scenario will be described.

[0166] See Figure 6, which is a background service flow chart of the voice-face driven service provided by the embodiment of the present application. As shown in Figure 6, the front-end module generates voice according to the input, and the voice-driven facial service platform performs controller parameter prediction processing on the voice by calling the parameter-driven model provided by the embodiment of the present application, and obtains the controller parameters of the corresponding virtual object model. The type of the front-end module is determined according to actual needs. For example, in the application scenario of the following virtual human live broadcast, the front-end module can be a module that integrates a dialogue generation service module and a speech synthesis service module. In the application scenario of the following facial animation production, the front-end module can also be a voice file input module. The type of the front-end module is selected according to the type of upstream task. As long as its output is voice, this application does not impose any restrictions on this.

[0167] See Figure 7, which is a flow chart of the background service of the virtual human live broadcast provided by the embodiment of the present application. As shown in Figure 7, in the virtual human live broadcast scene, the bullet comments in the live broadcast scene are captured, and the dialogue content of the response bullet comments is generated by the dialogue generation service module. The dialogue content is converted into speech by the speech synthesis service module, and the facial animation parameters corresponding to the speech are generated by the speech-driven facial service module, that is, the controller parameters. Finally, the virtual human is driven to make corresponding facial animations according to the facial animation parameters, and interacts with the bullet comments on the client in conjunction with the speech. Among them, the speech-driven facial service module includes the parameter-driven model provided by the embodiment of the present application.

[0168] Refer to Figure 8, which is a schematic diagram of the architecture of the parameter-driven model provided in an embodiment of the present application. As shown in Figure 8, the input of the parameter-driven model is a voice signal and emotion data. One voice signal corresponds to one emotion data. For example, for the voice signal a1, the corresponding emotion data is [0.1, 0.2, 0.3, 0.4, 0.1, 0.05], and the emotion data is one-dimensional. The output of the parameter-driven model is the facial animation parameters corresponding to the binding animation drive system, specifically the controller parameters. Among them, the input emotion data can be manually specified by the user, or it can be directly extracted from the speech using the emotion recognition module. For the input voice signal, the voice content features are first extracted from the voice signal through the voice content feature extraction network. Then, the emotion data is converted into emotion features through the emotion feature extraction network. The speech content features and emotional features are superimposed to obtain superimposed features. These superimposed features are then fed into a multi-layer neural network model, which includes N cascaded feed-forward transformer (FFT) modules. The multi-layer neural network model performs feed-forward self-attention processing on the superimposed features. Finally, the results of the feed-forward self-attention processing are regressed through a linear prediction layer to obtain the final facial animation parameters, i.e., controller parameters. The animation drive system drives the faces of multiple virtual human characters based on the controller parameters, thereby achieving driving effects for multiple different characters. The following describes the operation process of the parameter-driven model in detail.

[0169] The parameter-driven model mainly consists of two parts: the first speech content feature extraction network and the emotion feature extraction network. During the model training process, the emotion data samples input into the emotion feature extraction network are manually labeled based on speech and animation data. During the training phase, each second speech sample has only one emotion, and the intensity is 1, that is, the speech is manually labeled with only one emotion from [joy, anger, sadness, surprise, fear, disgust, nothing]. If the labeled emotion is joy, the corresponding emotion data sample is [1, 0, 0, 0, 0, 0], and so on. Let the speech signal corresponding to the second speech sample be A = {a 1 ,…,a T}, where T is the number of audio sampling points. Taking 16 kHz audio as an example, the number of samples for 1 second of audio is 16,000. x (x=1, ..., T) is the speech signal of the xth sampling point, and the real controller parameter corresponding to the speech signal A is Y={y 1 ,…,y M}, M is the number of animation frames, generally 50 frames per second, y i(i=1, ..., M) is the real controller parameter corresponding to the i-th animation frame. The speech signal corresponding to the second speech sample is input into the first speech content feature extraction network (the first speech content feature extraction network can be a wav2vec2 model or other models that can be used to extract speech content features). The second sample speech content feature H is obtained by formula (11) c : H c ={wav2vec2(A)} (11)

[0170] in, is the second sample speech content feature corresponding to the i-th animation frame, wav2vec2(A) is the wav2vec2 model for the speech signal A={a 1 ,…,a T After obtaining the second sample speech content feature, it is necessary to input the manually labeled emotion data sample e into the emotion feature extraction network, e∈{joy, anger, sadness, surprise, fear, disgust}, corresponding to 6 emotions, and obtain the sample emotion feature H through formula (12) e , where the emotion feature extraction network has only a fully connected layer, and the output of the emotion feature extraction network is the sample emotion feature H e :

[0171] Among them, e i is the i-th dimension emotion data in the 6-dimensional emotion data of the emotion data sample e, EMB(e i ) is the emotion feature extraction network corresponding to e i After fusing the six emotion feature extraction results, we get the sample emotion feature H e .

[0172] Since there is only one non-zero value in the manually labeled emotion data sample e, during the training phase, formula (12) can be written as formula (13): H e =EMB(e) (13)

[0173] Among them, EMB(e) is the output of the emotion feature extraction network with emotion data sample e as input.

[0174] Then, the second sample speech content feature and the sample emotion feature are fused. The fusion here can be addition or splicing to obtain the sample fusion feature H e +H c , and the sample fusion feature H e +H cInput the multi-layer feed-forward self-attention (FFT) module to obtain the sample animation features. Finally, through the fully connected layer, the sample animation features are mapped to the prediction controller parameters according to formula (14)

[0175] Here, FFT is the output of the FFT module, and Linear is the output of the fully connected layer. Continuing with Figure 8, the architecture of the feedforward self-attention (FFT) module is shown on the right side of Figure 8. Each FFT module mainly consists of a multi-head attention layer and a convolution layer. The original input of the FFT module first passes through the multi-head attention layer for multi-head attention processing. After obtaining the intermediate result, the original input and the intermediate result are subjected to residual processing to obtain an intermediate input. The intermediate input is convolved through the convolution layer to obtain the convolution result. The convolution result and the intermediate input are subjected to residual processing to obtain the final output of the FFT module.

[0176] Get the predictive controller parameters After that, the loss function L1 is calculated by formula (15), and the parameter-driven model is trained according to the second loss function L1:

[0177] Where Y is the actual controller parameter corresponding to the speech signal A, is the prediction controller parameter corresponding to the speech signal A. By updating the parameters of the parameter-driven model, we finally get a parameter-driven model whose input is the emotion label (i.e., emotion data) and speech signal and whose output is the controller parameter.

[0178] In the above training process, the emotion data samples used are manually labeled. However, in the application process, it is sometimes not possible to provide emotion data corresponding to the speech through manual labeling. In this case, the emotion extraction network can be used to automatically extract the emotion from the speech to obtain the emotion data corresponding to the speech. The details are explained below.

[0179] It is necessary to build an emotion extraction network for automatic speech emotion extraction. The emotion extraction network includes a wav2vec2 module for extracting speech content features as the second speech content feature extraction network (this part can also be other models that can be used to extract speech content features), a fully connected layer, and a normalization layer (softmax layer). In the emotion extraction network training process, it is necessary to first build a parallel data pair of the first speech sample and the emotion data label.<A,e> , where A is the first speech sample, e is the emotion data label corresponding to the first speech sample, e∈{joy, anger, sadness, surprise, fear, disgust}, input the speech A into the wav2vec2 module of the emotion extraction network, and obtain the first sample speech content feature H according to formula (16)a : H a =wav2vec2(A) (16)

[0180] Among them, wav2vec2 is the output of the wav2vec2 module.

[0181] Then, the first sample speech content feature H a Input the fully connected layer and the normalization layer (softmax layer), and obtain the distribution probability of the six emotions according to formula (17), that is, the predicted emotion data P: P = softmax(linear(H a )) (17)

[0182] Among them, linear(H a ) is the first sample speech content feature H of the fully connected layer a The second linear mapping result obtained by full connection processing, softmax(linear(H a )) is the second linear mapping result linear(H a ) is the output result obtained by maximum likelihood processing, that is, the predicted emotion data P corresponding to the first speech sample, P = {p 喜 , p 怒 , p 哀 , p 惊 , p 恐 , p 厌}.

[0183] Finally, the cross entropy loss function L2 (i.e., the first loss function) is calculated by formula (18) to train the emotion extraction network and achieve network optimization: L2 = CE(e, P) (18)

[0184] Among them, e is the emotion data label e∈{joy, anger, sadness, surprise, fear, disgust} corresponding to the first language sample, P is the predicted emotion data corresponding to the first language sample, P={p 喜 , p 怒 , p 哀 , p 惊 , p 恐 , p 厌}, CE(e, P) is the cross entropy loss function calculated based on the emotion data label e and the predicted emotion data P.

[0185] The final predicted emotion data P can be used as the emotion data of the input parameter driving model on the application side.

[0186] In actual application, a voice and the emotion data corresponding to the voice can be directly input into the parameter-driven model. The emotion data can be the emotion data extracted by the emotion extraction network, or the emotion data obtained by the user after customized adjustment of the results extracted by the emotion extraction network, or the emotion data customized by the user. Table 1 is an example table of emotion data.

[0187] Table 1: Example table of sentiment data

[0188] Through the emotion feature extraction network, the emotion data is processed by emotion feature extraction. According to formula (19), the emotion feature H corresponding to the emotion data is obtained. e :

[0189] Among them, e i is the ith dimension of the 6-dimensional emotional data, EMB(e i ) is the emotion feature extraction network corresponding to e i After fusing the six emotion feature extraction results, we get the emotion feature H e .

[0190] At this point, the first speech content features and emotional features corresponding to the speech are obtained. After the first speech content features and emotional features are fused, they are input into the multi-layer FFT module for feedforward self-attention processing and mapped through the fully connected layer, and finally the controller parameters corresponding to the speech are obtained.

[0191] Below, an exemplary application of the embodiment of the present application in an actual facial animation production application scenario will be described.

[0192] After obtaining the voice and the corresponding emotional data, the parameter-driven model is called to perform controller parameter prediction processing on the voice and the corresponding emotional data to obtain the controller parameters corresponding to the voice. The animation-driven system adjusts the controller parameters of the virtual object facial model according to the controller parameters, controls the virtual object model to present the target image, and obtains the target facial expression that matches the voice and emotional data.

[0193] Refer to Figure 9, which is a schematic diagram of the character animation production page provided by an embodiment of the present application. As shown in Figure 9, voice A is sent to the server by uploading an audio file. The emotion extraction network and parameter-driven model deployed in the server process voice A, obtain the controller parameters corresponding to voice A, and feed back the controller parameters to the terminal. The character animation production page of the terminal displays controllers corresponding to different emotions, including six controllers corresponding to joy, anger, sadness, surprise, fear, and disgust, respectively. The six controllers corresponding to the six emotions of joy, anger, sadness, surprise, fear, and disgust in the emotional data are adjusted according to the controller parameters of the corresponding voice, thereby achieving adjustment of the character animation. Specifically, the terminal receives the user's upload operation for voice A, obtains voice A, and displays the audio file as voice A in the interactive interface 901 in the character animation production page. The animation generation panel of the interactive interface 901 includes a manual mode control 902 and an automatic mode control 903. In response to a selection operation on manual mode control 902, manual mode control 902 is in a selected state, and the emotion controller 904 in the facial animation generation panel enters an editable state. At this time, the user can slide the progress bar of the emotion controller for any emotion. For example, in response to a sliding operation on the progress bar of the six emotion controllers corresponding to the six emotions of joy, anger, sadness, surprise, fear, and disgust, six emotion intensity data corresponding to the six emotions of joy, anger, sadness, surprise, fear, and disgust are displayed, and the six emotion intensity data are used to form the emotion data corresponding to voice A. The terminal obtains voice A and the emotion data and sends them to the server. The server calls a parameter-driven model, performs parameter prediction processing on voice A and the emotion data, and obtains controller parameters corresponding to voice A and the emotion data. The terminal drives the character facial model to present a target image corresponding to voice A and the emotion data based on the controller parameters (the target image in this case is the target facial expression and lip shape of the character facial model), and presents the target image of the character facial model in the display area of ​​the interactive interface 901. In response to the selection operation for the automatic mode control 903, the automatic mode control 903 is in the selected state, the terminal sends voice A to the server, the server calls the emotion extraction network to perform emotion extraction processing on voice A, and obtains the emotion data corresponding to voice A. The server calls the parameter-driven model to perform parameter prediction processing on voice A and emotion data, and obtains the controller parameters corresponding to voice A and emotion data. The terminal drives the character facial model to present the target image corresponding to voice A and emotion data according to the controller parameters, and displays it in the display area of ​​the interactive interface 901. In response to the trigger operation for generating facial animation control, facial animation of the character facial model is generated, that is, head animation of the character facial model can also be generated.When generating facial animation, the controller parameters include the expression and lip shape control parameters of the character's facial model, and when generating the head animation of the character's facial model, the controller parameters also include the control parameters of the head movements of the corresponding character's facial model, such as nodding, shaking head, tilting head and other head movements, and presenting the target image of the character's facial model in the display area of ​​the interactive interface 901, and generating head animation in response to the trigger operation for generating the head animation control.

[0194] The embodiment of the present application has a smoother driving effect and adopts a solution of outputting controller parameters, which can effectively improve the developer's user experience. Through automatic emotion extraction of voice, different emotional animations can be automatically adapted according to the input voice.

[0195] The following continues to describe an exemplary structure of the artificial intelligence-based data processing device 253 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in Figure 2, the software modules stored in the artificial intelligence-based data processing device 253 in the memory 250 may include: an acquisition module 2531, configured to acquire speech and acquire emotional data corresponding to the speech; a feature extraction module 2532, configured to perform a first speech content feature extraction process on the speech to obtain a first speech content feature of the corresponding speech, and perform an emotional feature extraction process on the emotional data to obtain an emotional feature of the corresponding speech; an overlay processing module 2533, configured to perform a fusion process on the first speech content feature and the emotional feature to obtain a fusion feature of the corresponding speech; a mapping processing module 2534, configured to perform animation parameter mapping processing on the fusion feature to obtain controller parameters of the corresponding speech, wherein the controller parameters are used to control the virtual object model to present a target image, and the target image matches the content of the speech and the emotional data.

[0196] In some embodiments, the acquisition module 2531 is further configured to perform any one of the following processes: obtaining emotional data input by the object; performing second speech content feature extraction processing on the speech to obtain the second speech content feature of the corresponding speech; performing emotional data mapping processing on the second speech content feature to obtain emotional data of the corresponding speech.

[0197] In some embodiments, the acquisition module 2531 is further configured to acquire a first voice sample and acquire an emotion data label corresponding to the first voice sample; perform a second voice content feature extraction process on the first voice sample through the initialized emotion extraction network to obtain a first sample voice content feature corresponding to the first voice sample; perform an emotion data mapping process on the first sample voice content feature through the initialized emotion extraction network to obtain predicted emotion data corresponding to the first voice sample; determine a first loss function based on the emotion data label and the predicted emotion data corresponding to the first voice sample; and update the initialized emotion extraction network based on the first loss function to obtain a trained emotion extraction network.

[0198] In some embodiments, the acquisition module 2531 is further configured to perform full connection processing on the second speech content feature to obtain a first linear mapping result; and perform maximum likelihood processing on the first linear mapping result to obtain emotion data of the corresponding speech.

[0199] In some embodiments, the feature extraction module 2532 is further configured to obtain training samples and real controller parameters of the training samples, wherein the training samples include a second speech sample and an emotion data sample corresponding to the second speech sample; the following processing is performed through the initialized parameter-driven model: the second speech sample is subjected to first speech content feature extraction processing to obtain a second sample speech content feature corresponding to the second speech sample, and the emotion data sample is subjected to emotion feature extraction processing to obtain a sample emotion feature corresponding to the second speech sample; the second sample speech content feature and the sample emotion feature are subjected to fusion processing to obtain a sample fusion feature corresponding to the second speech sample; the sample fusion feature is subjected to animation parameter mapping processing to obtain a predicted controller parameter corresponding to the second speech sample; based on the predicted controller parameters and the real controller parameters, a second loss function is determined, and the initialized parameter-driven model is updated based on the second loss function to obtain a trained parameter-driven model.

[0200] In some embodiments, the feature extraction module 2532 is further configured to extract real control parameters corresponding to each animation frame from the real controller parameters; extract predicted control parameters corresponding to each animation frame from the predicted controller parameters; perform the following processing for each animation frame: determine the difference between the real control parameters of the corresponding animation frame and the predicted control parameters of the corresponding animation frame; fuse the differences corresponding to multiple animation frames to obtain a second loss function.

[0201] In some embodiments, the mapping processing module 2534 is configured to perform feedforward processing based on the attention mechanism on the fusion features to obtain animation parameter features corresponding to the speech; and perform full connection processing on the animation parameter features to obtain controller parameters corresponding to the speech.

[0202] In some embodiments, the mapping processing module 2534 is further configured to perform an attention-based mapping process on the input of the nth feedforward network through the nth feedforward network in the N cascaded feedforward networks to obtain an nth mapping result, and transmit the nth mapping result output by the nth feedforward network to the n+1th feedforward network to continue the attention-based mapping process to obtain the n+1th mapping result corresponding to the n+1th feedforward network; use the Nth mapping result output by the Nth feedforward network as the animation parameter feature of the corresponding speech; wherein, n is an integer variable whose value increases from 1, n is not less than 1 and less than N, when n is 1, the input of the nth feedforward network is the fusion feature, when n is not less than 2 and less than N, the input of the nth feedforward network is the n-1th mapping result output by the n-1th feedforward network; the feedforward processing based on the attention mechanism is implemented by the attention feedforward network, and the attention feedforward network includes N cascaded feedforward networks, and N is an integer greater than 1.

[0203] In some embodiments, when the value of n is not less than 2 and less than N, the mapping processing module 2534 is further configured to perform attention processing on the n-1th mapping result to obtain the nth attention feature; perform addition processing on the n-1th mapping result and the nth attention feature to obtain the nth intermediate feature; perform convolution processing on the nth intermediate feature to obtain the nth convolution feature; perform addition processing on the nth convolution feature and the nth intermediate feature to obtain the nth mapping result.

[0204] In some embodiments, the mapping processing module 2534 is further configured to obtain the query matrix, key matrix and value matrix of each attention network; perform the following processing through each attention network: based on the query matrix, key matrix and value matrix of the attention network, perform single-head attention processing on the n-1th mapping result to obtain the single-head attention result corresponding to the attention network; perform fusion processing on the single-head attention results corresponding to multiple attention networks to obtain the nth attention feature; the nth feedforward network includes multiple attention networks.

[0205] An embodiment of the present application provides a computer program product comprising computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the artificial intelligence-based data processing method described above in the embodiment of the present application.

[0206] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the artificial intelligence-based data processing method provided by an embodiment of the present application, for example, the artificial intelligence-based data processing method shown in Figure 3A.

[0207] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0208] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0209] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, e.g., in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0210] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0211] To sum up, by acquiring voice and acquiring the emotion data corresponding to the voice, the content of the voice and the emotion of the corresponding voice are obtained, the first voice content feature extraction process is performed on the voice, and the first voice content feature of the corresponding voice is obtained, which is used to characterize the voice itself, and the emotion feature extraction process is performed on the emotion data to obtain the emotion feature of the corresponding voice, which is used to characterize the voice emotion, and the voice content feature and the emotion feature are fused to obtain the fusion feature of the corresponding voice, that is, the characteristics of the voice itself and the emotion characteristics can be characterized at the same time, and the fusion feature is animated parameter mapping processed to obtain the controller parameters of the corresponding voice, and the controller parameters are used to control the virtual object model to present the target image, and the target image matches the content and emotion data of the voice. Here, a mapping process from features to controller parameters is directly established, so that the controller parameters used to drive the virtual object model to present the target image can be directly obtained, which improves the driving efficiency while also improving the driving efficiency of the target image of the virtual object model from the two dimensions of voice and emotion.

[0212] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data processing method based on artificial intelligence, the method being performed by an electronic device, the method comprising: Acquire speech and obtain emotion data corresponding to the speech; Performing a first voice content feature extraction process on the voice to obtain a first voice content feature corresponding to the voice, and performing an emotion feature extraction process on the emotion data to obtain an emotion feature corresponding to the voice; fusing the first speech content feature and the emotion feature to obtain a fusion feature corresponding to the speech; Performing animation parameter mapping processing on the fusion feature to obtain controller parameters corresponding to the voice; The controller parameters are used to control the virtual object model to be presented in a target image, and the target image matches the content of the speech and the emotion data.

2. The method according to claim 1, wherein The acquiring of emotion data corresponding to the speech includes: Perform any of the following: Get the emotional data input by the subject; A second voice content feature extraction process is performed on the voice to obtain a second voice content feature corresponding to the voice, and an emotion data mapping process is performed on the second voice content feature to obtain emotion data corresponding to the voice.

3. The method according to claim 2, wherein: The performing emotion data mapping processing on the second speech content feature to obtain emotion data corresponding to the speech includes: Performing full connection processing on the second speech content feature to obtain a first linear mapping result; Maximum likelihood processing is performed on the first linear mapping result to obtain emotion data corresponding to the speech.

4. The method according to any one of claims 1 to 3, wherein The emotion data is obtained by calling a trained emotion extraction network; the method further comprises: Obtaining a first voice sample and obtaining an emotion data label corresponding to the first voice sample; Performing second speech content feature extraction processing on the first speech sample through the initialized emotion extraction network to obtain a first sample speech content feature corresponding to the first speech sample; Performing emotion data mapping processing on the first sample speech content features through the initialized emotion extraction network to obtain predicted emotion data corresponding to the first speech sample; Determining a first loss function based on the emotion data label corresponding to the first speech sample and the predicted emotion data; Based on the first loss function, the initialized emotion extraction network is updated to obtain a trained emotion extraction network.

5. The method according to any one of claims 1 to 4, wherein The controller parameters are obtained by calling a trained parameter-driven model; the method further comprises: Acquire a training sample and a real controller parameter of the training sample, wherein the training sample includes a second speech sample and an emotion data sample corresponding to the second speech sample; The model is driven by the initialized parameters to perform the following processing: performing a first speech content feature extraction process on the second speech sample to obtain a second sample speech content feature corresponding to the second speech sample, and performing an emotion feature extraction process on the emotion data sample to obtain a sample emotion feature corresponding to the second speech sample; fusing the second sample speech content feature and the sample emotion feature to obtain a sample fusion feature corresponding to the second speech sample; Performing animation parameter mapping processing on the sample fusion features to obtain prediction controller parameters corresponding to the second speech sample; Based on the predicted controller parameters and the real controller parameters, a second loss function is determined, and based on the second loss function, the initialized parameter-driven model is updated to obtain a trained parameter-driven model.

6. The method according to claim 5, wherein: The determining of a second loss function based on the predicted controller parameters and the real controller parameters includes: Extracting real control parameters corresponding to each animation frame from the real controller parameters; Extracting prediction control parameters corresponding to each of the animation frames from the prediction controller parameters; The following processing is performed for each of the animation frames: determining a difference between a real control parameter corresponding to the animation frame and a predicted control parameter corresponding to the animation frame; The difference values corresponding to the plurality of animation frames are fused to obtain the second loss function.

7. The method according to any one of claims 1 to 6, wherein The performing animation parameter mapping processing on the fusion feature to obtain controller parameters corresponding to the voice includes: Performing feedforward processing based on an attention mechanism on the fused features to obtain animation parameter features corresponding to the speech; Fully connected processing is performed on the animation parameter features to obtain controller parameters corresponding to the voice.

8. The method according to claim 7, wherein: The feedforward processing based on the attention mechanism is implemented by an attention feedforward network, and the attention feedforward network includes N cascaded feedforward networks, where N is an integer greater than 1; The performing feedforward processing on the fusion feature based on the attention mechanism to obtain animation parameter features corresponding to the speech includes: Performing a mapping process based on the attention mechanism on the input of an nth feedforward network among the N cascaded feedforward networks to obtain an nth mapping result, and transmitting the nth mapping result output by the nth feedforward network to an n+1th feedforward network to continue performing the mapping process based on the attention mechanism to obtain an n+1th mapping result corresponding to the n+1th feedforward network; Using the Nth mapping result output by the Nth feedforward network as the animation parameter feature corresponding to the speech; Wherein, n is an integer variable whose value increases from 1, n is not less than 1 and less than N. When n is 1, the input of the nth feedforward network is the fusion feature. When n is not less than 2 and less than N, the input of the nth feedforward network is the n-1th mapping result output by the n-1th feedforward network.

9. The method according to claim 8, wherein When the value of n is not less than 2 and less than N, the input of the nth feedforward network is mapped based on the attention mechanism to obtain the nth mapping result, including: Performing attention processing on the n-1th mapping result to obtain an nth attention feature; Adding the n-1th mapping result and the nth attention feature to obtain an nth intermediate feature; Performing convolution processing on the nth intermediate feature to obtain an nth convolution feature; The nth convolution feature and the nth intermediate feature are added together to obtain the nth mapping result.

10. The method according to claim 9, wherein: The nth feedforward network includes a plurality of attention networks, and performing attention processing on the n-1th mapping result to obtain the nth attention feature includes: Obtaining a query matrix, a key matrix, and a value matrix for each of the attention networks; Performing the following processing by each of the attention networks: performing single-head attention processing on the (n-1)th mapping result based on the query matrix, key matrix, and value matrix of the attention network to obtain a single-head attention result corresponding to the attention network; The single-head attention results corresponding to the multiple attention networks are fused to obtain the nth attention feature.

11. The method according to claim 10, wherein: The query matrix, key matrix, and value matrix of the attention network are used to perform single-head attention processing on the n-1th mapping result to obtain a single-head attention result corresponding to the attention network, including: determining a dot product of the query matrix and the key matrix; using a ratio of the dot product to a square root of a dimension as a similarity score matrix, wherein the dimension is the dimension of the key vector in the key matrix; Normalizing the similarity score matrix to obtain a normalized matrix; Perform matrix multiplication on the normalized matrix and the value matrix to obtain the single-head attention result corresponding to the attention network.

12. A data processing device based on artificial intelligence, comprising: an acquisition module configured to acquire speech and acquire emotion data corresponding to the speech; a feature extraction module configured to perform a first voice content feature extraction process on the speech to obtain a first voice content feature corresponding to the speech, and perform an emotion feature extraction process on the emotion data to obtain an emotion feature corresponding to the speech; a superposition processing module configured to perform a fusion process on the first speech content feature and the emotion feature to obtain a fusion feature corresponding to the speech; A mapping processing module is configured to perform animation parameter mapping processing on the fusion feature to obtain controller parameters corresponding to the voice, wherein the controller parameters are used to control the virtual object model to present a target image, and the target image matches the content of the voice and the emotional data.

13. An electronic device, comprising: a memory for storing computer-executable instructions; A processor, configured to implement the artificial intelligence-based data processing method according to any one of claims 1 to 111 when executing computer-executable instructions stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the artificial intelligence-based data processing method according to any one of claims 1 to 11.

15. A computer program product comprising computer executable instructions, wherein when the computer executable instructions are executed by a processor, the artificial intelligence-based data processing method according to any one of claims 1 to 11 is implemented.

Images