Digital face generation method and device based on artificial intelligence, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence and financial science and technology, and relates to a digital face generation method and device based on artificial intelligence, computer equipment and a storage medium, and the method comprises the steps: obtaining a target emotion label, target voice information and a target face image inputted by a user; encoding the target emotion label based on an emotion encoder to obtain a target emotion implicit vector; performing mapping processing on the target emotion implicit vector based on a target stream model to obtain a target audio emotion implicit vector; coding the target voice information based on the voice emotion recognition model to obtain a target audio implicit vector; encoding the target face image based on a visual encoder to obtain a target visual implicit vector; generating a comprehensive vector based on the target audio emotion implicit vector, the target audio implicit vector and the target visual implicit vector; and processing the comprehensive vector based on a digital face generation model to generate a target digital face. According to the invention, the generation efficiency and fidelity of the target digital face are effectively improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence development technology and financial technology, and in particular to artificial intelligence-based digital face generation methods, devices, computer equipment and storage media. Background Art

[0002] With the rapid development of artificial intelligence technology, digital human generation technology, as an important branch of artificial intelligence image processing, is gradually showing its huge application potential and commercial value. The rise of this technology stems from the high demand for simulating human appearance, behavior and even emotions. It aims to generate highly realistic and credible digital human images through computer algorithms and deep learning technology. Digital human generation technology has not only brought revolutionary changes to entertainment, education, medical care and other fields, but also provided strong support for the digital transformation of service industries such as the financial industry.

[0003] At present, the external realization of digital human generation technology mainly relies on artificial 3D modeling technology. Artificial 3D modeling technology uses professional 3D modeling software, and experienced designers build three-dimensional models of digital humans according to needs. Although this method can create delicate facial features, complex clothing textures, and realistic light and shadow effects, it hides high labor costs and a long modeling cycle. In addition, the customization of expressions also requires additional expenses and time investment, making the processing efficiency low. And due to the limitations of manual operation, the fidelity of the generated results is also relatively low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a method, device, computer equipment and storage medium for generating digital faces based on artificial intelligence, so as to solve the technical problems that the existing method of generating digital faces based on artificial 3D modeling technology has low processing efficiency and low realism of the generated results.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for generating digital faces based on artificial intelligence, which adopts the following technical solutions:

[0006] Obtain the target emotion label, target voice information and target face image input by the user;

[0007] Encode the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector;

[0008] Mapping the target emotion latent vector based on a preset target stream model to obtain a corresponding target audio emotion latent vector;

[0009] Encoding the target speech information based on a pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector;

[0010] Encoding the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector;

[0011] Generate a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector, and the target visual latent vector;

[0012] The comprehensive vector is processed based on a preset digital face generation model to generate a corresponding target digital face.

[0013] Furthermore, the step of mapping the target emotion latent vector based on the preset target stream model to obtain the corresponding target audio emotion latent vector also includes:

[0014] Acquire a pre-constructed speech data set; wherein the speech data set includes first speech information and a true emotion label corresponding to the first speech information;

[0015] Loading the speech emotion recognition model, and processing the first speech information in the speech data set through the speech emotion recognition model to obtain a corresponding audio latent vector and an emotion prediction label;

[0016] Performing feature dimensionality reduction processing on the audio latent vector to obtain a corresponding audio emotion latent vector;

[0017] Load the emotion encoder and call the pre-created initial flow model;

[0018] Encoding the true emotion label based on the emotion encoder to obtain a corresponding emotion latent vector;

[0019] Using the emotion latent vector and the corresponding audio emotion latent vector as an input and output pair of the initial flow model;

[0020] Based on the maximum likelihood estimation loss function, the initial flow model is trained using the emotion latent vector and the corresponding audio emotion latent vector to obtain a specified flow model that meets the preset training expectations;

[0021] The designated flow model is used as the target flow model.

[0022] Furthermore, the step of performing feature dimensionality reduction processing on the audio latent vector to obtain the corresponding audio emotion latent vector specifically includes:

[0023] Call the preset feature dimension reduction network;

[0024] Performing feature dimensionality reduction processing on the audio latent vector based on the feature dimensionality reduction network to obtain a corresponding first latent vector;

[0025] Regularizing the first latent vector to obtain a corresponding second latent vector;

[0026] The second latent vector is used as the audio emotion latent vector.

[0027] Furthermore, before the step of processing the comprehensive vector based on a preset digital face generation model to generate a corresponding target digital face, the step further includes:

[0028] Acquire a pre-constructed voice-face generation data set; wherein the voice-face generation data set includes second voice information and a face image;

[0029] Encode the second speech information based on the speech emotion recognition model to obtain a corresponding specified audio latent vector;

[0030] Processing the second speech information based on the emotion encoder and the target stream model to obtain a corresponding specified audio emotion latent vector;

[0031] Processing the face image based on the visual encoder to obtain a corresponding specified visual latent vector;

[0032] Generate a corresponding specified comprehensive vector based on the specified audio emotion latent vector, the specified audio latent vector and the specified visual latent vector;

[0033] Inputting the specified comprehensive vector into a generator of a preset adversarial generative network to obtain a corresponding predicted digital face image;

[0034] Calculating the L1 loss between the predicted digital face image and the real image;

[0035] Using a preset expert discriminator to evaluate the audio lip sync of the predicted digital face image to obtain a corresponding adversarial loss;

[0036] Optimizing the generative adversarial network based on the L1 loss and the adversarial loss to obtain an optimized generative adversarial network;

[0037] The optimized generative adversarial network is used as the digital face generation model.

[0038] Furthermore, the step of encoding the target speech information based on the pre-trained speech emotion recognition model to obtain the corresponding target audio latent vector specifically includes:

[0039] Calling the speech emotion recognition model, and inputting the target speech information into the speech emotion recognition model;

[0040] Performing audio extraction on the target speech information based on the audio processing module in the speech emotion recognition model to obtain corresponding target audio features;

[0041] Processing the target audio features based on the emotion processing module in the speech emotion recognition model to obtain a corresponding emotion output result;

[0042] The emotion output result is used as the target audio emotion latent vector.

[0043] Furthermore, the step of generating a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector and the target visual latent vector specifically includes:

[0044] Get the preset vector splicing strategy;

[0045] Based on the vector splicing strategy, the target audio emotion latent vector, the target audio latent vector and the target visual latent vector are spliced ​​to obtain a corresponding splicing vector;

[0046] The concatenated vector is used as the integrated vector.

[0047] Furthermore, the step of processing the comprehensive vector based on a preset digital face generation model to generate a corresponding target digital face specifically includes:

[0048] Calling the digital face generation model;

[0049] Inputting the comprehensive vector into a generator in the digital face generation model;

[0050] Processing the comprehensive vector based on the generator to obtain a corresponding output result;

[0051] The output result is used as the target digital face.

[0052] In order to solve the above technical problems, the embodiment of the present application also provides a digital face generation device based on artificial intelligence, which adopts the following technical solution:

[0053] The first acquisition module is used to acquire the target emotion label, target voice information and target face image input by the user;

[0054] A first encoding module, used for encoding the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector;

[0055] A mapping module, used for mapping the target emotion latent vector based on a preset target stream model to obtain a corresponding target audio emotion latent vector;

[0056] A second encoding module is used to encode the target speech information based on a pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector;

[0057] A third encoding module, used to encode the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector;

[0058] A first generating module, configured to generate a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector and the target visual latent vector;

[0059] The second generation module is used to process the comprehensive vector based on a preset digital face generation model to generate a corresponding target digital face.

[0060] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0061] Obtain the target emotion label, target voice information and target face image input by the user;

[0062] Encode the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector;

[0063] Mapping the target emotion latent vector based on a preset target stream model to obtain a corresponding target audio emotion latent vector;

[0064] Encoding the target speech information based on a pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector;

[0065] Encoding the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector;

[0066] Generate a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector, and the target visual latent vector;

[0067] The comprehensive vector is processed based on a preset digital face generation model to generate a corresponding target digital face.

[0068] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0069] Obtain the target emotion label, target voice information and target face image input by the user;

[0070] Encode the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector;

[0071] Mapping the target emotion latent vector based on a preset target stream model to obtain a corresponding target audio emotion latent vector;

[0072] Encoding the target speech information based on a pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector;

[0073] Encoding the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector;

[0074] Generate a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector, and the target visual latent vector;

[0075] The comprehensive vector is processed based on a preset digital face generation model to generate a corresponding target digital face.

[0076] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0077] The present application first obtains the target emotion label, target voice information and target face image input by the user; then encodes the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector; and maps the target emotion latent vector based on a preset target flow model to obtain a corresponding target audio emotion latent vector; then encodes the target voice information based on a pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector; subsequently, encodes the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector; further generates a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector and the target visual latent vector; finally, processes the comprehensive vector based on a preset digital face generation model to generate a corresponding target digital face. The present application encodes and combines the target emotion label, target speech information and target face image input by the user through the use of an emotion encoder, a target flow model, a speech emotion recognition model and a visual encoder to generate a comprehensive vector containing emotion, speech and face information, and then uses a digital face generation model to process the comprehensive vector to quickly and accurately generate a realistic and consistent target digital face, thereby improving the generation efficiency of the target digital face, as well as improving the accuracy and realism of the generated target digital face. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0079] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0080] Figure 2 A flowchart of an embodiment of a method for generating digital faces based on artificial intelligence according to the present application;

[0081] Figure 3 is a structural schematic diagram of an embodiment of an artificial intelligence-based digital face generation device according to the present application;

[0082] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0083] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0084] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0085] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0086] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0087] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0088] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0089] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0090] It should be noted that the artificial intelligence-based digital face generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the artificial intelligence-based digital face generation device is generally set in the server / terminal device.

[0091] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0092] Continue to refer Figure 2, showing a flow chart of an embodiment of a method for generating digital faces based on artificial intelligence according to the present application. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted. The method for generating digital faces based on artificial intelligence provided in the embodiment of the present application can be applied to any scenario where digital faces need to be generated, and the method for generating digital faces based on artificial intelligence can be applied to products in these scenarios, for example, digital faces in the financial insurance field. The method for generating digital faces based on artificial intelligence comprises the following steps:

[0093] Step S201, obtaining the target emotion label, target voice information and target face image input by the user.

[0094] In this embodiment, the method for generating a digital face based on artificial intelligence is executed on an electronic device (eg Figure 1 The server / terminal device shown in the figure) can obtain the target emotion tag, target voice information and target face image input by the user through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection methods currently known or to be developed in the future. The above-mentioned target emotion tag, target voice information and target face image can be selected and input according to the user's personal needs. Among them, the above-mentioned target emotion tag is a specific emotion tag input (such as happy, sad, etc.). The above-mentioned target voice information, which can also be called a voice signal, will be used to generate facial expressions synchronized with the voice. The above-mentioned target face image is an initial or reference face image, which will serve as the basis for generating expressions. Among them, facial expressions can be edited by using emotion tags. By introducing a method of using emotion tags to generate faces with corresponding emotions, the subsequently generated digital faces are more in line with the expected emotional characteristics, thereby improving the accuracy and expressiveness of the generated digital faces.

[0095] Step S202: Encode the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector.

[0096] In this embodiment, the above-mentioned emotion encoder is a pre-built encoder with the function of converting any input emotion label into a corresponding emotion latent vector. The target emotion label is input into the emotion encoder, and the target emotion label is encoded by the emotion encoder to output the corresponding target emotion latent vector, which represents a specific emotion feature.

[0097] Step S203: mapping the target emotion latent vector based on a preset target stream model to obtain a corresponding target audio emotion latent vector.

[0098] In this embodiment, the target emotion latent vector can be input into the target stream model to map the target emotion latent vector through the target stream model, so as to map the target emotion latent vector to the corresponding target audio emotion latent vector. The training process of the target stream model will be further described in detail in the subsequent specific embodiments, and will not be elaborated on here.

[0099] Step S204: encode the target speech information based on the pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector.

[0100] In this embodiment, the target audio latent vector includes the acoustic features and potential emotional information in the target speech information. The specific implementation process of encoding the target speech information based on the pre-trained speech emotion recognition model to obtain the corresponding target audio latent vector will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0101] Step S205, encoding the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector.

[0102] In this embodiment, the above-mentioned visual encoder is a pre-built encoder with the function of converting an input face image into a corresponding visual latent vector. By inputting the target emotion label into the visual encoder, the target face image is encoded by the visual encoder to output the corresponding target visual latent vector, which represents the visual features of the face.

[0103] Step S206, generating a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector and the target visual latent vector.

[0104] In this embodiment, the specific implementation process of generating the comprehensive vector based on the target audio emotion latent vector, the target audio latent vector and the target visual latent vector will be described in further detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0105] Step S207, processing the comprehensive vector based on a preset digital face generation model to generate a corresponding target digital face.

[0106] In this embodiment, the specific implementation process of processing the comprehensive vector based on the preset digital face generation model to generate the corresponding target digital face will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0107] The present application encodes and combines the target emotion label, target speech information and target face image input by the user through the use of an emotion encoder, a target flow model, a speech emotion recognition model and a visual encoder to generate a comprehensive vector containing emotion, speech and face information, and then uses a digital face generation model to process the comprehensive vector to quickly and accurately generate a realistic and consistent target digital face, thereby improving the generation efficiency of the target digital face, as well as improving the accuracy and realism of the generated target digital face.

[0108] In some optional implementations, before step S203, the electronic device may further perform the following steps:

[0109] Get pre-built speech datasets.

[0110] In this embodiment, during the training phase, the present application adopts a phased training strategy to ensure that effective features can be learned between different tasks. Specifically, the training process is divided into two main stages. The first stage is the alignment of speech and emotional features of the stream model training. Among them, the above-mentioned speech data set contains first speech information and a real emotion label corresponding to the first speech information. In addition, there is no specific limitation on the data volume of the speech data set, which can be determined according to the actual model training requirements.

[0111] The speech emotion recognition model is loaded, and the first speech information in the speech data set is processed by the speech emotion recognition model to obtain a corresponding audio latent vector and an emotion prediction label.

[0112] In this embodiment, the above-mentioned speech emotion recognition model is a pre-trained speech emotion recognition model, which can receive speech information as input and output corresponding audio latent vectors or emotion labels. Specifically, the first speech information is encoded and decoded by using the speech emotion recognition model, and the encoded feature vector is extracted as the audio latent vector, and the emotion type corresponding to the current frame speech is predicted as the emotion prediction label. In addition, the multi-classification cross entropy loss can be calculated based on the predicted emotion label and the real emotion label. This loss will be used to subsequently optimize the parameters of the pre-trained speech emotion recognition model, that is, to measure the difference between the real emotion label and the predicted label to maintain the accuracy of the speech emotion recognition model. Among them, the multi-classification cross entropy loss formula specifically includes: Among them, N represents the number of samples, M represents the type of category, and y ic Represents 0 or 1. If the true category of the sample is equal to c, it takes 1, otherwise it takes 0. ic It represents the probability that the observed sample i belongs to category c.

[0113] Perform feature dimensionality reduction processing on the audio latent vector to obtain a corresponding audio emotion latent vector.

[0114] In this embodiment, the audio latent vector is subjected to feature dimensionality reduction to capture the emotion-related part and encode it into the feature space of the audio emotion latent vector. The specific implementation process of the above-mentioned feature dimensionality reduction processing on the audio latent vector to obtain the corresponding audio emotion latent vector will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0115] Load the sentiment encoder and call the pre-created initial flow model.

[0116] In this embodiment, the emotion encoder is a pre-built encoder that can encode emotion tags into emotion latent vectors. The emotion encoder can be a simple fully connected network, whose input is the unique hot encoding or embedding vector of the emotion tag. The initial flow model can specifically adopt FlowNet or Glow. The initial flow model can learn the mapping from one distribution to another. In this application, the initial flow model will learn the mapping from emotion latent vectors to audio emotion latent vectors.

[0117] The true emotion label is encoded based on the emotion encoder to obtain a corresponding emotion latent vector.

[0118] In this embodiment, the real emotion label is input into the emotion encoder, and the emotion encoder encodes the real emotion label to obtain a corresponding emotion latent vector.

[0119] The emotion latent vector and the corresponding audio emotion latent vector are used as the input and output pair of the initial flow model.

[0120] In this embodiment, the mapping relationship from the emotion latent vector to the audio emotion latent vector is learned by using a flow model.

[0121] Based on the maximum likelihood estimation loss function, the initial flow model is trained using the emotion latent vector and the corresponding audio emotion latent vector to obtain a specified flow model that meets preset training expectations.

[0122] In this embodiment, the flow model is trained by using the maximum likelihood estimation (MLE) loss function as an optimization algorithm so that the flow model can accurately generate the corresponding audio emotion latent vector from the emotion latent vector. During the training process, the loss can be calculated by comparing the generated audio emotion latent vector with the real audio emotion latent vector, and the loss is used to update the parameters of the flow model to obtain the corresponding specified flow model. Among them, the formula of the maximum likelihood estimation loss function includes: For a given sample X=(x1,x2,...,x n ), n is the number of samples, and by inferring the model parameters θ, the probability of a given sample being generated by the model is maximized, that is, the likelihood function likehood(θ) is maximized.

[0123] The designated flow model is used as the target flow model.

[0124] The present application uses a pre-constructed speech data set to train the initial flow model based on a speech emotion recognition model, an emotion encoder, and a maximum likelihood estimation loss function, so as to quickly and intelligently complete the construction of the target flow model, improve the construction efficiency of the target flow model, and ensure the model effect of the obtained target flow model.

[0125] In some optional implementations of this embodiment, performing feature dimensionality reduction processing on the audio latent vector to obtain the corresponding audio emotion latent vector includes the following steps:

[0126] Call the preset feature dimension reduction network.

[0127] In this embodiment, the feature dimensionality reduction network specifically adopts a neural network including a fully connected layer. The feature dimensionality reduction network is used to perform feature dimensionality reduction on the audio latent vector, and the output of the network will be an audio emotion latent vector, which has a lower dimension than the original audio latent vector but retains the information most relevant to the emotion.

[0128] Based on the feature dimensionality reduction network, feature dimensionality reduction processing is performed on the audio latent vector to obtain a corresponding first latent vector.

[0129] In this embodiment, the audio latent vector is input into the feature dimensionality reduction network, and the audio latent vector is subjected to feature dimensionality reduction processing by the feature dimensionality reduction network, so as to obtain a corresponding first latent vector.

[0130] Regularization is performed on the first latent vector to obtain a corresponding second latent vector.

[0131] In this embodiment, regularization is performed on the first latent vector output by the feature dimensionality reduction network, so as to ensure that the audio emotion latent vector better represents the emotion information.

[0132] The second latent vector is used as the audio emotion latent vector.

[0133] The present application calls a preset feature dimensionality reduction network; then performs feature dimensionality reduction processing on the audio latent vector based on the feature dimensionality reduction network to obtain a corresponding first latent vector; then performs regularization processing on the first latent vector to obtain a corresponding second latent vector; and subsequently uses the second latent vector as the audio emotion latent vector. The present application can quickly and intelligently complete the dimensionality reduction processing of the audio latent vector and obtain the corresponding latent vector by using a feature dimensionality reduction network. In addition, the obtained latent vector will be further regularized to obtain the final audio emotion latent vector, thereby ensuring that the audio emotion latent vector can better represent the emotional information and improve the accuracy of the generated audio emotion latent vector.

[0134] In some optional implementations, before step S207, the electronic device may further perform the following steps:

[0135] Get the pre-built speech face generation dataset.

[0136] In this embodiment, in the second stage of training, the present application focuses on training the adversarial generative network using voice, emotion and face information to produce more realistic and coordinated images. The voice face generation data set includes the second voice information and the face image. In addition, there is no specific limitation on the amount of data in the voice face generation data set, which can be determined according to the actual model training requirements.

[0137] The second speech information is encoded based on the speech emotion recognition model to obtain a corresponding specified audio latent vector.

[0138] In this embodiment, the second speech information is encoded by using a speech emotion recognition model, and the encoded feature vector is extracted as the above-mentioned designated audio latent vector.

[0139] The second speech information is processed based on the emotion encoder and the target stream model to obtain a corresponding specified audio emotion latent vector.

[0140] In this embodiment, an emotion label corresponding to the second voice information is obtained, and the emotion label is given by the pre-trained speech emotion recognition model frozen after the previous stage of training. Then, the emotion label is encoded by using an emotion encoder to obtain a corresponding specific emotion latent vector, and then the target flow model is used to map the specific emotion latent vector to obtain the corresponding specified audio emotion latent vector.

[0141] The face image is processed based on the visual encoder to obtain a corresponding specified visual latent vector.

[0142] In this embodiment, the face image is input into a visual encoder so as to encode the face image using the visual encoder, thereby obtaining a corresponding designated visual latent vector.

[0143] A corresponding specified comprehensive vector is generated based on the specified audio emotion latent vector, the specified audio latent vector and the specified visual latent vector.

[0144] In this embodiment, the designated audio emotion latent vector, the designated audio latent vector and the designated visual latent vector may be concatenated to obtain a corresponding designated comprehensive vector.

[0145] The specified comprehensive vector is input into a preset generator of a generative adversarial network to obtain a corresponding predicted digital face image.

[0146] In this embodiment, the above-mentioned generative adversarial network (GAN) includes a discriminator and a generator, and the generator can specifically adopt a convolutional neural network, a residual network, etc.

[0147] Calculate the L1 loss between the predicted digital face image and the real image.

[0148] In this embodiment, the L1 loss between the predicted digital face image and the real image is calculated to evaluate the pixel-level difference. The L1 loss is calculated as the sum of the absolute values ​​of the differences between the generated predicted digital face image and the corresponding pixel values ​​of the real image. The formula of the L1 loss includes: Among them, L g is the image generated by the generator, L G A real image of a human face.

[0149] The audio lip synchronization of the predicted digital face image is evaluated using a preset expert discriminator to obtain a corresponding adversarial loss.

[0150] In this embodiment, an expert discriminator is pre-built, which is a model that can identify the relationship between lip shape and speech signal, and can evaluate the audio lip sync of the generated image. The adversarial loss (L_sync) of the generated predicted digital face image is calculated by using the expert discriminator, and the loss reflects the degree of lip sync between the generated image and the input speech signal. Specifically, the formula of the adversarial loss includes: and Where N: represents the total number of samples in the batch, v and s are the video feature vector and audio feature vector after ReLU activation function, respectively, and P sync is the similarity between the video feature vector and the audio feature vector.

[0151] The generative adversarial network is optimized based on the L1 loss and the adversarial loss to obtain an optimized generative adversarial network.

[0152] In this embodiment, the total loss function is formed by combining the L1 loss and the adversarial loss. Specifically, a weighted sum combination can be adopted, that is, total loss = α*L1 loss + β*L_sync, where α and β are hyperparameters used to adjust the relative importance of the two loss terms. Then, gradient descent or other optimization algorithms are used to update the parameters of the generator in the adversarial generative network to minimize the total loss function, thereby obtaining an optimized adversarial generative network. By optimizing the generator, it can better generate realistic images that match the input information, ensuring the fusion and coordination between audio, emotion and face information.

[0153] The optimized generative adversarial network is used as the digital face generation model.

[0154] In this embodiment, the above-mentioned optimized generative adversarial network is a generative adversarial network that can generate facial images synchronized with a given audio emotion latent vector, and is used as the above-mentioned digital face generation model. The present application uses a comprehensive application of flow models and generative adversarial networks. In the process of digital face generation, flow models and generative adversarial networks are simultaneously used. By comprehensively utilizing two different technical means, more diversified digital face generation is achieved while maintaining the consistency of lip shape and full facial expression. This helps to improve the fidelity of the generated results, making the generated faces more expressive and natural. In addition, the present application adopts a staged training strategy so that the training of the model no longer depends on a small amount of multimodal data sets. By using a large amount of speech emotion recognition and speech data sets in two training stages, the generalization ability of the model and its adaptability to different data types are improved. This strategy helps to train the model more comprehensively, making it more reliable in practical applications.

[0155] This application uses a pre-built speech face generation data set to train the adversarial generative network based on a speech emotion recognition model, an emotion encoder, a target flow model, a visual encoder, an L1 loss, and an expert discriminator, so as to quickly and intelligently complete the construction of a digital face generation model, thereby improving the construction efficiency of the digital face generation model and ensuring the model effect of the obtained digital face generation model.

[0156] In some optional implementations, step S204 includes the following steps:

[0157] The speech emotion recognition model is called, and the target speech information is input into the speech emotion recognition model.

[0158] In this embodiment, the speech emotion recognition model is a pre-trained speech emotion recognition model, which can receive speech information as input and output a corresponding audio latent vector or emotion label. The speech emotion recognition model is composed of an audio processing module and an emotion processing module.

[0159] Based on the audio processing module in the speech emotion recognition model, audio extraction is performed on the target speech information to obtain corresponding target audio features.

[0160] In this embodiment, the audio processing module specifically refers to the audio processing part in the speech emotion recognition model, which is usually the first few layers of the model. The audio processing module is used to extract audio features from the target speech information, such as Mel frequency cepstral coefficients (MFCCs), logarithmic Mel spectrogram, etc., and these audio features will be used as inputs for subsequent network layers.

[0161] The target audio features are processed based on the emotion processing module in the speech emotion recognition model to obtain corresponding emotion output results.

[0162] In this embodiment, the above-mentioned emotion processing module refers to the fully connected layer or recurrent neural network layer of the speech emotion recognition model. The target audio features are processed by using the emotion processing module to generate a corresponding audio latent vector, that is, the above-mentioned emotion output result. The audio latent vector contains the acoustic features and potential emotion information in the speech information.

[0163] The emotion output result is used as the target audio emotion latent vector.

[0164] This application calls the speech emotion recognition model and inputs the target speech information into the speech emotion recognition model; then, based on the audio processing module in the speech emotion recognition model, audio extraction is performed on the target speech information to obtain corresponding target audio features; then, based on the emotion processing module in the speech emotion recognition model, the target audio features are processed to obtain corresponding emotion output results; and subsequently, the emotion output results are used as the target audio emotion latent vector. This application uses the audio processing module and the emotion processing module in the speech emotion recognition model to process the target speech information, so as to quickly and accurately generate a target audio latent vector corresponding to the target speech information, thereby improving the generation efficiency of the target audio latent vector and ensuring the accuracy of the obtained target audio latent vector.

[0165] In some optional implementations of this embodiment, step S206 includes the following steps:

[0166] Get the preset vector concatenation strategy.

[0167] In this embodiment, the above-mentioned vector concatenation strategy may include concatenation, or may also include more complex fusion methods, such as weighted sum, attention mechanism and other strategies.

[0168] Based on the vector splicing strategy, the target audio emotion latent vector, the target audio latent vector and the target visual latent vector are spliced ​​to obtain a corresponding splicing vector containing speech, emotion and face information.

[0169] In this embodiment, the target audio emotion latent vector, the target audio latent vector and the target visual latent vector may be spliced ​​according to the strategy content of the above-mentioned vector splicing strategy, so as to obtain a corresponding splicing vector.

[0170] The concatenated vector is used as the integrated vector.

[0171] The present application obtains a preset vector splicing strategy; then splices the target audio emotion latent vector, the target audio latent vector and the target visual latent vector based on the vector splicing strategy to obtain a corresponding splicing vector; and subsequently uses the splicing vector as the comprehensive vector. The present application can quickly and intelligently complete the splicing of the target audio emotion latent vector, the target audio latent vector and the target visual latent vector to obtain a corresponding comprehensive vector based on the use of a vector splicing strategy, thereby improving the generation efficiency and intelligence of the comprehensive vector.

[0172] In some optional implementations of this embodiment, step S207 includes the following steps:

[0173] The digital face generation model is called.

[0174] In this embodiment, the digital face generation model is generated by training a preset adversarial generation network based on a pre-constructed voice face generation data set, and the digital face generation model is further stored after generation for subsequent calls. There is no specific limitation on the storage method of the digital face generation model, for example, blockchain storage, local database storage, cloud server storage, etc. can be used.

[0175] The comprehensive vector is input into the generator in the digital face generation model.

[0176] In this embodiment, the digital face generation model includes a discriminator and a generator.

[0177] The integrated vector is processed based on the generator to obtain a corresponding output result.

[0178] In this embodiment, after the comprehensive vector is input into the generator in the digital face generation model, the generator will generate a face image with corresponding emotions and synchronized with the voice according to the comprehensive vector, that is, the above-mentioned output result.

[0179] The output result is used as the target digital face.

[0180] This application calls the digital face generation model; then inputs the comprehensive vector into the generator in the digital face generation model; then processes the comprehensive vector based on the generator to obtain the corresponding output result; and subsequently uses the output result as the target digital face. This application uses the digital face generation model and the generator in the digital face generation model to process the comprehensive vector, so as to quickly and intelligently generate a realistic target digital face that matches the target emotion label, target voice information, and target face image corresponding to the input comprehensive vector, and can ensure the fusion and coordination between audio, emotion, and face information, effectively improving the accuracy and expressiveness of the generated target digital face.

[0181] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0182] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target digital face, the above-mentioned target digital face can also be stored in a node of a blockchain.

[0183] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0184] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0185] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0186] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0187] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0188] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a digital face generation device based on artificial intelligence, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0189] like Figure 3 As shown, the artificial intelligence-based digital face generation device 300 described in this embodiment includes: a first acquisition module 301, a first encoding module 302, a mapping module 303, a second encoding module 304, a third encoding module, a first generation module 306 and a second generation module 307. Among them:

[0190] The first acquisition module 301 is used to acquire the target emotion label, target voice information and target face image input by the user;

[0191] A first encoding module 302 is used to encode the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector;

[0192] A mapping module 303 is used to map the target emotion latent vector based on a preset target stream model to obtain a corresponding target audio emotion latent vector;

[0193] A second encoding module 304 is used to encode the target speech information based on a pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector;

[0194] A third encoding module 305 is used to encode the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector;

[0195] A first generating module 306, configured to generate a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector and the target visual latent vector;

[0196] The second generation module 307 is used to process the comprehensive vector based on a preset digital face generation model to generate a corresponding target digital face.

[0197] In this embodiment, the operations performed by the above modules or units correspond one-to-one to the steps of the artificial intelligence-based digital face generation method in the aforementioned implementation mode, and are not described in detail here.

[0198] In some optional implementations of this embodiment, the artificial intelligence-based digital face generation device further includes:

[0199] A second acquisition module is used to acquire a pre-constructed speech data set; wherein the speech data set includes first speech information and a true emotion label corresponding to the first speech information;

[0200] A first processing module is used to load the speech emotion recognition model and process the first speech information in the speech data set through the speech emotion recognition model to obtain a corresponding audio latent vector and an emotion prediction label;

[0201] A dimension reduction module, used for performing feature dimension reduction processing on the audio latent vector to obtain a corresponding audio emotion latent vector;

[0202] A first calling module, used for loading the emotion encoder and calling a pre-created initial flow model;

[0203] A fourth encoding module, used for encoding the real emotion label based on the emotion encoder to obtain a corresponding emotion latent vector;

[0204] A first determination module, configured to use the emotion latent vector and the corresponding audio emotion latent vector as an input and output pair of the initial flow model;

[0205] A training module, used to train the initial flow model using the emotion latent vector and the corresponding audio emotion latent vector based on a maximum likelihood estimation loss function to obtain a specified flow model that meets preset training expectations;

[0206] The second determining module is used to use the designated flow model as the target flow model.

[0207] In some optional implementations of this embodiment, the dimension reduction module includes:

[0208] The first calling submodule is used to call a preset feature dimension reduction network;

[0209] A dimension reduction submodule, used for performing feature dimension reduction processing on the audio latent vector based on the feature dimension reduction network to obtain a corresponding first latent vector;

[0210] A regularization submodule, used to perform regularization processing on the first latent vector to obtain a corresponding second latent vector;

[0211] The first determination submodule is used to use the second latent vector as the audio emotion latent vector.

[0212] In some optional implementations of this embodiment, the artificial intelligence-based digital face generation device further includes:

[0213] A third acquisition module is used to acquire a pre-constructed voice-face generation data set; wherein the voice-face generation data set includes the second voice information and the face image;

[0214] A fifth encoding module, used for encoding the second speech information based on the speech emotion recognition model to obtain a corresponding specified audio latent vector;

[0215] A second processing module, configured to process the second speech information based on the emotion encoder and the target stream model to obtain a corresponding specified audio emotion latent vector;

[0216] A third processing module, configured to process the face image based on the visual encoder to obtain a corresponding specified visual latent vector;

[0217] A third generating module, configured to generate a corresponding specified comprehensive vector based on the specified audio emotion latent vector, the specified audio latent vector and the specified visual latent vector;

[0218] An input module, used for inputting the specified comprehensive vector into a preset adversarial generative network generator to obtain a corresponding predicted digital face image;

[0219] A calculation module, used for calculating the L1 loss between the predicted digital face image and the real image;

[0220] An evaluation module, used for evaluating the audio lip sync of the predicted digital face image using a preset expert discriminator to obtain a corresponding adversarial loss;

[0221] An optimization module, configured to optimize the generative adversarial network based on the L1 loss and the adversarial loss to obtain an optimized generative adversarial network;

[0222] The third determination module is used to use the optimized generative adversarial network as the digital face generation model.

[0223] In some optional implementations of this embodiment, the second encoding module 304 includes:

[0224] A second calling submodule is used to call the speech emotion recognition model and input the target speech information into the speech emotion recognition model;

[0225] An extraction submodule, used for performing audio extraction on the target speech information based on the audio processing module in the speech emotion recognition model to obtain corresponding target audio features;

[0226] A first processing submodule, used to process the target audio feature based on the emotion processing module in the speech emotion recognition model to obtain a corresponding emotion output result;

[0227] The second determination submodule is used to use the emotion output result as the target audio emotion latent vector.

[0228] In some optional implementations of this embodiment, the first generating module 306 includes:

[0229] The acquisition submodule is used to obtain the preset vector splicing strategy;

[0230] A splicing submodule, used for splicing the target audio emotion latent vector, the target audio latent vector and the target visual latent vector based on the vector splicing strategy to obtain a corresponding splicing vector;

[0231] The third determining submodule is configured to use the concatenated vector as the comprehensive vector.

[0232] In some optional implementations of this embodiment, the second generating module 307 includes:

[0233] A third calling submodule is used to call the digital face generation model;

[0234] An input submodule, used for inputting the comprehensive vector into a generator in the digital face generation model;

[0235] A second processing submodule, used for processing the comprehensive vector based on the generator to obtain a corresponding output result;

[0236] The fourth determination submodule is used to use the output result as the target digital face.

[0237] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0238] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0239] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0240] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a digital face generation method based on artificial intelligence, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0241] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the artificial intelligence-based digital face generation method.

[0242] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0243] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based digital face generation method.

[0244] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0245] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for generating digital faces based on artificial intelligence, characterized in that: The steps include: Obtain the target emotion label, target voice information and target face image input by the user; Encode the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector; Mapping the target emotion latent vector based on a preset target stream model to obtain a corresponding target audio emotion latent vector; Encoding the target speech information based on a pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector; Encoding the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector; Generate a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector, and the target visual latent vector; The comprehensive vector is processed based on a preset digital face generation model to generate a corresponding target digital face.

2. The method for generating digital faces based on artificial intelligence according to claim 1, characterized in that: The step of mapping the target emotion latent vector based on the preset target stream model to obtain the corresponding target audio emotion latent vector also includes: Acquire a pre-constructed speech data set; wherein the speech data set includes first speech information and a true emotion label corresponding to the first speech information; Loading the speech emotion recognition model, and processing the first speech information in the speech data set through the speech emotion recognition model to obtain a corresponding audio latent vector and an emotion prediction label; Performing feature dimensionality reduction processing on the audio latent vector to obtain a corresponding audio emotion latent vector; Load the emotion encoder and call the pre-created initial flow model; Encoding the true emotion label based on the emotion encoder to obtain a corresponding emotion latent vector; Using the emotion latent vector and the corresponding audio emotion latent vector as an input and output pair of the initial flow model; Based on the maximum likelihood estimation loss function, the initial flow model is trained using the emotion latent vector and the corresponding audio emotion latent vector to obtain a specified flow model that meets the preset training expectations; The designated flow model is used as the target flow model.

3. The method for generating digital faces based on artificial intelligence according to claim 2, characterized in that: The step of performing feature dimensionality reduction processing on the audio latent vector to obtain the corresponding audio emotion latent vector specifically includes: Call the preset feature dimension reduction network; Performing feature dimensionality reduction processing on the audio latent vector based on the feature dimensionality reduction network to obtain a corresponding first latent vector; Regularizing the first latent vector to obtain a corresponding second latent vector; The second latent vector is used as the audio emotion latent vector.

4. The method for generating digital faces based on artificial intelligence according to claim 1, characterized in that: Before the step of processing the comprehensive vector based on the preset digital face generation model to generate the corresponding target digital face, the method further includes: Acquire a pre-constructed voice-face generation data set; wherein the voice-face generation data set includes second voice information and a face image; Encode the second speech information based on the speech emotion recognition model to obtain a corresponding specified audio latent vector; Processing the second speech information based on the emotion encoder and the target stream model to obtain a corresponding specified audio emotion latent vector; Processing the face image based on the visual encoder to obtain a corresponding specified visual latent vector; Generate a corresponding specified comprehensive vector based on the specified audio emotion latent vector, the specified audio latent vector and the specified visual latent vector; Inputting the specified comprehensive vector into a generator of a preset adversarial generative network to obtain a corresponding predicted digital face image; Calculating the L1 loss between the predicted digital face image and the real image; Using a preset expert discriminator to evaluate the audio lip sync of the predicted digital face image to obtain a corresponding adversarial loss; Optimizing the generative adversarial network based on the L1 loss and the adversarial loss to obtain an optimized generative adversarial network; The optimized generative adversarial network is used as the digital face generation model.

5. The method for generating digital faces based on artificial intelligence according to claim 1, characterized in that: The step of encoding the target speech information based on the pre-trained speech emotion recognition model to obtain the corresponding target audio latent vector specifically includes: Calling the speech emotion recognition model, and inputting the target speech information into the speech emotion recognition model; Performing audio extraction on the target speech information based on the audio processing module in the speech emotion recognition model to obtain corresponding target audio features; Processing the target audio features based on the emotion processing module in the speech emotion recognition model to obtain a corresponding emotion output result; The emotion output result is used as the target audio emotion latent vector.

6. The method for generating digital faces based on artificial intelligence according to claim 1, characterized in that: The step of generating a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector and the target visual latent vector specifically includes: Get the preset vector splicing strategy; Based on the vector splicing strategy, the target audio emotion latent vector, the target audio latent vector and the target visual latent vector are spliced ​​to obtain a corresponding splicing vector; The concatenated vector is used as the integrated vector.

7. The method for generating digital faces based on artificial intelligence according to claim 1, characterized in that: The step of processing the comprehensive vector based on a preset digital face generation model to generate a corresponding target digital face specifically includes: Calling the digital face generation model; Inputting the comprehensive vector into a generator in the digital face generation model; Processing the comprehensive vector based on the generator to obtain a corresponding output result; The output result is used as the target digital face.

8. A digital face generation device based on artificial intelligence, characterized in that: include: The first acquisition module is used to acquire the target emotion label, target voice information and target face image input by the user; A first encoding module, used for encoding the target emotion label based on a preset emotion encoder to obtain a corresponding target emotion latent vector; A mapping module, used for mapping the target emotion latent vector based on a preset target stream model to obtain a corresponding target audio emotion latent vector; A second encoding module is used to encode the target speech information based on a pre-trained speech emotion recognition model to obtain a corresponding target audio latent vector; A third encoding module, used to encode the target face image based on a preset visual encoder to obtain a corresponding target visual latent vector; A first generating module, configured to generate a comprehensive vector based on the target audio emotion latent vector, the target audio latent vector and the target visual latent vector; The second generation module is used to process the comprehensive vector based on a preset digital face generation model to generate a corresponding target digital face.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the artificial intelligence-based digital face generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the artificial intelligence-based digital face generation method according to any one of claims 1 to 7 are implemented.

Images