Digital human generation method, device, equipment and program product

Abstract

The invention discloses a digital human generation method and device, equipment and a program product, and belongs to the technical field of digital human. The method comprises the following steps: acquiring a face image of a user; based on facial features of a human face in the human face image, generating a digital human image of a digital human, the digital human image being a virtual image with the facial features; and determining the voice emotion of the digital person based on the facial expression of the face in the facial image. According to the scheme provided by the embodiment of the invention, personalized customization of the digital human image and the voice is realized, and the matching degree between the expression of the digital human image and the emotion expressed by the digital human voice is improved.

Description

Technical Field

[0001] The present application relates to the field of digital human technology, and in particular to a digital human generation method, device, equipment and program product. Background Art

[0002] Digital human is a digital human image close to human image created by digital technology. With the continuous development of digital human technology, digital human is applied to more and more daily scenes.

[0003] In the related art, when a user uses the digital human function provided by a device, the device displays a digital human image on the screen and controls the digital human to interact with the user by voice. In addition, the user can select a digital human image that meets his or her personal preferences from a plurality of preset candidate digital human images, or select a digital human voice that meets his or her personal preferences from a plurality of preset digital human voices.

[0004] The digital human image and digital human voice in the related art are all preset and cannot meet the personalized needs of different users. Summary of the invention

[0005] The present application provides a method, device, equipment and program product for generating a digital human, which can improve the personalization of the digital human. The technical solution is as follows:

[0006] On the one hand, an embodiment of the present application provides a method for generating a digital human, the method comprising:

[0007] Get the user's face image;

[0008] Based on the facial features of the human face in the facial image, generating a digital human image of the digital human, wherein the digital human image is a virtual image having the facial features;

[0009] Based on the facial expression of the human face in the facial image, the voice emotion of the digital human is determined.

[0010] On the other hand, an embodiment of the present application provides a digital human generation device, the device comprising:

[0011] An acquisition module is used to acquire a user's face image;

[0012] A generating module, configured to generate a digital human image of a digital human based on the facial features of the human face in the facial image, wherein the digital human image is a virtual image having the facial features;

[0013] The generating module is further used to determine the voice emotion of the digital human based on the facial expression of the human face in the facial image.

[0014] In some embodiments, the digital human image is generated by an image generation model, and the image generation model includes a convolutional neural network and a generative adversarial network;

[0015] The generating module is used for:

[0016] Extracting features of the face image by using the convolutional neural network to obtain the facial features of the face in the face image;

[0017] The facial features are input into the generative adversarial network, and the digital human image of the digital human is generated through the generative adversarial network.

[0018] In some embodiments, the apparatus further comprises a training module for:

[0019] Extracting features of the sample face image by using the convolutional neural network to obtain sample facial features of the sample face in the sample face image;

[0020] Inputting the sample facial features into the generative adversarial network, and generating a sample digital human image through the generative adversarial network;

[0021] Determining a model loss of the image generation model based on the sample digital human image and the sample facial features;

[0022] The convolutional neural network and the generative adversarial network are trained based on the model loss.

[0023] In some embodiments, the training module is used to:

[0024] Extracting features of the sample digital human image by using the convolutional neural network to obtain facial features of the sample digital human of the sample digital human image;

[0025] Determine facial feature loss based on the facial features of the sample digital person and the sample facial features;

[0026] Determining the image generation loss based on the discrimination result of the sample digital human image and the real digital human image by the discriminator in the generative adversarial network;

[0027] Based on the facial feature loss, the first loss weight corresponding to the facial feature loss, the image generation loss and the second loss weight corresponding to the image generation loss, the model loss of the image generation model is determined, wherein the first loss weight is negatively correlated with the training round, and the second loss weight is positively correlated with the training round.

[0028] In some embodiments, the generating module is further used to:

[0029] The image generation model corresponding to the digital human image style is determined, wherein different image generation models are used to generate digital human images of different styles.

[0030] In some embodiments, the generating module is used to:

[0031] Performing expression classification on the facial features of the human face in the face image by an expression classifier to obtain expression confidences corresponding to each candidate human face expression label; determining the voice emotion of the digital human based on the candidate human face expression label and the expression confidences;

[0032] or,

[0033] Based on the facial key points of the human face in the facial image, expression features of multiple dimensions are extracted; the expression features of multiple dimensions are input into an expression recognition network to obtain expression confidences corresponding to each candidate human face expression label output by the expression recognition network; based on the candidate human face expression label and the expression confidences, the voice emotion of the digital human is determined.

[0034] In some embodiments, the acquisition module is further used to:

[0035] Obtaining the pre-recorded voice of the user;

[0036] The generating module is further used for:

[0037] Based on the voice emotion and the pre-recorded voice, a voice package of the digital human is generated, wherein the voice package is used to characterize the voice features of the user, and the voice features have the voice emotion.

[0038] In some embodiments, the generating module is used to:

[0039] Extracting speech features from the pre-recorded speech using an LSTM model to obtain the speech features of the user;

[0040] The prosodic features in the voice features are adjusted based on the voice emotion to obtain the voice package of the digital human.

[0041] In some embodiments, the generating module is used to:

[0042] Determining a feature adjustment method based on the emotion type of the speech emotion, and determining a feature adjustment amount based on the emotion degree of the speech emotion;

[0043] Based on the feature debugging method and the feature adjustment amount, feature adjustment is performed on the prosodic feature in the voice feature to obtain the voice package of the digital human.

[0044] In some embodiments, the acquisition module is used to:

[0045] In the absence of the historical speech of the user, obtaining the pre-recorded speech corresponding to the target corpus, where the target corpus is the corpus required to generate the speech package;

[0046] In the case where there is a historical voice of the user, the missing corpus is determined based on the existing corpus corresponding to the historical voice and the target corpus, where the target corpus is the corpus required to generate the voice package; and the pre-recorded voice corresponding to the missing corpus is obtained.

[0047] On the other hand, an embodiment of the present application provides a computer device, comprising: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the digital human generation method as described in the above aspect.

[0048] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which executable instructions are stored. The executable instructions are loaded and executed by a processor to implement the digital human generation method described in the above aspects.

[0049] On the other hand, an embodiment of the present application provides a computer program product, which includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the digital human generation method described in the above aspects.

[0050] The beneficial effects of the technical solution provided by this application include at least:

[0051] In the embodiment of the present application, based on the facial features of the user's face in the face image, a digital human image with the facial features is generated, thereby improving the similarity between the image of the digital human and the user; based on the facial expression of the face in the face image, the voice emotion of the digital human is determined, thereby improving the matching degree between the expression of the digital human image and the emotion expressed by the digital human voice, thereby realizing personalized customization of the digital human image and voice. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1is a schematic diagram of a digital human generation system provided by an exemplary embodiment of the present application;

[0054] Figure 2 is a flowchart of a method for generating a digital human provided by an exemplary embodiment of the present application;

[0055] Figure 3 It is a schematic diagram of an exemplary embodiment of the present application showing a process of generating a digital human image by an image generation model;

[0056] Figure 4 is a flowchart of an image generation model training process provided by an exemplary embodiment of the present application;

[0057] Figure 5 It is a schematic diagram of an implementation of an image generation model training process provided by an exemplary embodiment of the present application;

[0058] Figure 6 is a schematic diagram of an implementation of a speech emotion determination process shown in an exemplary embodiment of the present application;

[0059] Figure 7 is a structural schematic diagram of a digital human generation device provided by an exemplary embodiment of the present application;

[0060] Figure 8 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application.

[0061] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. DETAILED DESCRIPTION

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0063] For ease of understanding, the terms involved in the embodiments of the present application are explained below.

[0064] Convolutional Neural Network (CNN): A deep learning architecture that is used to extract features through convolutional layers. It can identify deep features of the input content and is often used to perform tasks such as image and video recognition and classification. Convolutional neural networks reduce model complexity and improve computational efficiency through features such as local connectivity, weight sharing, automatic feature extraction, and hierarchical structure. In the embodiment of the present application, a convolutional neural network is used to extract features from facial images.

[0065] Generative Adversarial Network (GAN): It consists of a generator and a discriminator. The generator is responsible for generating simulated data, and the discriminator is responsible for judging whether the input data is real or generated. The generator continuously optimizes so that the discriminator cannot distinguish between real data and generated data, and the discriminator improves the accuracy of distinguishing between real data and generated data through continuous optimization. Common generative adversarial networks include StyleGAN, Pix2Pix-GAN, etc. In an embodiment of the present application, a generative adversarial network is used to generate a digital human image with the same or similar facial features as the user's face.

[0066] Long Short-Term Memory (LSTM): A special recurrent neural network (RNN) designed specifically to solve the gradient vanishing and gradient exploding problems encountered by traditional RNNs when processing long sequence data. By introducing a gating mechanism, LSTM can more effectively capture long-distance dependencies and is suitable for processing and predicting important events with very long intervals and delays in time series data. In an embodiment of the present application, the LSTM network is used to extract the user's voice features from the user's pre-recorded corpus, and to generate the user's personalized voice package with the voice features.

[0067] Figure 1 FIG. 1 is a schematic diagram of a digital human generation system provided by an exemplary embodiment of the present application. Figure 1 As shown, the system includes a terminal 110 and a server 120 .

[0068] Terminal 110 is an electronic device that has the need to use a digital human, and the electronic device may be a smart phone, a tablet computer, a smart TV, a wearable device, or a personal computer, etc. In some embodiments, the digital human can act as an intelligent assistant, and perform corresponding operations on terminal 110 according to the instructions issued by the user, or interact with the user by voice. During the interaction, terminal 110 will display the image of the digital human on the screen and play the voice of the digital human.

[0069] In other embodiments, the digital person can act as a virtual interviewer to interact with the interviewee. During the interaction between the virtual interviewer and the interviewee, the interactive content can be automatically generated by AI or input by a real interviewer.

[0070] In a possible implementation, when a personalized digital human needs to be generated, the terminal 110 uploads the user's facial image to the server 120, and the server 120 generates a customized digital human image based on the facial image and sets the digital human's voice emotion based on the facial expression. The facial image can be obtained by taking a photo of the terminal 110.

[0071] In other embodiments, in order to further enhance the degree of personalization of the digital human, the terminal 110 may upload the user's pre-recorded voice to the server 120, and the server 120 generates a customized digital human voice package based on the pre-recorded voice and voice emotion. The pre-recorded voice may be recorded by the terminal 110.

[0072] The terminal 110 may be connected to the server 120 via a wired network or a wireless network.

[0073] The server 120 is a background server for realizing the digital human generation function. The server 120 may be a single server, a server cluster consisting of multiple servers, or a cloud computing center.

[0074] In some embodiments, after the server 120 obtains the user's facial image and pre-recorded voice, it generates a digital human image with the same or similar facial features as the user based on the facial image, and generates a voice package with the same or similar voice features as the user and voice emotions that match the user's facial expressions based on the facial image and the pre-recorded voice.

[0075] Optionally, the server 120 sends the generated digital human image and voice package to the terminal 110. The terminal 110 can then display the digital human image in a corresponding usage scenario and use the voice package to perform voice interaction with the user.

[0076] In the above embodiment, the generation of the digital human is described by taking the server 120 as an example. In other possible implementations, when the terminal 110 itself has strong processing performance, the digital human image and voice package can also be generated locally by the terminal 110, and the embodiment of the present application does not limit this. For the convenience of description, the following embodiment is described by taking the digital human generation method as being executed by a computer device (which can be a terminal or a server).

[0077] Figure 2 FIG. 1 is a flow chart of a method for generating a digital human provided by an exemplary embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0078] Step 201, obtaining a user's face image.

[0079] In some embodiments, the computer device captures the user's facial image through an image acquisition device (such as a camera), or the computer device obtains a facial image uploaded by other devices.

[0080] Optionally, when the face image is collected locally by the computer device, the computer device can display the face image collection requirements, such as the need to collect the camera directly and the face needs to be within the specified frame area, so as to collect the face image that meets the requirements. Of course, after completing the collection, the computer device can perform a quality check on the collected face image to ensure the quality of the subsequent digital human generation.

[0081] When the face image is uploaded by other devices, the computer device can perform a quality check on the face image, and if the quality check passes, execute the subsequent digital human generation steps. The quality check may include a face image clarity check, a face angle check, a face lighting check, etc., which are not limited in this embodiment.

[0082] Step 202: Generate a digital human image of the digital human based on the facial features of the human face in the facial image. The digital human image is a virtual image with facial features.

[0083] In order to make the digital human have an image similar to the user, the computer device extracts the facial features of the face in the face image, thereby generating a digital human image with the facial features. The specific method of generating the digital human image will be described in detail in the following embodiment.

[0084] Optionally, the digital human image may be a simulated human image, a cartoon image, an animation image or an image having a different style from the real user image.

[0085] In some embodiments, the facial features may be concrete features such as face shape, hairstyle, skin color, eye shape, lip shape, nose shape, ear size, etc.

[0086] In other embodiments, the facial features may also be abstract features extracted by a neural network model, wherein the abstract features may be represented by feature vectors or feature graphs.

[0087] Step 203, determining the voice emotion of the digital human based on the facial expression of the face in the face image.

[0088] Generally, since the digital human image of the digital human only supports the presentation of preset expressions, and the digital human voice does not have voice emotions, there is a mismatch between the digital human image and the voice when the digital human image is displayed and the digital human voice is played. In the embodiment of the present application, in order to improve the matching degree between the digital human image and the voice, the computer device performs expression analysis on the face in the face image to obtain the facial expression, and then determines the voice emotion of the digital human based on the facial expression, wherein the voice emotion matches the facial expression. For example, when the facial expression is happy, the voice emotion is happy; when the facial expression is angry, the voice emotion is angry.

[0089] In some embodiments, when the facial expression is a single expression, the voice emotion may be a single emotion; when the facial expression is a complex expression (such as happy and excited), the voice emotion may be a complex emotion.

[0090] In some embodiments, the same facial expression may correspond to different expression levels, and correspondingly, the same voice emotion may correspond to different emotion levels. The same sentence may have different voice features under different voice emotions, and the same sentence may have different voice features under different emotion levels of the same voice emotion.

[0091] In a possible implementation, the computer device sets a correspondence between facial expressions and voice emotions, and the computer device determines the voice emotions of the digital person from the correspondence based on the recognized facial expressions.

[0092] In some embodiments, the computer device applies the voice emotion to the digital human, and when subsequently performing voice interaction with the digital human, the digital human interacts with the user through voice with the voice emotion. Optionally, the computer device can apply the voice emotion to the default voice package of the digital human.

[0093] To summarize, in the embodiments of the present application, a digital human image having the facial features is generated based on the facial features of the user's face in the face image, thereby improving the image similarity between the digital human and the user; based on the facial expressions of the face in the face image, the voice emotions of the digital human are determined, thereby improving the matching degree between the expressions of the digital human image and the emotions expressed by the digital human voice, thereby achieving personalized customization of the digital human image and voice.

[0094] In order to further improve the personalization of the digital human voice, the computer device can generate a voice package with the same or similar voice characteristics as the user according to the user's needs.

[0095] In a possible implementation, a computer device acquires a pre-recorded voice of a user, and generates a voice package of a digital human based on the voice emotion and the pre-recorded voice, wherein the voice package is used to characterize the voice features of the user, and the voice features have voice emotions.

[0096] In some embodiments, the computer device records the user's pre-recorded voice through an audio collection device (such as a microphone), or the computer device obtains the pre-recorded voice uploaded by other devices.

[0097] Optionally, when the pre-recorded voice is collected locally by the computer device, the computer device can display the voice text to be read aloud and the voice recording requirements, such as the speech speed requirement when reading the voice text aloud, so as to collect the pre-recorded voice that meets the requirements. Of course, after completing the collection, the computer device can perform a quality check on the collected pre-recorded voice to ensure the quality of subsequent voice package generation.

[0098] When the pre-recorded voice is uploaded by other devices, the computer device can perform a quality check on the pre-recorded voice, and if the quality check passes, execute the subsequent digital human voice package generation step. The quality check may include voice clarity check, voice integrity check, pronunciation accuracy check, etc., which are not limited in this embodiment.

[0099] In order to make the speech uttered by the digital human have speech emotions that match human facial expressions, when generating the digital human's speech package based on the pre-recorded speech, the computer device integrates the speech emotions into the speech package, so that the speech package can not only be used to characterize the user's speech features, but also has speech emotions that match human facial expressions.

[0100] In some embodiments, the voice package includes voice features of voice primitives extracted from pre-recorded voices, and when integrating voice emotions, the computer device adjusts the voice features of the voice primitives. The specific method of generating a voice package with voice emotions is described in detail in the following embodiments.

[0101] It should be noted that the generation of the digital human image and the voice package can be executed sequentially or synchronously (i.e., the facial image and the pre-recorded voice are obtained synchronously), and the embodiment of the present application does not limit the generation sequence of the two.

[0102] In some embodiments, the computer device may display the generated digital human image and voice package. During the display of the voice package, the computer device may play the voice corresponding to the preset reading text based on the voice package. The user may further adjust the generated digital human image, or trigger the regeneration of the digital human image and voice package.

[0103] In the embodiment of the present application, based on the facial features of the user's face in the face image, a digital human image with the facial features is generated, thereby improving the similarity between the digital human and the user's image; based on the facial expression of the face in the face image, the digital human's voice emotion is determined, and then the voice emotion and pre-recorded voice are combined to generate a voice package that can express the voice emotion, which not only improves the similarity between the digital human and the user's voice, but also improves the matching degree between the expression of the digital human image and the emotion expressed by the digital human voice, thereby realizing personalized customization of the digital human image and voice.

[0104] In a possible implementation, the digital human image is generated by an image generation model, and the image generation model is a pre-trained neural network model.

[0105] In addition, in order to improve the quality of digital human image generation, the image generation model adopts the structure of convolutional neural network + generative adversarial network. The input of the convolutional neural network is the face image, the output of the convolutional neural network is connected to the input of the generative adversarial network, and the generative adversarial network is used to output the generated digital human image based on the output of the convolutional neural network.

[0106] In some embodiments, since convolutional neural networks have advantages in image processing (extraction of image features), the computer device extracts features of the face image through a convolutional neural network to obtain the facial features of the face in the face image, and inputs the facial features into a generative adversarial network to generate a digital human image of the digital human through the generative adversarial network.

[0107] Among them, the digital human image is generated by the generator in the generative adversarial network.

[0108] Optionally, the facial feature may be a feature map output by a convolutional layer in a convolutional neural network.

[0109] Optionally, when the convolutional neural network has a classification layer, the facial feature may also be a facial feature classification result output by the classification layer, such as a face shape classification result, a skin color classification result, a mouth shape classification result, and the like.

[0110] Optionally, the convolutional neural network can be a traditional CNN, AlexNet, VGG, etc., and the generative adversarial network can be a traditional GAN, StyleGAN, Pix2Pix-GAN, etc., which is not limited in the embodiments of the present application.

[0111] Indicative, such as Figure 3 As shown, the computer device inputs the face image 301 into the convolutional neural network 302, and the convolutional neural network 302 extracts features from the face image 301 to obtain facial features 303. The generative adversarial network 304 generates an image based on the facial features 303 to obtain a digital human image 305 having the facial features 303.

[0112] The combination of "convolutional neural network + generative adversarial network" enables the convolutional neural network to give full play to its advantages in image feature extraction and more efficiently extract accurate and comprehensive facial features, while the generative adversarial network can give full play to its advantages in image generation and generate digital human images that conform to facial features, thereby improving the generation quality of digital human images.

[0113] In order to further improve the personalization of the digital human image, the computer device is pre-trained with an image generation model for generating digital human images of different styles. Before generating the digital human image, the computer device determines the image generation model corresponding to the digital human image style. Subsequently, the computer device generates a digital human image that conforms to the digital human image style through the image generation model corresponding to the digital human image style.

[0114] The digital human image style can be selected by the user from a number of candidate digital human image styles. For example, the computer device provides a digital human image style selection interface, which contains example digital human images of different candidate digital human image styles for the user to select.

[0115] Optionally, when the candidate digital human image styles provided cannot meet the user's personalized needs, the user can also choose to upload several virtual image pictures of the same desired style (such as screenshots of character images of different characters in the same animation), and the computer device will be trained based on the virtual image pictures to generate a digital human image of the desired style, thereby further improving the personalization of the digital human image.

[0116] In some embodiments, the convolutional neural network and the generative adversarial network in the image generation model are trained in steps. Illustratively, the computer device first trains the convolutional neural network, and after the convolutional neural network training is completed, the computer device further uses the generative adversarial network. When training the generative adversarial network, the computer device freezes the network parameters of the convolutional neural network and only updates the network parameters of the generative adversarial network.

[0117] Since the facial features required by the generative adversarial network may be different under different image styles, for example, image style A pays more attention to eye features and ignores nose features, while image style B pays more attention to mouth features and ignores eye features, when the convolutional neural network and the generative adversarial network are trained step by step, the convolutional neural network may not be able to learn the facial features required by the generative adversarial network under a specific image style, affecting the quality of the final generated digital human image.

[0118] In order to further improve the quality of digital human image generation, in other embodiments, the convolutional neural network and the generative adversarial network in the image generation model are jointly trained, so that the convolutional neural network can extract facial features that are more suitable for the current image style according to the downstream image generation tasks, thereby improving the matching degree between the digital human image generated by the generative adversarial network and the current image style.

[0119] In a possible implementation, Figure 4 As shown, the joint training process of convolutional neural network and generative adversarial network can include the following steps.

[0120] Step 401, extracting features from a sample face image through a convolutional neural network to obtain sample facial features of a sample face in the sample face image.

[0121] During the training process, the computer device extracts features of the sample face image through the convolutional neural network to be trained to obtain sample facial features of the sample face.

[0122] In some embodiments, the convolutional neural network has multiple convolutional layers, and the sample face image is convolutionally processed by the multiple convolutional layers to obtain the sample facial features. In addition to the convolutional layer, the convolutional neural network also includes a pooling layer for pooling the intermediate feature map output by the convolutional layer. This embodiment does not limit the specific structure of the convolutional neural network.

[0123] Indicatively, Figure 5 As shown, the computer device inputs the sample face image 501 into the convolutional neural network 502 to be trained for feature extraction to obtain the sample facial features 503.

[0124] Step 402: Input the sample facial features into a generative adversarial network, and generate a sample digital human image through the generative adversarial network.

[0125] The computer device inputs the sample facial features into the generative adversarial network to be trained, and the generative adversarial network generates a sample digital human image based on the sample facial features.

[0126] Among them, the generative adversarial network is composed of a generator and a discriminator. During the training process, the computer device obtains the sample digital human image generated by the generator based on the sample facial features.

[0127] Indicatively, Figure 5 As shown, the computer device inputs the extracted sample facial features 503 into the generator 504 to obtain the sample digital human image 505 generated by the generator 504.

[0128] Step 403: Determine the model loss of the image generation model based on the sample digital human image and the sample facial features.

[0129] Furthermore, the computer device determines the model loss of the image generation model based on the sample digital human image and the sample facial features. In order to enable the convolutional neural network to extract the facial features necessary for generating a digital human image of a specific style and to improve the accuracy of the generative adversarial network in restoring facial features when generating a digital human image, in some embodiments, the model loss of the image generation model consists of two parts: facial feature loss and image generation loss. Accordingly, the model loss determination process may include the following sub-steps:

[0130] Sub-step 1: extracting features of the sample digital human image through a convolutional neural network to obtain facial features of the sample digital human image.

[0131] In order to determine whether the generative adversarial network applies the extracted facial features to the generated digital human image, the computer device extracts features of the sample digital human image generated by the generator in the generative adversarial network through a convolutional neural network to obtain the sample digital human facial features of the sample digital human image.

[0132] In some embodiments, the facial features of the sample digital person may be represented by a feature map, or the facial features of the sample digital person may be represented by a facial feature classification result.

[0133] Indicatively, Figure 5 As shown, the computer device inputs the sample digital human image 505 generated by the generator into the convolutional neural network 502 for feature extraction to obtain the facial features 506 of the sample digital human.

[0134] Sub-step 2: Determine facial feature loss based on the facial features of the sample digital person and the sample facial features.

[0135] When the generative adversarial network fully utilizes the extracted facial features when generating a digital human image, the facial features extracted from the sample face image and the sample digital human image using the same convolutional neural network should be similar; conversely, if the extracted facial features are not fully utilized, the facial features extracted from the sample face image and the sample digital human image using the same convolutional neural network will differ greatly. Therefore, the computer device determines the facial feature loss by calculating the feature difference between the facial features of the sample digital human and the sample facial features, wherein the smaller the facial feature loss, the closer the facial features of the sample digital human are to the sample facial features, and the larger the facial feature loss, the greater the difference between the facial features of the sample digital human and the sample facial features.

[0136] In some embodiments, the facial feature loss can be a mean square error loss, a cross entropy loss, a multi-label classification loss, a cosine embedding loss, etc. between features. The embodiments of the present application do not limit the specific loss calculation method between features (the loss calculation method can be selected according to the expression of the facial features).

[0137] Indicative, such as Figure 5 As shown, the computer device determines the facial feature loss 507 based on the sample digital human facial features 506 and the sample facial features 503 .

[0138] Sub-step 3: Determine the image generation loss based on the discrimination results of the sample digital human image and the real digital human image by the discriminator in the generative adversarial network.

[0139] In addition to the generator for the user-generated digital human image, the generative adversarial network also includes a discriminator for determining whether the input image is a real image or a generated image. In order to improve the authenticity of the generated digital human image, the computer device uses the discriminator to discriminate the sample digital human image and obtains a discriminant result that characterizes whether the sample digital human image is a real digital human image or a generated digital human image.

[0140] Furthermore, in order to make the generated digital human image have a specific digital human image style, the computer device inputs the real digital human image with the specific digital human image style into the discriminator, and the discriminator discriminates the real digital human image.

[0141] Based on the discrimination results of the sample digital human image and the real digital human image, the computer device determines the image generation loss. Optionally, the image generation loss includes the generation loss of the generator and the discrimination loss of the discriminator. The generation loss is determined based on the discrimination result of the sample digital human image by the discriminator, and the discrimination loss is determined based on the discrimination result of the sample digital human image and the real digital human image by the discriminator.

[0142] Indicative, such as Figure 5As shown, the computer device determines the image generation loss 510 based on the discrimination result of the sample digital human image 505 and the real digital human image 508 by the discriminator 509 .

[0143] Sub-step 4: Determine the model loss of the image generation model based on the facial feature loss, the first loss weight corresponding to the facial feature loss, the image generation loss, and the second loss weight corresponding to the image generation loss.

[0144] In some embodiments, in order to balance the importance of facial feature extraction tasks and image generation tasks in the model training process, the computer device sets respective loss weights for facial feature loss and image generation loss, thereby determining the model loss of the image generation model based on the weighted results of facial feature loss, image generation loss and loss weights.

[0145] Schematically, the model loss can be expressed as:

[0146] L 总 =λ1*L1+λ2*L2

[0147] Among them, L 总 is the model loss, L1 is the facial feature loss, λ1 is the first loss weight corresponding to the facial feature loss, L2 is the image generation loss, and λ2 is the second loss weight corresponding to the image generation loss.

[0148] Indicatively, Figure 5 As shown, the computer device performs weighted calculation on the facial feature loss 507 and the first loss weight 511 to obtain the facial feature weighted loss, and performs weighted calculation on the image generation loss 510 and the second loss weight 512 to obtain the image generation weighted loss, thereby determining the sum of the two weighted losses as the model loss 513.

[0149] In a possible implementation, the first loss weight corresponding to the facial feature loss and the second loss weight corresponding to the image generation loss may be fixed hyperparameters.

[0150] In another possible implementation, the first loss weight and the second loss weight may be hyperparameters that are dynamically adjusted at different training stages, thereby focusing on learning different tasks at different training stages.

[0151] Optionally, the first loss weight is negatively correlated with the training round, and the second loss weight is positively correlated with the training round. For example, in the first 10 training rounds, the first loss weight is 0.7 and the second loss weight is 0.3, and starting from the 11th training round, the first loss weight is 0.3 and the second loss weight is 0.7.

[0152] By adopting this weight setting method, the training intensity of the facial feature extraction task can be increased in the early stage of model training (the quality of facial feature extraction will directly affect the final image generation quality, so it is focused on training in the early stage); and in the later stage of model training, increasing the training intensity of the image generation task will help improve the model training quality and model convergence speed.

[0153] Step 404: training the convolutional neural network and the generative adversarial network based on the model loss.

[0154] In some embodiments, the computer device uses a back-propagation algorithm to train the convolutional neural network and the generative adversarial network in the model based on the model loss.

[0155] Optionally, when the number of training rounds is reached or the model loss converges, the computer device stops model training and obtains a trained image generation model. The embodiment of the present application does not limit the conditions for completing model training.

[0156] In this embodiment, in the process of jointly training the convolutional neural network and the generative adversarial network, the model loss is determined based on the facial feature loss between the sample digital human facial features and the sample facial features, and the image generation loss corresponding to the discrimination result of the discriminator, so that in the subsequent training process based on the model loss, it is possible to improve the accuracy of the convolutional neural network in extracting facial features of a specific image style, and improve the image quality of the digital human image generated by the generative adversarial network based on facial features.

[0157] In addition, during the training process, the loss weights of facial feature loss and image generation loss are dynamically adjusted at different training stages, so that the model can focus on learning different tasks at different training stages, which helps to improve the model convergence speed and training quality.

[0158] Regarding the method of generating the voice package of the digital human, since the LSTM network has obvious advantages in capturing the temporal dependencies in voice, in one possible implementation, the computer device extracts voice features of the pre-recorded voice through the LSTM model to obtain the user's voice features.

[0159] In order to make the generated voice package have voice emotion, the computer device further processes the voice features based on the voice emotion. Generally, when the user expresses different emotions, the prosodic features of the voice emitted are different. Therefore, the computer device adjusts the prosodic features in the voice features based on the voice emotion to obtain the voice package of the digital person.

[0160] Optionally, the prosodic feature includes at least one of the following aspects: intonation, speaking speed, sentence duration, fundamental frequency (maximum value, minimum value), and stress position duration.

[0161] In some embodiments, different voice emotions correspond to different prosodic feature adjustment methods. For example, when the voice emotion is happy, the intonation is adjusted to a high-pitched narrow range; for another example, when the voice emotion is angry, the intonation is adjusted to a high-pitched wide range; for another example, when the voice emotion is happy, the fundamental frequency change is adjusted to be smooth, and when the voice emotion is angry, the fundamental frequency change is adjusted to fluctuate; for another example, when the voice emotion is happy, the speech speed is adjusted to the first speed, and when the voice emotion is angry, the speech speed is adjusted to the second speed (less than the first speed).

[0162] In some embodiments, the same type of speech emotions but different degrees correspond to different feature adjustment amounts. For example, when the speech emotion is generally happy, the speech speed is adjusted to the first speed, and when the speech emotion is very happy, the speech speed is adjusted to the third speed (greater than the first speed).

[0163] In one possible implementation, a computer device determines a feature adjustment method based on the emotion type of the speech emotion, and determines a feature adjustment amount based on the emotion degree of the speech emotion, thereby performing feature adjustment on the prosodic features in the speech features based on the feature adjustment method and the feature adjustment amount to obtain a voice package of a digital human.

[0164] Optionally, the emotion type and emotion degree may be determined in the process of determining speech emotion based on facial expressions.

[0165] Optionally, the computer device presets a correspondence between the speech emotion type, speech emotion degree, feature adjustment method and feature adjustment amount. The computer device subsequently determines the feature adjustment method and feature adjustment amount suitable for the current speech emotion from the correspondence.

[0166] In this embodiment, the computer device utilizes the advantage of the LSTM network in capturing the temporal dependencies in speech, and uses LSTM to extract speech features of pre-recorded speech, thereby improving the accuracy of speech feature extraction; and, based on the emotion type and emotion degree, the extracted speech features are finely adjusted so that the adjusted speech features can express accurate and rich emotions.

[0167] Regarding the method of determining the voice emotion of the digital human, in some embodiments, since the facial features of the human face are extracted through a convolutional neural network during the process of generating the digital human image, the computer device can reuse the facial features to determine the facial expressions and then determine the voice emotion of the digital human.

[0168] In a possible implementation, an expression classifier is further provided after the convolutional neural network for extracting facial features. The computer device uses the expression classifier to classify the facial features of the face in the face image, and obtains the expression confidence corresponding to each candidate face expression label, thereby determining the voice emotion of the digital human based on the candidate face expression label and the expression confidence.

[0169] Optionally, the expression classifier adopts structures such as deep neural networks (DNN) and multi-layer perceptrons (MLP), which is not limited in the embodiments of the present application.

[0170] In some embodiments, the expression classifier has multiple classification heads, each of which corresponds to a respective candidate facial expression label. When the expression classifier performs expression classification based on the input facial features, it outputs the expression confidence of each candidate facial expression label. The higher the expression confidence, the higher the probability that the face has the expression represented by the candidate facial expression label.

[0171] Optionally, different candidate facial expression labels correspond to different types of facial expressions (such as happy, angry, sad, shy, excited, etc.), or different candidate facial expression labels correspond to different degrees of facial expressions of the same type (such as very happy, generally happy, very angry, generally angry, etc.).

[0172] When determining the voice emotion of the digital human based on the expression confidence of the candidate facial expression label, in one possible implementation, the computer device determines the candidate facial expression represented by the candidate facial expression label corresponding to the highest expression confidence, and determines the candidate voice emotion corresponding to the candidate facial expression as the voice emotion of the digital human.

[0173] Since human facial expressions are complex, in another possible implementation, the computer device determines k candidate facial expressions represented by the first k candidate facial expression labels whose expression confidences are higher than the confidence threshold, and determines the composite voice emotions corresponding to the k candidate facial expressions as the voice emotions of the digital human. Optionally, the computer device sets a correspondence between the facial expression combination and the composite voice emotion, and the computer device determines the composite voice emotions corresponding to the k candidate facial expressions based on the correspondence.

[0174] exist Figure 3 On the basis of Figure 6 As shown, the computer device inputs the facial features 303 extracted by the convolutional neural network 302 into the expression classifier 307, and the expression classifier 307 performs expression classification to obtain the facial expression 308 of the face in the face image 301, and then determines the voice emotion 309 of the digital person based on the facial expression 308.

[0175] It should be noted that in order to improve the accuracy of facial expression classification, sample face images with true facial expression labels need to be used to train the expression classifier during the training process.

[0176] Of course, in addition to reusing the facial features extracted by the convolutional neural network for facial expression recognition, in another possible implementation, the computer device extracts expression features of multiple dimensions based on the facial key points of the face in the face image, and inputs the expression features of multiple dimensions into the expression recognition network to obtain the expression confidence corresponding to each candidate facial expression label output by the expression recognition network, thereby determining the voice emotion of the digital person based on the candidate facial expression label and the expression confidence.

[0177] Optionally, the computer device determines the facial key points of the face in the face image through a facial key point recognition algorithm, and determines the expression features of multiple dimensions based on the position information of the recognized facial key points. The expression features of different dimensions may be morphological features and / or color features of different facial regions.

[0178] In some embodiments, expression features of different dimensions may include at least one of the following: eye opening degree feature (which can be determined based on the distance between the eye key points), eyebrow curvature degree feature (which can be determined based on the connecting line between the eyebrow key points), mouth corner angle feature (which can be determined based on the connecting line between the mouth key points), mouth opening and closing feature (which can be determined based on the distance between the mouth key points), mouth shape feature (which can be determined based on the mouth outline enclosed by the mouth key points), forehead shape feature, and cheek color feature.

[0179] After obtaining the facial expression features of multiple dimensions, the computer device processes the facial expression features so that the processed facial expression features meet the network input requirements of the facial expression recognition network, thereby extracting and classifying the input facial expression features through the facial expression recognition network.

[0180] In some embodiments, the expression recognition network may include a feature extraction layer and a classification layer, wherein the feature extraction layer is used to further extract the input expression features to obtain deep-level features, and the classification layer has multiple classification heads for performing multiple classification tasks based on the deep-level features.

[0181] In this embodiment, the computer device reuses the facial feature extraction results of the convolutional neural network on the face image, determines the facial expression of the face in the face image through the expression classifier, and then determines the voice emotion of the digital person based on the facial expression, which helps to reduce the amount of calculation while ensuring the accuracy of voice emotion recognition.

[0182] In order to ensure the quality of the generated voice package, the user needs to record a voice containing sufficient corpus. In this process, the computer device can display the corpus that the user needs to read aloud, and the user reads aloud and records the voice according to the corpus. In order to reduce the time spent by the user in recording the voice, the computer device can use the user's historical voice as a supplementary corpus, thereby reducing the corpus that the user needs to read aloud when recording the voice.

[0183] In one possible implementation, when there is a historical voice of the user, the computer device determines the missing corpus based on the existing corpus corresponding to the historical voice and the target corpus, thereby obtaining the pre-recorded speech corresponding to the missing corpus; and when there is no historical voice of the user, the computer device obtains the pre-recorded speech corresponding to the target corpus.

[0184] The target corpus is the corpus required to generate the voice package.

[0185] Optionally, the historical voice may be the voice that the user has historically spoken during voice interaction. Furthermore, in order to improve the quality of the generated voice package, the historical voice used for subsequent co-generation with the pre-recorded voice needs to meet quality requirements. The quality requirements may include voice clarity requirements, voice length requirements, voice loudness requirements, etc.

[0186] Based on the historical corpus that meets the quality requirements, the computer device can determine the existing corpus of the current user. By comparing the differences between the existing corpus and the target corpus, the computer device generates the missing corpus.

[0187] In a possible implementation, the target corpus is a corpus containing a full range of syllable types (such as vowels, consonants, semivowels, nasals, stops, affricates, fricatives, laterals, and glides). The computer determines the existing syllable types contained in the existing corpus, thereby determining the missing syllable types based on the full range of syllable types and the existing syllable types, and then generating the missing corpus containing the missing syllable types.

[0188] Optionally, the missing corpus can be extracted from the target corpus, or can be regenerated based on the missing syllable type.

[0189] After determining the missing corpus, the computer device can display the missing corpus, and the user can read and record the speech according to the missing corpus. Since the corpus volume of the missing corpus is less than the corpus volume of the target corpus, the user needs to read less corpus when recording the speech, which helps to improve the speech recording speed.

[0190] It should be noted that the above-mentioned facial images, pre-recorded voices and historical user voices are all obtained with the user's permission.

[0191] Figure 7: is a schematic diagram of the structure of a digital human generation device provided by an exemplary embodiment of the present application. The device includes:

[0192] The acquisition module 701 is used to acquire the user's face image;

[0193] A generating module 702, configured to generate a digital human image of a digital human based on the facial features of the human face in the facial image, wherein the digital human image is a virtual image having the facial features;

[0194] The generating module 702 is further used to determine the voice emotion of the digital human based on the facial expression of the face in the facial image.

[0195] Optionally, the digital human image is generated by an image generation model, and the image generation model includes a convolutional neural network and a generative adversarial network;

[0196] The generating module 702 is used for:

[0197] Extracting features of the face image by using the convolutional neural network to obtain the facial features of the face in the face image;

[0198] The facial features are input into the generative adversarial network, and the digital human image of the digital human is generated through the generative adversarial network.

[0199] Optionally, the device further comprises a training module, which is used to:

[0200] Extracting features of the sample face image by using the convolutional neural network to obtain sample facial features of the sample face in the sample face image;

[0201] Inputting the sample facial features into the generative adversarial network, and generating a sample digital human image through the generative adversarial network;

[0202] Determining a model loss of the image generation model based on the sample digital human image and the sample facial features;

[0203] The convolutional neural network and the generative adversarial network are trained based on the model loss.

[0204] Optional, training module for:

[0205] Extracting features of the sample digital human image by using the convolutional neural network to obtain facial features of the sample digital human of the sample digital human image;

[0206] Determine facial feature loss based on the facial features of the sample digital person and the sample facial features;

[0207] Determining the image generation loss based on the discrimination result of the sample digital human image and the real digital human image by the discriminator in the generative adversarial network;

[0208] Based on the facial feature loss, the first loss weight corresponding to the facial feature loss, the image generation loss and the second loss weight corresponding to the image generation loss, the model loss of the image generation model is determined, wherein the first loss weight is negatively correlated with the training round, and the second loss weight is positively correlated with the training round.

[0209] Optionally, the generating module 702 is further used to:

[0210] The image generation model corresponding to the digital human image style is determined, wherein different image generation models are used to generate digital human images of different styles.

[0211] Optionally, the generating module 702 is used to:

[0212] Performing expression classification on the facial features of the human face in the face image by an expression classifier to obtain expression confidences corresponding to each candidate human face expression label; determining the voice emotion of the digital human based on the candidate human face expression label and the expression confidences;

[0213] or,

[0214] Based on the facial key points of the human face in the facial image, expression features of multiple dimensions are extracted; the expression features of multiple dimensions are input into an expression recognition network to obtain expression confidences corresponding to each candidate human face expression label output by the expression recognition network; based on the candidate human face expression label and the expression confidences, the voice emotion of the digital human is determined.

[0215] Optionally, the acquisition module 701 is further used to:

[0216] Obtaining the pre-recorded voice of the user;

[0217] The generating module 702 is further used for:

[0218] Based on the voice emotion and the pre-recorded voice, a voice package of the digital human is generated, wherein the voice package is used to characterize the voice features of the user, and the voice features have the voice emotion.

[0219] Optionally, the generating module 702 is used to:

[0220] Extracting speech features from the pre-recorded speech using an LSTM model to obtain the speech features of the user;

[0221] The prosodic features in the voice features are adjusted based on the voice emotion to obtain the voice package of the digital human.

[0222] Optionally, the generating module 702 is used to:

[0223] Determining a feature adjustment method based on the emotion type of the speech emotion, and determining a feature adjustment amount based on the emotion degree of the speech emotion;

[0224] Based on the feature adjustment method and the feature adjustment amount, feature adjustment is performed on the prosodic feature in the voice feature to obtain the voice package of the digital human.

[0225] Optionally, the acquisition module 701 is used to:

[0226] In the absence of the historical speech of the user, obtaining the pre-recorded speech corresponding to the target corpus, where the target corpus is the corpus required to generate the speech package;

[0227] In the case where there is a historical voice of the user, the missing corpus is determined based on the existing corpus corresponding to the historical voice and the target corpus, where the target corpus is the corpus required to generate the voice package; and the pre-recorded voice corresponding to the missing corpus is obtained.

[0228] To summarize, in the embodiments of the present application, a digital human image having the facial features is generated based on the facial features of the user's face in the face image, thereby improving the image similarity between the digital human and the user; based on the facial expressions of the face in the face image, the voice emotions of the digital human are determined, thereby improving the matching degree between the expressions of the digital human image and the emotions expressed by the digital human voice, thereby achieving personalized customization of the digital human image and voice.

[0229] It should be noted that the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the digital human generation device provided in the above embodiment and the digital human generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0230] An embodiment of the present application also provides a computer device, comprising: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the digital human generation method described in the above embodiment.

[0231] For example, Figure 81 is a schematic diagram of a computer device provided by an exemplary embodiment of the present application. The computer device can be implemented as a terminal or a server.

[0232] The computer device 800 includes a central processing unit (CPU) 801, a system memory 804 including a random access memory (RAM) 802 and a read-only memory (ROM) 803, and a system bus 805 connecting the system memory 804 and the central processing unit 801. The computer device 800 also includes a basic input / output system (I / O system) 806 for facilitating information transmission between various components in the computer device, and a large-capacity storage device 807 for storing an operating system 813, application programs 814, and other program modules 815.

[0233] The basic input / output system 806 includes a display 808 for displaying information and an input device 809 such as a mouse and a keyboard for user inputting information. The display 808 and the input device 809 are connected to the central processing unit 801 through an input / output controller 810 connected to the system bus 805. The basic input / output system 806 may also include an input / output controller 810 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 810 also provides output to a display screen, a printer, or other types of output devices.

[0234] The mass storage device 807 is connected to the central processing unit 801 via a mass storage controller (not shown) connected to the system bus 805. The mass storage device 807 and its associated computer-readable storage medium provide non-volatile storage for the computer device 800. That is, the mass storage device 807 may include a computer-readable storage medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0235] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable storage instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer storage medium is not limited to the above. The above-mentioned system memory 804 and mass storage device 807 can be collectively referred to as memory.

[0236] The memory stores one or more programs, and the one or more programs are configured to be executed by one or more central processing units 801. The one or more programs contain instructions for implementing the above-mentioned method embodiments. The central processing unit 801 executes the one or more programs to implement the methods provided by the above-mentioned method embodiments.

[0237] According to various embodiments of the present application, the computer device 800 can also be connected to a remote computer device on a network through a network such as the Internet. That is, the computer device 800 can be connected to a network 812 through a network interface unit 811 connected to the system bus 805, or the network interface unit 811 can be used to connect to other types of networks or remote computer device systems (not shown).

[0238] The memory also includes one or more programs, which are stored in the memory and include steps executed by a computer device in the method provided in the embodiment of the present application.

[0239] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores executable instructions, and the executable instructions are loaded and executed by a processor to implement the digital human generation method described in the above embodiment.

[0240] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the digital human generation method provided by each of the above method embodiments.

[0241] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned computer-readable storage medium may be a read-only memory, a disk or an optical disk, etc.

[0242] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switching, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating a digital human, characterized in that: The method comprises: Get the user's face image; Based on the facial features of the human face in the facial image, generating a digital human image of the digital human, wherein the digital human image is a virtual image having the facial features; Based on the facial expression of the human face in the facial image, the voice emotion of the digital human is determined.

2. The method according to claim 1, characterized in that The digital human image is generated by an image generation model, and the image generation model includes a convolutional neural network and a generative adversarial network; The step of generating a digital human image of a digital human based on the facial features of the human face in the facial image comprises: Extracting features of the face image by using the convolutional neural network to obtain the facial features of the face in the face image; The facial features are input into the generative adversarial network, and the digital human image of the digital human is generated through the generative adversarial network.

3. The method according to claim 2, characterized in that The method further comprises: Extracting features of the sample face image by using the convolutional neural network to obtain sample facial features of the sample face in the sample face image; Inputting the sample facial features into the generative adversarial network, and generating a sample digital human image through the generative adversarial network; Determining a model loss of the image generation model based on the sample digital human image and the sample facial features; The convolutional neural network and the generative adversarial network are trained based on the model loss.

4. The method according to claim 3, characterized in that The determining of the model loss of the image generation model based on the sample digital human image and the sample facial features includes: Extracting features of the sample digital human image by using the convolutional neural network to obtain facial features of the sample digital human of the sample digital human image; Determine facial feature loss based on the facial features of the sample digital person and the sample facial features; Determining the image generation loss based on the discrimination result of the sample digital human image and the real digital human image by the discriminator in the generative adversarial network; Based on the facial feature loss, the first loss weight corresponding to the facial feature loss, the image generation loss and the second loss weight corresponding to the image generation loss, the model loss of the image generation model is determined, wherein the first loss weight is negatively correlated with the training round, and the second loss weight is positively correlated with the training round.

5. The method according to claim 2, characterized in that: The method further comprises: The image generation model corresponding to the digital human image style is determined, wherein different image generation models are used to generate digital human images of different styles.

6. The method according to claim 2, characterized in that The step of determining the voice emotion of the digital human based on the facial expression of the human face in the facial image comprises: Performing expression classification on the facial features of the human face in the face image by an expression classifier to obtain expression confidences corresponding to each candidate human face expression label; determining the voice emotion of the digital human based on the candidate human face expression label and the expression confidences; or, Based on the facial key points of the human face in the facial image, expression features of multiple dimensions are extracted; the expression features of multiple dimensions are input into an expression recognition network to obtain expression confidences corresponding to each candidate human face expression label output by the expression recognition network; based on the candidate human face expression label and the expression confidences, the voice emotion of the digital human is determined.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Obtaining the pre-recorded voice of the user; Based on the voice emotion and the pre-recorded voice, a voice package of the digital human is generated, wherein the voice package is used to characterize the voice features of the user, and the voice features have the voice emotion.

8. The method according to claim 7, characterized in that The step of generating the voice package of the digital human based on the voice emotion and the pre-recorded voice comprises: Extracting speech features from the pre-recorded speech using an LSTM model to obtain the speech features of the user; The prosodic features in the voice features are adjusted based on the voice emotion to obtain the voice package of the digital human.

9. The method according to claim 8, characterized in that The step of adjusting the prosodic features in the voice features based on the voice emotion to obtain the voice package of the digital human comprises: Determining a feature adjustment method based on the emotion type of the speech emotion, and determining a feature adjustment amount based on the emotion degree of the speech emotion; Based on the feature adjustment method and the feature adjustment amount, feature adjustment is performed on the prosodic feature in the voice feature to obtain the voice package of the digital human.

10. The method according to claim 7, characterized in that The obtaining of the pre-recorded voice of the user comprises: In the absence of the historical speech of the user, obtaining the pre-recorded speech corresponding to the target corpus, where the target corpus is the corpus required to generate the speech package; In the case where there is a historical voice of the user, the missing corpus is determined based on the existing corpus corresponding to the historical voice and the target corpus, where the target corpus is the corpus required to generate the voice package; and the pre-recorded voice corresponding to the missing corpus is obtained.

11. A digital human generation device, characterized in that: The device comprises: An acquisition module is used to acquire a user's face image; A generating module, configured to generate a digital human image of a digital human based on the facial features of the human face in the facial image, wherein the digital human image is a virtual image having the facial features; The generating module is further used to determine the voice emotion of the digital human based on the facial expression of the human face in the facial image.

12. A computer device, characterized in that: The computer device comprises: a processor and a memory, wherein the memory stores at least one program; the processor is used to execute the at least one program in the memory to implement the digital human generation method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The readable storage medium stores executable instructions, and the executable instructions are loaded and executed by a processor to implement the digital human generation method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor reads and executes the computer instructions from the computer-readable storage medium to implement the digital human generation method as described in any one of claims 1 to 10.

Images