Interactive digital human generation method and system based on artificial intelligence

Abstract

The invention relates to an artificial intelligence-based interactive digital human generation method, which comprises the following steps of defining basic features and generation requirements of a digital human through a condition vector, generating a potential space representation by using an encoder, generating a preliminary image by using a decoder, and adjusting the image in combination with a style migration technology to adapt to a target environment, and improving the resolution and details by using a super-resolution network, and finally generating a static digital human image. And collecting multi-modal data, and integrating features through a fusion function to generate a unified representation. Based on conditional vectors, random noise, and multi-modal unified features, expressions and speech representations conforming to emotions or states are generated. And a basis is provided for expression and voice response of the digital human by identifying the emotion of the user. Real-time response is generated through a reaction function, and more accurate and intelligent interaction is achieved by combining reinforcement learning optimization behaviors and response strategies. A high-quality personalized digital human image can be generated, and intelligent response is realized through multi-modal interaction and emotion perception.

Description

Technical Field

[0001] The present invention relates to the field of digital human interaction technology, and more specifically, to an interactive digital human generation method and system based on artificial intelligence. Background Art

[0002] In the current field of digital human technology, although there have been many developments and applications, there are still some key technical challenges and limitations, including the complexity of data integration, the lack of realism in generated data, the accuracy of emotion recognition, and the need to optimize real-time responses. When generating facial expressions or voice responses, existing digital humans often cannot adapt well to the emotions or states of specific situations, lacking realism and natural fluency, which affects the user's immersive experience and satisfaction. The current system often lacks accuracy and efficiency in identifying user emotions and generating corresponding responses, especially when faced with complex emotional expressions. The system's response is not accurate or timely, resulting in low interaction efficiency. Summary of the invention

[0003] The present invention provides an interactive digital human generation method based on artificial intelligence, which can improve customers' financial management ability and acceptance rate and reduce risk factors. At the same time, by reducing the risk factor, it can reversely improve customers' financial management ability and acceptance rate, thereby finding a profit balance point and improving business profitability.

[0004] The present invention provides an interactive digital human generation method based on artificial intelligence, comprising the following steps:

[0005] Digital human image generation steps: define the input conditions of the digital human, which are represented by a conditional vector that describes the basic characteristics and generation requirements of the digital human; input the conditional vector into the encoder to generate a latent space representation, pass the sampled latent vector through the decoder to generate a preliminary digital human image, use reconstruction loss to ensure that the generated image meets the characteristics of the input conditions, use KL divergence to ensure the continuity of the latent space distribution, apply style transfer technology to adapt the preliminary generated digital human image to the target environment or style, use a super-resolution network to improve the resolution and detail quality of the image, and output the final high-resolution image;

[0006] Digital human behavior generation steps: collect multimodal data and convert it into a corresponding feature vector set, wherein the multimodal data includes facial expressions, voices and body movements; construct a fusion function, wherein the fusion function is used to integrate multiple feature vectors in the feature vector set into a unified feature representation, wherein the fusion function is a multi-layer neural network, and the data of different modalities are mapped to a unified feature space through the weight matrix in the neural network; define a loss function, which is used to evaluate the difference between the generated digital human behavior and the preset target; adjust the loss function parameters through an optimization algorithm to minimize the loss function value, wherein the optimization algorithm includes using a stochastic gradient descent or an Adam algorithm;

[0007] Digital human performance generation steps: set a conditional vector to represent a specific emotion or state; receive the unified feature representation of random noise, conditional vector and multimodal data through the generator, and generate facial expression or voice data corresponding to the set emotion or state; use the discriminator to evaluate the matching degree between the generated data and the conditional vector and judge its authenticity; optimize the generator and discriminator by minimizing the adversarial loss function to improve the authenticity and matching degree of the generated data, and construct a performance data synthesizer;

[0008] Digital human emotion recognition steps: obtain the user's facial image, use the pre-trained convolutional neural network model to extract features from the user's facial image; input the extracted feature vector into the classifier to recognize the user's specific expression, the classifier is a support vector machine or a multi-layer perceptron; according to the recognized expression category, the digital human generates a corresponding facial expression or voice response;

[0009] Digital human response optimization steps: receiving user input information, which includes voice commands or text information; processing user input and generating corresponding responses through a preset reaction function; adjusting the parameters of the reaction function through feedback to ensure the accuracy and timeliness of the response.

[0010] Preferably, in the digital human behavior generation step, feature vectors of facial expressions, voices and body movements are extracted from real-time video and audio data, the fusion function is in the form of a combination of convolutional layers and fully connected layers, which are used to process and integrate input data of different modalities, and the loss function is cross entropy or mean square error, which is used to evaluate the consistency between the output predicted by the model and the actual label or expected output.

[0011] Preferably, in the digital human behavior generation step, the multimodal data includes facial expressions E, voice S, and body movements A, expressed as:

[0012] E={e1,e2,…,e m}

[0013] S={s1,s2,…,sn}

[0014] A={a1,a2,…,a p}

[0015] where e i ,s j ,a k Represent the feature vectors of collected facial expressions, voices, and body movements respectively;

[0016] Construct a fusion function F, which is used to fuse feature vectors into a unified representation: multi =F(e i ,s j ,a k ), the fusion function F is a multi-layer neural network, through the weight matrix W e ,W s ,W a Mapping data of different modalities into a common feature space can be expressed as:

[0017] x multi =F = Activation(W e e+W s e+W a a+b)

[0018] Where W e ,W s ,W a are the weight matrices corresponding to facial expressions, speech, and body movements, b is the bias term, and Activation is the activation function;

[0019] The loss function L that minimizes the fusion function is defined as: L fusion =∑ i ∥x multi,i -x target,i ∥ 2 , use stochastic gradient descent to optimize the above loss function L.

[0020] Preferably, in the step of generating a digital human performance,

[0021] Set a condition vector C emotion , representing a specific emotion or state;

[0022] Generator G(z,c) is used to accept random noise z, conditional vector C emotion and a unified feature representation x for multimodal data multi , output the generated data I behavior , expressed as: I behavior =G(z,x multi ,C emotion θbehavior ), the generated data I behavior For the digital human’s facial expressions or voice;

[0023] Discriminator D(I behavior ,C emotion ) is used to evaluate the authenticity of the generated data, specifically expressed as: D(I behavio ,C emotion )∈[0,1];

[0024] Minimize the following adversarial loss function of the generator and discriminator, expressed as:

[0025]

[0026] The objective function G is a function used to generate the behavior and performance of the digital human. It accepts input features and outputs the generated digital human performance, expressed as: G(x multi ,C emotion ,z;θ G )→I behavior .

[0027] Preferably, in the digital human emotion recognition step, the user's facial image face_image is first obtained, and then a pre-trained CNN model is used to extract features from the user's facial image, which is expressed as: F = CNN (face_image; θ cnn ), where F is the high-dimensional feature vector of the user’s facial image, θ cnn are the parameters of the CNN model;

[0028] The extracted feature vector is input into the classifier to identify the specific emotion, which is expressed as: E = Classifier(F; θ cls ), where E is the identified emotion category, θ cls are the parameters of the classifier;

[0029] The recognized expression feature E is used as an additional input feature to supplement the multimodal data to adjust the fusion function. The adjusted fusion function is expressed as x = F(e i ,s j ,a k ,E); (to be explained in the subsequent instructions)

[0030] Taking the emotion feature E as a condition and combining it with random noise, the generator outputs facial expressions or voice data consistent with the emotion, expressed as x′=G(z,c;θ g ). (The following instructions will explain)

[0031] Preferably, in the step of generating the digital human image, the input condition C of the digital human is defined, the condition vector and the random noise are input into the encoder, the latent space representations μ and σ are generated, and the latent vector z is generated in the latent space, which is expressed as:

[0032] z~N(μ,σ 2 ),[μ,σ]=Encoder(C;θenc)

[0033] The sampled latent vector z is passed through the decoder to generate a preliminary digital human image I inutual , expressed as:

[0034] I unitial =Decoder(z;θdec)

[0035] The reconstruction loss is used to ensure that the generated image meets the characteristics of the input conditions, expressed as:

[0036] The initially generated image I initial Adjust to the target style S and get the adjusted image L style , and apply the content loss function L content Ensure that the generated image retains the content characteristics of the initial image, expressed as:

[0037] Apply the style loss function L style , minimize the difference in style representation between the generated image and the reference style image, expressed as:

[0038] Apply the content loss function L content Ensure that the generated image retains the content characteristics of the initial image, expressed as:

[0039] Apply the style loss function L style , minimize the difference in style representation between the generated image and the reference style image, expressed as: Where ψ(x) represents the style features of the style image x, G represents the Gram matrix at the lth layer, and w l is the layer weight;

[0040] Combining content loss and style loss, the visual effect of the generated image is adjusted by optimizing the total loss function to match the specific virtual environment, which is expressed as: L total =αL content +βL style , where α and β are the weights of content loss and style loss respectively;

[0041] Output the final static image I static .

[0042] Preferably, in the digital human image generation step, the step further includes: static and multimodal representation I behavior , generate dynamic animation, which includes action sequences and expression changes. The specific steps are:

[0043] Enter action sequence condition C motion , generate action data A motion , expressed as:

[0044] A motion =M(C motion θ motion )

[0045] The generated motion data is combined with the static image and performance data to generate animation frames, represented as:

[0046] I animated =Animation(I static ,I behavior ,A motion ).

[0047] Preferably, in the step of optimizing the response of the digital human, the user's input data u(t) occurs at time t, where u(t) is a voice command or text information, expressed as u(t)=Input(t); the response r(t) of the digital human is generated using the reaction function R, and the reaction function R is expressed as: r(t)=R(u(t),θ r ), where θ r are the parameters of the reaction function R, and the parameters of the reaction function R are optimized through training data.

[0048] Preferably, it also includes a method of continuously optimizing the behavior and reaction strategy of the digital human by using reinforcement learning, which specifically comprises the following steps:

[0049] Define a set of possible actions based on the current system state and the context of user interaction; let S t A represents the system state at time t, including the current state of the digital human and the context of user interaction. t Indicates that in state S t The set of actions that can be taken under state S is learned through the reinforcement learning algorithm. Strategy π selects t The best action a t , expressed as: a t =π(S t ,θ π ), where θ π are the parameters of the strategy;

[0050] Actiona t Causes the state to change to S t+1, and generates an immediate reward r t+1 , expressed as:

[0051] S t+1 ,r t+1 =Environment(S t ,a t )

[0052] Using the reinforcement learning algorithm, learn the strategy of selecting actions to maximize the expected value of the cumulative reward; the goal of reinforcement learning is to optimize the strategy π to maximize the expected value of the cumulative reward, expressed as:

[0053]

[0054] Where γ is the discount factor, which represents the current value of future rewards;

[0055] According to the results of the action and the immediate rewards generated, the policy parameters are updated to improve future behavior decisions; the policy parameters θ π Updated by the gradient ascent method to maximize the long-term benefit, expressed as:

[0056]

[0057] Among them, α is the learning rate, is J versus θ π The gradient of is calculated by the policy gradient method;

[0058] Generate comprehensive response function R ′ (t), expressed as: R ′ (t) = {r(t), behavior(a t )}.

[0059] The present invention also provides an interactive digital human generation system based on artificial intelligence, based on the aforementioned interactive digital human generation method based on artificial intelligence, characterized in that it includes the following modules:

[0060] Digital human image generation module: used to define the input conditions of the digital human. The input conditions are represented by a condition vector, which describes the basic characteristics and generation requirements of the digital human. The condition vector is input into the encoder to generate a latent space representation. The sampled latent vector is passed through the decoder to generate a preliminary digital human image. The reconstruction loss is used to ensure that the generated image meets the characteristics of the input conditions. The KL divergence is used to ensure the continuity of the latent space distribution. Based on the preliminary generated digital human image, the style transfer technology is applied to adapt it to the target environment or style. The super-resolution network is used to improve the resolution and detail quality of the image, and the high-resolution image is output as the final image.

[0061] Digital human behavior generation module: used to collect multimodal data and convert it into a corresponding feature vector set, wherein the multimodal data includes facial expressions, voices and body movements; construct a fusion function, wherein the fusion function is used to integrate multiple feature vectors in the feature vector set into a unified feature representation, wherein the fusion function is a multi-layer neural network, and the data of different modalities are mapped to a unified feature space through the weight matrix in the neural network; define a loss function, which is used to evaluate the difference between the generated digital human behavior and the preset target; adjust the loss function parameters through an optimization algorithm to minimize the loss function value, wherein the optimization algorithm includes using a stochastic gradient descent or an Adam algorithm;

[0062] Digital human performance generation module: used to set a conditional vector to represent a specific emotion or state; through the generator, receive the unified feature representation of random noise, conditional vector and multimodal data, and generate facial expression or voice data corresponding to the set emotion or state; use the discriminator to evaluate the matching degree between the generated data and the conditional vector and judge its authenticity; optimize the generator and discriminator by minimizing the adversarial loss function to improve the authenticity and matching degree of the generated data, and construct a performance data synthesizer;

[0063] Digital human emotion recognition module: used to obtain the user's facial image, extract features from the user's facial image using a pre-trained convolutional neural network model; input the extracted feature vector into a classifier to recognize the user's specific expression, the classifier being a support vector machine or a multi-layer perceptron; based on the recognized expression category, the digital human generates a corresponding facial expression or voice response;

[0064] Digital human response optimization module: used to receive user input information, which includes voice instructions or text information; process user input and generate corresponding responses through preset reaction functions; adjust the parameters of the reaction function through feedback to ensure the accuracy and timeliness of the response.

[0065] The beneficial effects of the present invention are as follows: the present scheme constructs an intelligent, efficient and multifunctional digital human system through a complete digital human image, behavior, performance, emotion and response generation and optimization process. First, based on the generation of conditional vectors and the structured representation of latent space, style transfer and super-resolution technology are combined. The continuity of the latent space ensures that the generated digital human image strikes a balance between feature consistency and diversity. Style transfer technology enables the digital human image to adapt to different virtual environments or artistic styles and has higher visual adaptability. The application of super-resolution networks significantly improves the resolution and detail quality of images, meeting the high-demand visual presentation requirements. The generated digital human image not only has personalized features, but also can adapt to a variety of application scenarios, with clear image details and extremely high visual appeal. Generation based on the fusion and unified feature representation of multimodal data. The fusion function constructed by a multi-layer neural network maps multimodal data to a common feature space to capture the correlation information between different modalities. The generated digital human behavior has high precision and high consistency, can accurately express the target emotions and actions, and achieve a natural and smooth interactive experience, which is suitable for virtual hosts, education and training and other scenarios. A generative adversarial network framework combining random noise, emotional conditional vectors and multimodal feature representation. Make the generated digital human behave naturally and realistically, and the emotional expression is highly consistent with the set goals. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work:

[0067] Figure 1 It is a flowchart of an interactive digital human generation method based on artificial intelligence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solution in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0069] like Figure 1 As shown, the present invention provides an interactive digital human generation method based on artificial intelligence, comprising the following steps:

[0070] Digital human image generation steps: define the input conditions of the digital human, which are represented by a conditional vector that describes the basic characteristics and generation requirements of the digital human; input the conditional vector into the encoder to generate a latent space representation, pass the sampled latent vector through the decoder to generate a preliminary digital human image, use reconstruction loss to ensure that the generated image meets the characteristics of the input conditions, use KL divergence to ensure the continuity of the latent space distribution, apply style transfer technology to adapt the preliminary generated digital human image to the target environment or style, use a super-resolution network to improve the resolution and detail quality of the image, and output the final high-resolution image;

[0071] Digital human behavior generation steps: collect multimodal data and convert it into a corresponding feature vector set, wherein the multimodal data includes facial expressions, voices and body movements; construct a fusion function, wherein the fusion function is used to integrate multiple feature vectors in the feature vector set into a unified feature representation, wherein the fusion function is a multi-layer neural network, and the data of different modalities are mapped to a unified feature space through the weight matrix in the neural network; define a loss function, which is used to evaluate the difference between the generated digital human behavior and the preset target; adjust the loss function parameters through an optimization algorithm to minimize the loss function value, wherein the optimization algorithm includes using a stochastic gradient descent or an Adam algorithm;

[0072] Digital human performance generation steps: set a conditional vector to represent a specific emotion or state; receive the unified feature representation of random noise, conditional vector and multimodal data through the generator, and generate facial expression or voice data corresponding to the set emotion or state; use the discriminator to evaluate the matching degree between the generated data and the conditional vector and judge its authenticity; optimize the generator and discriminator by minimizing the adversarial loss function to improve the authenticity and matching degree of the generated data, and construct a performance data synthesizer;

[0073] Digital human emotion recognition steps: obtain the user's facial image, use the pre-trained convolutional neural network model to extract features from the user's facial image; input the extracted feature vector into the classifier to recognize the user's specific expression, the classifier is a support vector machine or a multi-layer perceptron; according to the recognized expression category, the digital human generates a corresponding facial expression or voice response;

[0074] Digital human response optimization steps: receiving user input information, which includes voice commands or text information; processing user input and generating corresponding responses through a preset reaction function; adjusting the parameters of the reaction function through feedback to ensure the accuracy and timeliness of the response.

[0075] This solution builds an intelligent, efficient and multifunctional digital human system through a complete digital human image, behavior, performance, emotion and response generation and optimization process. First, based on the generation of conditional vectors and the structured representation of latent space, style transfer and super-resolution technology are combined. The continuity of the latent space ensures that the generated digital human image strikes a balance between feature consistency and diversity. Style transfer technology enables digital human images to adapt to different virtual environments or artistic styles and have higher visual adaptability. The application of super-resolution networks significantly improves the resolution and detail quality of images to meet the high-demand visual presentation requirements. The generated digital human image not only has personalized features, but also can adapt to a variety of application scenarios (such as the metaverse, film and television production, virtual assistants), with clear image details and extremely high visual appeal.

[0076] Based on the fusion and unified feature representation generation of multimodal data (facial expressions, voice, body movements). Through the fusion function constructed by multi-layer neural network, multimodal data is mapped to a common feature space to capture the correlation information between different modalities. The generated digital human behavior has high precision and high consistency, can accurately express the target emotions and movements, and achieve a natural and smooth interactive experience. It is suitable for scenarios such as virtual hosts, education and training.

[0077] The generative adversarial network framework combines random noise, emotional condition vectors and multimodal feature representation. The generated digital human is natural and realistic, and the emotional expression is highly consistent with the set goals, which is suitable for emotional interaction scenarios.

[0078] The user's facial features are extracted through a convolutional neural network, and combined with a support vector machine or a multi-layer perceptron classifier to identify specific expression categories. Accurate emotion recognition algorithms provide a basis for digital humans to generate targeted facial expressions or voice responses. Digital humans can quickly perceive user emotions and make adaptive performances, improving the personalization of interactions and user experience satisfaction.

[0079] The reaction function and reinforcement learning strategy optimization based on user input ensure that the response of the digital human continues to adapt to user needs. The parameters of the reaction function are adjusted through instant feedback to ensure the real-time and accuracy of the response. The behavior and response strategy are optimized in the long term using reinforcement learning to maximize the long-term benefits of interaction. The digital human can not only respond quickly to user input, but also continuously improve the interaction effect through learning to form an adaptive and intelligent response mechanism, which is suitable for application scenarios such as smart assistants and virtual assistants that require long-term learning. This solution seamlessly combines multiple modules such as image generation, behavior generation, performance generation, emotion recognition, and response optimization to build a complete digital human generation and optimization system.

[0080] This solution is not only suitable for static scenes (such as advertising and artistic creation), but also supports dynamic interactive scenes (such as virtual meetings, education and training, and metaverse interaction), and has strong scalability.

[0081] This solution builds an intelligent, personalized, and adaptable digital human system through clear creative points and optimization technologies in each link of digital human generation, which has the following outstanding advantages:

[0082] High-quality image generation: The visual effects are clear and realistic with diverse styles.

[0083] Accurate behavioral performance: natural coordination, matching user emotions.

[0084] Intelligent response optimization: real-time response, continuous learning, and enhanced interactive experience.

[0085] In the digital human behavior generation step of this embodiment, feature vectors of facial expressions, voices, and body movements are extracted from real-time video and audio data. The fusion function is a combination of a convolutional layer and a fully connected layer to process and integrate input data of different modalities. The loss function is a cross entropy or mean square error, which is used to evaluate the consistency between the output predicted by the model and the actual label or expected output.

[0086] In the digital human behavior generation step of this embodiment, the multimodal data includes facial expressions E (feature vectors extracted from video frames by facial recognition algorithms, each feature vector e i Represents facial expression features at a moment), voice S (voice features extracted by audio processing algorithms, such as Mel frequency cepstral coefficients), and body movements A (bone movement or joint angle features extracted by motion capture technology, each a k represents a feature vector of an action frame), expressed as:

[0087] E={e1,e2,…,e m}

[0088] S={s1,s2,…,s n}

[0089] A={a1,a2,…,a p}

[0090] where e i ,s j ,a k Represent the feature vectors of facial expressions, voices, and body movements respectively; Multimodal data comes from different acquisition frequencies, and the time axis of different modal data needs to be synchronized through a time alignment algorithm to ensure that the features of corresponding time points can be fused; Multimodal data is fused through a fusion module;

[0091] Construct a fusion function F, which is used to fuse feature vectors into a unified representation: multi =F(e i ,s j ,a k ), the fusion function F is a multi-layer neural network, through the weight matrix W e ,W s ,W a Mapping data of different modalities into a common feature space can be expressed as:

[0092] x multi =F = Activation(W e e+W s ·s+W a a+b)

[0093] Where W e ,W s ,W a are the weight matrices corresponding to facial expressions, speech, and body movements, respectively. b is the bias term and Activation is the activation function. The activation function Activation can select a nonlinear function (such as ReLU, Tanh, or Sigmoid) to introduce nonlinear capabilities.

[0094] The network structure of the fusion model includes the input layer (receiving e i ,s j ,a k three modal features), hidden layer (several neural network layers, used to learn the correlation between different modalities), output layer (generate a unified feature representation x multi ).

[0095] The loss function L that minimizes the fusion function is defined as: L fusion =∑ i ∥x multi,i -x target,i ∥ 2 , where x multi,i is the feature representation generated by the fusion function F, x target,i is the target feature vector, which comes from the expected output. The above loss function L is optimized using stochastic gradient descent, expressed as:

[0096] The training process of the fusion model is as follows: Initialize the network weight W e ,W s ,W a and bias b; input multimodal feature e i ,s j ,a k ; Forward propagation calculation xmulti ; Calculate the loss L fusion ; Back propagation updates the parameters θ F ; Repeat the training until the loss converges.

[0097] The fusion function uses a neural network to integrate multimodal features (facial expressions, voice, and actions) to generate a unified feature representation. The generated unified feature representation more comprehensively reflects the user's emotions, intentions, and behavioral characteristics, and improves the accuracy of subsequent digital human behavior generation. The loss function directly optimizes the matching degree between the fused features and the target features, and continuously narrows the feature gap during the training process, making the fused features more in line with the real situation. Use random gradient descent or algorithms to optimize network parameters, quickly converge to the optimal parameters, improve training efficiency, and adapt to large-scale data training. The deep fusion of multimodal features provides high-quality input for the digital human behavior generation step. By deeply fusing multimodal features in the behavior generation step, this solution achieves the unified expression of multimodal data, the precise matching of target features, and the efficient optimization of the training process. These improvements significantly improve the accuracy and coordination of digital human behavior generation, giving it obvious technical advantages and application value in terms of realism, naturalness, and adaptability.

[0098] In the step of generating the digital human performance in this embodiment,

[0099] Set a condition vector C emotion , representing a specific emotion or state;

[0100] Generator G(z,c) is used to accept random noise z, conditional vector C emotion and a unified feature representation x for multimodal data multi , output the generated data I behavior , expressed as: I behavior =G(z,x multi ,C emotion θ behavior ), the generated data I behavio For the digital human’s facial expressions or voice;

[0101] Discriminator D(I behavior ,C emotion ) is used to evaluate the authenticity of the generated data, specifically expressed as: D(I behavior ,C emotion )∈[0,1];

[0102] Minimize the following adversarial loss function of the generator and discriminator, expressed as:

[0103]

[0104] The objective function G is a function used to generate the behavior and performance of the digital human. It accepts input features and outputs the generated digital human performance, expressed as: G(x multi ,C emotion ,z;θ G )→I behavior .

[0105] Condition vector C emotion Represents a specific emotion or state of a digital person, such as happiness, seriousness, anger, etc. Use one-hot encoding to represent the type of emotion, or use continuous values ​​(such as emotional intensity) to represent the amplitude and subtle changes of emotions. Random noise z is used to introduce diversity so that the generated representation is not limited to fixed features and avoids mode collapse. Usually sampled from a standard normal distribution. x multi It is a unified feature representation obtained by fusing facial expression, speech and body movement features. multi It provides the behavioral foundation for the digital human, ensuring that the generated performance is consistent with the input data.

[0106] The generator structure includes: input layer (receive z, x multi and C emotion ), hidden layer (using a multi-layer neural network to capture the relationship between multimodal features and emotional conditions), and output layer (generating target performance data I behavior ).

[0107] The discriminator structure includes: input layer (receiving I behavior and C emotion ), hidden layer (using a multi-layer neural network to extract features), and output layer (binary classification probability value to evaluate the authenticity of the input data).

[0108] For real data I real The log probability of , encourages the discriminator to correctly identify the real data.

[0109] E fake [log(1-D(G(z,x multi ,C emotion ),C emotion ))] is to generate data I fake The goal of the generator is to make the discriminator think that the generated data is real.

[0110] Alternately update the generator and discriminator until the generator can generate realistic-looking representation data.

[0111] This embodiment achieves high-quality generation of digital human performance by fusing conditional vectors, random noise and multimodal features, combined with a generative adversarial network. The optimization of the adversarial loss function enables the generated data to achieve a high degree of balance in terms of authenticity, diversity and emotional matching, providing an integrated solution for digital human performance generation, which is suitable for a variety of application scenarios such as virtual reality, metaverse, and intelligent assistants.

[0112] In the digital human emotion recognition step of this embodiment, the user's facial image face_image is first obtained, and then the pre-trained CNN model is used to extract features from the user's facial image, which is expressed as: F = CNN (face_image; θ cnn ), where F is the high-dimensional feature vector of the user’s facial image, θ cnn are the parameters of the CNN model;

[0113] The extracted feature vector is input into the classifier to identify the specific emotion, which is expressed as: E = Classifier(F; θ cls ), where E is the identified emotion category, θ cls are the parameters of the classifier;

[0114] The recognized expression feature E is used as an additional input feature to supplement the multimodal data to adjust the fusion function. The adjusted fusion function is expressed as x = F(e i ,s j ,a k ,E);

[0115] Taking the emotion feature E as a condition and combining it with random noise, the generator outputs facial expressions or voice data consistent with the emotion, expressed as x′=G(z,c;θ g ).

[0116] The user's facial image face_image is captured in real time by a camera and preprocessed to ensure the accuracy of feature extraction.

[0117] Preprocessing steps:

[0118] Grayscale or normalize the image.

[0119] Resize the image to match the CNN input size (e.g. 224 x 224).

[0120] Data augmentation (such as rotation, flipping) is used to improve the generalization ability of the model.

[0121] Use a pre-trained convolutional neural network (CNN) to extract a high-dimensional feature vector F from the facial image; CNN model: You can choose architectures such as ResNet, VGG, or MobileNet, and the pre-trained parameters 0cmn come from large-scale facial datasets (such as FER-2013 or AffectNet). The extracted features F include information about local facial areas (such as eyes, corners of the mouth, etc.), which helps to accurately recognize expressions.

[0122] The extracted feature vector F is input into the classifier to identify the specific emotion category. The classifier type can be support vector machine (SVM) or multi-layer perceptron (MLP). The output emotion category E represents the identified emotion, such as "happy", "angry", "neutral", etc.

[0123] The emotion feature E becomes part of the multimodal feature through the fusion function, and together with the facial expression, voice and action data, forms a more comprehensive unified representation. The emotion feature E provides emotional information and works in synergy with other modal data to improve the accuracy and naturalness of behavior generation and performance generation.

[0124] Using pre-trained CNN to extract facial features can capture micro-expression changes in local areas and greatly improve the accuracy of emotion recognition. The classifier further enhances the generalization ability of emotion recognition and is applicable to a variety of emotional scenarios. Taking emotional features as important inputs and supplementing them to multimodal features, the multimodal data's ability to perceive emotional changes is enhanced. By adjusting the fusion function structure, the synergy between emotional features and facial expressions, voice and actions is significantly enhanced, improving the quality of multimodal representation. Using emotional features and random noise as conditions, performance data consistent with specific emotions is generated. The features provided by the emotion recognition module are seamlessly connected to the multimodal fusion and performance generation modules to ensure the overall coordination of the system. The generated facial expression and voice data are consistent with the emotional conditions and have a higher sense of reality and naturalness.

[0125] In the digital human image generation step of this embodiment, the input condition C of the digital human is defined, and the condition vector and random noise are input into the encoder to generate latent space representations μ and σ, and a latent vector z is generated in the latent space, which is expressed as:

[0126] z~N(μ,σ 2 ),[μ,σ]=Encoder(C;θenc)

[0127] The sampled latent vector z is passed through the decoder to generate a preliminary digital human image I initial , expressed as:

[0128] I initial =Decoder(z;θdec)

[0129] The reconstruction loss is used to ensure that the generated image meets the characteristics of the input conditions, expressed as:

[0130] The initially generated image I initial Adjust to the target style S and get the adjusted image L style , and apply the content loss function L content Ensure that the generated image retains the content characteristics of the initial image, expressed as:

[0131] Apply the style loss function L style , minimize the difference in style representation between the generated image and the reference style image, expressed as:

[0132] Apply the content loss function L content Ensure that the generated image retains the content characteristics of the initial image, expressed as:

[0133] Apply the style loss function L style , minimize the difference in style representation between the generated image and the reference style image, expressed as: Where ψ(x) represents the style features of the style image x, G represents the Gram matrix at the lth layer, and w l is the layer weight;

[0134] Combining content loss and style loss, the visual effect of the generated image is adjusted by optimizing the total loss function to match the specific virtual environment, which is expressed as: L total =αL content +βL style , where α and β are the weights of content loss and style loss respectively;

[0135] Output the final static image I static .

[0136] The basic characteristics of the digital human (such as gender, age, style requirements, etc.) are provided by the conditional vector, and random noise is combined to ensure that each generated image is both conditional and diverse. The parameters of the encoder to generate the latent space ensure the continuity of the latent space distribution, so that the digital human image is consistent in the input conditions and supports flexible adjustment. The generated digital human image is highly personalized and can be adapted to a variety of application scenarios (such as virtual assistants, game characters, educational platforms, etc.). The initial generated image is converted into a reference style image so that the digital human image can adapt to different virtual environments (such as science fiction style, cartoon style, and realistic style). Content loss is used to ensure that the generated image still retains the core characteristics of the initial generated image after stylization adjustment. Style loss minimizes the difference in style representation between the generated image and the target style image, enhancing visual expression. The digital human image can be naturally integrated into different scenes, providing an immersive experience, which is suitable for a variety of scenarios such as the metaverse, film and television production, and social platforms. The features of the initial image, such as facial features, posture, etc., are retained through content loss, while style loss optimizes details such as color and texture. The generated image achieves a balance between content and style, with delicate visual expression and high-quality details, meeting high-resolution requirements.

[0137] In the digital human image generation step of this embodiment, the steps of generating a digital human image based on the static image I static and multimodal representation I behavior , generate dynamic animation, which includes action sequences and expression changes. The specific steps are:

[0138] Enter action sequence condition C motion , generate action data A motion , expressed as:

[0139] A motion =M(C motion θ motion )

[0140] The generated motion data is combined with the static image and performance data to generate animation frames, represented as:

[0141] I animated =Animation(I static ,I behavior ,A motion ).

[0142] This embodiment realizes the generation of dynamic animation by organically combining static image with multimodal expression and action sequence. This design has the following beneficial effects: Dynamicity and realism: The action sequence condition controls the generated action data, so that the expression changes and actions of the digital human are smooth and logical, and the generated animation frames are more natural and realistic visually, which enhances the immersiveness and expressiveness of the interaction. Modularity and flexibility: The dynamic animation generation process is based on the combination of independent modules (static image, expression data, action data), supports flexible expansion and personalized customization, and can adapt to various scene requirements (such as virtual reality, education and training, metaverse, etc.). Efficiency and controllability: The action sequence condition provides clear control input, and the user can accurately design the action sequence and expression changes. The generation process is efficient and the output animation is highly predictable, which is suitable for real-time applications. Wide applicability: The generated dynamic animation has realism and flexibility, and can be applied to virtual anchors, film and television production, intelligent assistants and other fields. Through the above design, this embodiment significantly improves the dynamic expression ability of the digital human image.

[0143] In the digital human response optimization step of this embodiment, the user's input data u(t) occurs at time t, where u(t) is a voice command or text information, expressed as u(t)=Input(t); the digital human's response r(t) is generated using the reaction function R, and the reaction function R is expressed as: r(t)=R(u(t),θ r ), where θ r are the parameters of the reaction function R, and the parameters of the reaction function R are optimized through training data.

[0144] In this embodiment, a method for continuously optimizing the behavior and reaction strategy of the digital human by using reinforcement learning is also included, which specifically comprises the following steps:

[0145] Define a set of possible actions based on the current system state and the context of user interaction; let S t A represents the system state at time t, including the current state of the digital human and the context of user interaction. t Indicates that in state S t The set of actions that can be taken under state S is learned through the reinforcement learning algorithm. Strategy π selects t The best action a t , expressed as: a t =π(S t ,θ π ), where θ π are the parameters of the strategy;

[0146] Actiona t Causes the state to change to S t+1 , and generates an immediate reward r t+1 , expressed as:

[0147] S t+1 ,r t+1 =Environment(S t ,a t )

[0148] Using the reinforcement learning algorithm, learn the strategy of selecting actions to maximize the expected value of the cumulative reward; the goal of reinforcement learning is to optimize the strategy π to maximize the expected value of the cumulative reward, expressed as:

[0149]

[0150] Where γ is the discount factor, which represents the current value of future rewards;

[0151] According to the results of the action and the immediate rewards generated, the policy parameters are updated to improve future behavior decisions; the policy parameters θ π Updated by the gradient ascent method to maximize the long-term benefit, expressed as:

[0152]

[0153] Among them, α is the learning rate, is J versus θ π The gradient of is calculated by the policy gradient method;

[0154] Generate comprehensive response function R ′ (t), expressed as: R ′ (t) = {r(t), behavior(a t )}.

[0155] This embodiment combines the generation of immediate response based on reaction function with the reinforcement learning optimization strategy to generate a comprehensive reaction function, thereby realizing the dynamic adaptability and long-term optimization ability of the digital human to user input. The immediate response function ensures that the digital human can quickly process user input and generate accurate real-time feedback to meet the needs of real-time interaction. Through continuous optimization of behavioral decision-making strategies through reinforcement learning, the optimal action is selected in combination with contextual information, and a comprehensive response is generated, making the response of the digital human more intelligent and more adaptable to the situation. The optimization goal is centered on cumulative rewards, balancing immediate rewards and long-term value, so that the digital human can continuously improve its own interactive performance, improve user satisfaction and system performance. The system adopts a modular design, combines immediate response with long-term strategy optimization, supports a variety of user scenarios, and is suitable for a variety of application fields such as intelligent assistants, virtual customer service, education and training. Through this method, the digital human system realizes the improvement of the whole process from real-time interaction to intelligent behavior strategy optimization, greatly improving the interaction effect and user experience.

[0156] The present invention also provides an embodiment of an interactive digital human generation system based on artificial intelligence, which, based on the previous embodiment, includes the following modules:

[0157] Digital human image generation module: used to define the input conditions of the digital human. The input conditions are represented by a condition vector, which describes the basic characteristics and generation requirements of the digital human. The condition vector is input into the encoder to generate a latent space representation. The sampled latent vector is passed through the decoder to generate a preliminary digital human image. The reconstruction loss is used to ensure that the generated image meets the characteristics of the input conditions. The KL divergence is used to ensure the continuity of the latent space distribution. Based on the preliminary generated digital human image, the style transfer technology is applied to adapt it to the target environment or style. The super-resolution network is used to improve the resolution and detail quality of the image, and the high-resolution image is output as the final image.

[0158] Digital human behavior generation module: used to collect multimodal data and convert it into a corresponding feature vector set, wherein the multimodal data includes facial expressions, voices and body movements; construct a fusion function, wherein the fusion function is used to integrate multiple feature vectors in the feature vector set into a unified feature representation, wherein the fusion function is a multi-layer neural network, and the data of different modalities are mapped to a unified feature space through the weight matrix in the neural network; define a loss function, which is used to evaluate the difference between the generated digital human behavior and the preset target; adjust the loss function parameters through an optimization algorithm to minimize the loss function value, wherein the optimization algorithm includes using a stochastic gradient descent or an Adam algorithm;

[0159] Digital human performance generation module: used to set a conditional vector to represent a specific emotion or state; through the generator, receive the unified feature representation of random noise, conditional vector and multimodal data, and generate facial expression or voice data corresponding to the set emotion or state; use the discriminator to evaluate the matching degree between the generated data and the conditional vector and judge its authenticity; optimize the generator and discriminator by minimizing the adversarial loss function to improve the authenticity and matching degree of the generated data, and construct a performance data synthesizer;

[0160] Digital human emotion recognition module: used to obtain the user's facial image, extract features from the user's facial image using a pre-trained convolutional neural network model; input the extracted feature vector into a classifier to recognize the user's specific expression, the classifier being a support vector machine or a multi-layer perceptron; based on the recognized expression category, the digital human generates a corresponding facial expression or voice response;

[0161] Digital human response optimization module: used to receive user input information, which includes voice instructions or text information; process user input and generate corresponding responses through preset reaction functions; adjust the parameters of the reaction function through feedback to ensure the accuracy and timeliness of the response.

[0162] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. An interactive digital human generation method based on artificial intelligence, characterized in that: The following steps are involved: Digital human image generation steps: define the input conditions of the digital human, which are represented by a condition vector, describing the basic characteristics and generation requirements of the digital human; The conditional vector is input into the encoder to generate a latent space representation. The sampled latent vector is passed through the decoder to generate a preliminary digital human image. The reconstruction loss is used to ensure that the generated image meets the characteristics of the input condition. The KL divergence is used to ensure the continuity of the latent space distribution. Based on the preliminary generated digital human image, the style transfer technology is applied to adapt it to the target environment or style. The super-resolution network is used to improve the resolution and detail quality of the image, and the final image of the high-resolution image is output; Digital human behavior generation steps: collect multimodal data and convert it into a corresponding feature vector set, wherein the multimodal data includes facial expressions, voices and body movements; construct a fusion function, wherein the fusion function is used to integrate multiple feature vectors in the feature vector set into a unified feature representation, wherein the fusion function is a multi-layer neural network, and the data of different modalities are mapped to a unified feature space through the weight matrix in the neural network; define a loss function, which is used to evaluate the difference between the generated digital human behavior and the preset target; adjust the loss function parameters through an optimization algorithm to minimize the loss function value, wherein the optimization algorithm includes using a stochastic gradient descent or an Adam algorithm; Digital human performance generation steps: set a conditional vector to represent a specific emotion or state; receive the unified feature representation of random noise, conditional vector and multimodal data through the generator, and generate facial expression or voice data corresponding to the set emotion or state; use the discriminator to evaluate the matching degree between the generated data and the conditional vector and judge its authenticity; optimize the generator and discriminator by minimizing the adversarial loss function to improve the authenticity and matching degree of the generated data, and construct a performance data synthesizer; Digital human emotion recognition steps: obtain the user's facial image, use the pre-trained convolutional neural network model to extract features from the user's facial image; input the extracted feature vector into the classifier to recognize the user's specific expression, the classifier is a support vector machine or a multi-layer perceptron; according to the recognized expression category, the digital human generates a corresponding facial expression or voice response; Digital human response optimization steps: receiving user input information, which includes voice commands or text information; processing user input and generating corresponding responses through a preset reaction function; adjusting the parameters of the reaction function through feedback to ensure the accuracy and timeliness of the response.

2. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: In the digital human behavior generation step, feature vectors of facial expressions, voices, and body movements are extracted from real-time video and audio data. The fusion function is a combination of convolutional layers and fully connected layers to process and integrate input data of different modalities. The loss function is cross entropy or mean square error, which is used to evaluate the consistency between the output predicted by the model and the actual label or expected output.

3. The method for generating an interactive digital human based on artificial intelligence according to claim 2, characterized in that: In the digital human behavior generation step, the multimodal data includes facial expressions E, voice S, and body movements A, which are expressed as: E={e1,e2,...e m } S={s1,s2,...,s n } <h2 style=";text-align:left;direction:ltr">A = {a1, a2,..., a<h2 style=";text-align:left;direction:ltr"> p <h2 style=";text-align:left;direction:ltr">} where e i ,s j , a k Represent the feature vectors of collected facial expressions, voices, and body movements respectively; Construct a fusion function F, which is used to fuse feature vectors into a unified representation: multi =F(e i ,s j , a k ), the fusion function F is a multi-layer neural network, through the weight matrix W e , W s , W a Mapping data of different modalities into a common feature space can be expressed as: x multi =F=Activation(W e ·e+W s ·s+W a ·a+b) Where W e , W s , W a are the weight matrices corresponding to facial expressions, speech, and body movements, b is the bias term, and Activation is the activation function; The loss function L that minimizes the fusion function is defined as: L fusion =∑ i ||x multi，i -x target，i || 2 , use stochastic gradient descent to optimize the above loss function L.

4. The method for generating an interactive digital human based on artificial intelligence according to claim 3, characterized in that: In the digital human performance generation step, Set a condition vector C emotion , representing a specific emotion or state; The generator G(z,c) is used to accept random noise z, conditional vector C emotion and a unified feature representation x for multimodal data multi , output the generated data I behavior , expressed as: I behavior =G(z,x multi , C emotion θ behavior ), the generated data I behavior For the digital human’s facial expressions or voice; Discriminator D(I behavior , C emotion ) is used to evaluate the authenticity of the generated data, specifically expressed as: D(I behavior , C emotion )∈[0,1]; Minimize the following adversarial loss function of the generator and discriminator, expressed as: The objective function G is a function used to generate the behavior and performance of the digital human. It accepts input features and outputs the generated digital human performance, expressed as: G(x multi , C emotion , z; θ G )→I behavior .

5. The method for generating an interactive digital human based on artificial intelligence according to claim 4, characterized in that: In the digital human emotion recognition step, the user's facial image face_image is first obtained, and then the pre-trained CNN model is used to extract features from the user's facial image, which is expressed as: F = CNN (face_image; θ cnn ), where F is the high-dimensional feature vector of the user’s facial image, θ cnn are the parameters of the CNN model; The extracted feature vector is input into the classifier to identify the specific emotion, which is expressed as: E = Classifier(F; θ cls ), where E is the identified emotion category, θ cls are the parameters of the classifier; The recognized expression feature E is used as an additional input feature to supplement the multimodal data to adjust the fusion function. The adjusted fusion function is expressed as x = F(e i ,s j , a k , E); Taking the emotion feature E as a condition and combining it with random noise, the generator outputs facial expressions or voice data consistent with the emotion, expressed as x′=G(z, c; θ g ).

6. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: In the digital human image generation step, the input condition C of the digital human is defined, and the condition vector and random noise are input into the encoder to generate latent space representations μ and σ. The latent vector z is generated in the latent space and is expressed as: z~N(μ,σ 2 ),[μ,σ]=Encoder(C;θenc) The sampled latent vector z is passed through the decoder to generate a preliminary digital human image I initial , expressed as: I initial =Decoder(z;θdec) The reconstruction loss is used to ensure that the generated image meets the characteristics of the input conditions, expressed as: The initially generated image I initial Adjust to the target style S and get the adjusted image L style , and apply the content loss function L content Ensure that the generated image retains the content characteristics of the initial image, expressed as: Apply the style loss function L style , minimize the difference in style representation between the generated image and the reference style image, expressed as: Apply the content loss function L content Ensure that the generated image retains the content characteristics of the initial image, expressed as: Apply the style loss function L style , minimize the difference in style representation between the generated image and the reference style image, expressed as: Where ψ(x) represents the style features of the style image x, G represents the Gram matrix at the lth layer, and w l is the layer weight; Combining content loss and style loss, the visual effect of the generated image is adjusted by optimizing the total loss function to match the specific virtual environment, which is expressed as: L total =αL content +βL style , where α and β are the weights of content loss and style loss respectively; Output the final static image I static .

7. The method for generating an interactive digital human based on artificial intelligence according to claim 6, characterized in that: The digital human image generation step also includes the following steps: static and multimodal representation I behavior , generate dynamic animation, which includes action sequences and expression changes. The specific steps are: Enter action sequence condition C motion , generate action data A motion , expressed as: A motion =M(C motion ;θ motion ) The generated motion data is combined with the static image and performance data to generate animation frames, represented as: I animated =Animation(I static ,I behavior ,A motion )。 8. The method for generating an interactive digital human based on artificial intelligence according to claim 1, characterized in that: In the digital human response optimization step, the user's input data u(t) occurs at time t, where u(t) is a voice command or text information, expressed as u(t)=Input(t); the digital human's response r(t) is generated using the reaction function R, which is expressed as: r(t)=R(u(t), θ r ), where θ r are the parameters of the reaction function R, and the parameters of the reaction function R are optimized through training data.

9. The method for generating an interactive digital human based on artificial intelligence according to claim 8, characterized in that: It also includes a method for continuously optimizing the behavior and response strategies of digital humans using reinforcement learning, which includes the following steps: Define a set of possible actions based on the current system state and the context of user interaction; let S t A represents the system state at time t, including the current state of the digital human and the context of user interaction. t Indicates that in state S t The set of actions that can be taken under state S is learned through the reinforcement learning algorithm. Strategy π selects t The best action a t , expressed as: a t =π(S t ,θ π ), where θ π are the parameters of the strategy; Actiona t Causes the state to change to S t+1 , and generates an immediate reward r t+1 , expressed as: S t+1 ,r t+1 =Environment(S t ,a t ) Using the reinforcement learning algorithm, learn the strategy of selecting actions to maximize the expected value of the cumulative reward; the goal of reinforcement learning is to optimize the strategy π to maximize the expected value of the cumulative reward, expressed as: Where γ is the discount factor, which represents the current value of future rewards; According to the results of the action and the immediate rewards generated, the policy parameters are updated to improve future behavior decisions; the policy parameters θ π Updated by the gradient ascent method to maximize the long-term benefit, expressed as: Among them, α is the learning rate, is J versus θ π The gradient of is calculated by the policy gradient method; Generate comprehensive response function R ′ (t), expressed as: R ′ (t) = {r(t), behavior(a t )}.

10. An interactive digital human generation system based on artificial intelligence, based on the interactive digital human generation method based on artificial intelligence according to any one of claims 1 to 9, characterized in that: Includes the following modules: Digital human image generation module: used to define the input conditions of the digital human. The input conditions are represented by a condition vector, which describes the basic characteristics and generation requirements of the digital human. The conditional vector is input into the encoder to generate a latent space representation. The sampled latent vector is passed through the decoder to generate a preliminary digital human image. The reconstruction loss is used to ensure that the generated image meets the characteristics of the input condition. The KL divergence is used to ensure the continuity of the latent space distribution. Based on the preliminary generated digital human image, the style transfer technology is applied to adapt it to the target environment or style. The super-resolution network is used to improve the resolution and detail quality of the image, and the final image of the high-resolution image is output; Digital human behavior generation module: used to collect multimodal data and convert it into a corresponding feature vector set, wherein the multimodal data includes facial expressions, voices and body movements; construct a fusion function, wherein the fusion function is used to integrate multiple feature vectors in the feature vector set into a unified feature representation, wherein the fusion function is a multi-layer neural network, and the data of different modalities are mapped to a unified feature space through the weight matrix in the neural network; define a loss function, which is used to evaluate the difference between the generated digital human behavior and the preset target; adjust the loss function parameters through an optimization algorithm to minimize the loss function value, wherein the optimization algorithm includes using a stochastic gradient descent or an Adam algorithm; Digital human performance generation module: used to set a conditional vector to represent a specific emotion or state; through the generator, receive the unified feature representation of random noise, conditional vector and multimodal data, and generate facial expression or voice data corresponding to the set emotion or state; use the discriminator to evaluate the matching degree between the generated data and the conditional vector and judge its authenticity; optimize the generator and discriminator by minimizing the adversarial loss function to improve the authenticity and matching degree of the generated data, and construct a performance data synthesizer; Digital human emotion recognition module: used to obtain the user's facial image, extract features from the user's facial image using a pre-trained convolutional neural network model; input the extracted feature vector into a classifier to recognize the user's specific expression, the classifier being a support vector machine or a multi-layer perceptron; based on the recognized expression category, the digital human generates a corresponding facial expression or voice response; Digital human response optimization module: used to receive user input information, which includes voice instructions or text information; process user input and generate corresponding responses through preset reaction functions; adjust the parameters of the reaction function through feedback to ensure the accuracy and timeliness of the response.

Images