A digital human generation method and system based on AI interactive information

Abstract

The present application provides a method and system for generating a digital human based on AI interactive information. The present application performs real-time emotional feature extraction based on facial collection data, and generates a first dynamic facial image of the digital human virtual image based on the real-time emotional features; performs emotional trajectory prediction based on the user's historical emotional features to obtain an initial emotional prediction trajectory, extracts emotional influencing factors based on historical barrage interaction information and historical emotional features, collects a barrage distribution time window, and generates a second dynamic facial image of the digital human virtual image based on the barrage distribution time window, the emotional influencing factors, and the initial emotional prediction trajectory; performs a fusion decision based on the first dynamic facial image and the second dynamic facial image to generate a dynamic digital human image for live broadcast push, and can dynamically adjust the digital human's facial emotions based on audience feedback, thereby improving the authenticity and interactivity of the digital human's expression.

Description

Technical Field

[0001] The present application relates to the technical field of digital human image generation, and more specifically, to a digital human generation method and system based on AI interactive information. Background Art

[0002] In recent years, it has developed rapidly driven by technology, culture and entertainment industries, showing a development trend of diverse personality and vigorous innovation. Its biggest feature is that real-life anchors use virtual digital human images to conduct live broadcast activities, and the digital human virtual image is the exclusive visual image created by the anchor based on digital technology. With the rapid development of virtual human technology, digital humans generated based on artificial intelligence are also widely used in virtual live broadcast, digital marketing, remote interaction and other scenarios.

[0003] In existing technologies, digital human virtual images usually express emotions through pre-set language models, making it difficult to achieve dynamic facial expression drive that corresponds to the user's actual emotional state and the audience's emotional interaction. This results in the emotional expression of digital humans lacking authenticity and personalization, affecting the user's sense of immersion and the audience's interactive experience. How to improve the naturalness and interactivity of digital human virtual images has become an urgent problem to be solved. Summary of the Invention

[0004] The present application provides a method and system for generating a digital human based on AI interactive information, which can dynamically adjust the facial emotions of the digital human based on audience feedback, thereby improving the authenticity and interactivity of the digital human's expression.

[0005] In the first aspect, the present application provides a method for generating a digital human based on AI interactive information. The method can be executed by a network device, or by a chip configured in the network device, and the present application does not limit this.

[0006] Specifically, the method includes:

[0007] Acquiring multimodal interaction information of a user, and generating a multidimensional identification feature of the user based on the multimodal interaction information;

[0008] Based on the pre-trained AI language model and image generation model, a digital human virtual image is generated through the multi-dimensional recognition features;

[0009] Acquiring user's facial data in real time, extracting real-time emotional features based on the facial data, and generating a first dynamic facial image of the digital human avatar based on the real-time emotional features;

[0010] Predicting an emotion trajectory based on the user's historical emotion characteristics to obtain an initial emotion prediction trajectory, extracting emotion influencing factors based on historical barrage interaction information and historical emotion characteristics, collecting a barrage distribution time window, and generating a second facial dynamic image of the digital human avatar based on the barrage distribution time window, the emotion influencing factors, and the initial emotion prediction trajectory;

[0011] A fusion decision is made based on the first dynamic facial image and the second dynamic facial image to generate a dynamic image of a digital human and use it for live broadcast push.

[0012] In conjunction with the first aspect, in certain implementations of the first aspect, generating a multi-dimensional identification feature of a user based on the multimodal interaction information specifically includes:

[0013] For the interaction information of any group of modalities in the multimodal interaction information, extracting original feature digital representations through a feature extraction network matching the modality;

[0014] For the original digital feature representation corresponding to any set of modalities, multiple modal features are uniformly mapped to the specified dimensions in the feature space through feature transformation, and all modal features are aligned and fused in the feature space according to the preset dimensions to obtain the user's multidimensional feature vector as the multidimensional identification feature.

[0015] In conjunction with the first aspect, in certain implementations of the first aspect, predicting an emotion trajectory based on the user's historical emotion characteristics to obtain an initial emotion prediction trajectory specifically includes:

[0016] Real-time recording of users’ facial emotional features during live broadcasts to form a time series sample set of historical emotional features;

[0017] A moving average autoregressive model of the emotion features is constructed according to the time series sample set of the historical emotion features, and an initial emotion prediction trajectory is determined based on the moving average autoregressive model of the emotion features.

[0018] In combination with the first aspect, in some implementations of the first aspect, before extracting the emotional influencing factor based on historical barrage interaction information and historical emotional characteristics, the method also includes: storing the barrage information received during the user's live broadcast to obtain historical barrage interaction information.

[0019] In conjunction with the first aspect, in certain implementations of the first aspect, extracting the emotion influencing factor based on historical barrage interaction information and historical emotion features specifically includes:

[0020] Using a natural language processing model to perform sentiment classification on the historical barrage interaction information, statistically analyzing the distribution of barrage emotion types within a set time window, and generating a barrage emotion distribution sequence;

[0021] A time series of historical emotional features is obtained, and an emotional influencing factor is determined based on a sequence correlation between a barrage emotional distribution sequence and the time series of the historical emotional features.

[0022] In conjunction with the first aspect, in certain implementations of the first aspect, generating a second animated facial image of the digital human avatar based on the bullet comment distribution time window, the emotion influencing factor, and the initial emotion prediction trajectory specifically includes:

[0023] Obtain the initial emotion prediction distribution through the initial emotion prediction trajectory;

[0024] Extracting the emotional features of the barrage based on the barrage distribution time window, and constructing the emotional impact prediction distribution based on the emotional features of the barrage and the emotional impact factors;

[0025] Obtaining a fusion weight of the initial emotion prediction distribution and the emotion impact prediction distribution, and performing weighted fusion on the initial emotion prediction distribution and the emotion impact prediction distribution to determine a final emotion prediction result;

[0026] The emotion prediction result and the static digital human avatar are input into an expression driving model to generate a second dynamic facial image of the digital human avatar.

[0027] In conjunction with the first aspect, in certain implementations of the first aspect, performing a fusion decision based on the first animated facial image and the second animated facial image to generate a dynamic digital human image for live broadcast push specifically includes:

[0028] Acquiring emotional consistency between the first animated facial image and the second animated facial image;

[0029] When the emotion consistency is higher than a preset threshold, extracting motion parameters of the first animated facial image and the second animated facial image respectively, performing weighted fusion on the motion parameters using preset weights, and generating final facial motion data;

[0030] When the emotional consistency is lower than a preset threshold, a fuzzy decision algorithm is used to select a facial dynamic image, and the final facial action data is determined based on the decision result;

[0031] Based on the final facial movement data, a dynamic image of the digital human is generated and broadcast live.

[0032] In a second aspect, the present application provides a digital human generation system based on AI interactive information, which includes a digital human image generation unit, wherein the digital human image generation unit includes:

[0033] A user information extraction module is used to obtain multimodal interaction information of the user and generate a multidimensional identification feature of the user based on the multimodal interaction information;

[0034] A virtual image generation module, configured to generate a digital human virtual image based on the multi-dimensional recognition features based on a pre-trained AI language model and an image generation model;

[0035] A dynamic image generation module is used to obtain the user's facial data in real time, extract real-time emotional features based on the facial data, and generate a first dynamic facial image of the digital human virtual image based on the real-time emotional features;

[0036] The dynamic image generation module is further configured to predict an emotion trajectory based on the user's historical emotion characteristics to obtain an initial emotion prediction trajectory, extract an emotion influencing factor based on historical barrage interaction information and historical emotion characteristics, collect a barrage distribution time window, and generate a second dynamic facial image of the digital human avatar based on the barrage distribution time window, the emotion influencing factor, and the initial emotion prediction trajectory;

[0037] The dynamic image generation module is further used to make a fusion decision based on the first dynamic facial image and the second dynamic facial image to generate a dynamic image of a digital human for live broadcast push.

[0038] In a third aspect, the present application provides a computer terminal device, which includes a memory and a processor, wherein the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned method for generating a digital human based on AI interactive information.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the above-mentioned digital human generation method based on AI interactive information.

[0040] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0041] The present application provides a method and system for generating a digital human based on AI interactive information. First, the user's multimodal interactive information is obtained, and the user's multidimensional identification features are generated based on the multimodal interactive information; based on the pre-trained AI language model and image generation model, a digital human virtual image is generated through the multidimensional identification features; the user's facial collection data is obtained in real time, real-time emotional features are extracted based on the facial collection data, and a first facial dynamic image of the digital human virtual image is generated based on the real-time emotional features; the emotional trajectory is predicted based on the user's historical emotional features to obtain an initial emotional prediction trajectory, the emotional influencing factor is extracted based on the historical barrage interaction information and historical emotional features, the barrage distribution time window is collected, and the second facial dynamic image of the digital human virtual image is generated based on the barrage distribution time window, the emotional influencing factor and the initial emotional prediction trajectory; a fusion decision is made based on the first facial dynamic image and the second facial dynamic image to generate a digital human dynamic image for live broadcast push.

[0042] Therefore, it can be seen that the present application uses real-time facial acquisition data to extract the user's current emotional characteristics, and uses the expression-driven model to map the emotions to generate a first facial dynamic image, thereby realizing the synchronous expression of the user's current real emotions on the digital human, ensuring that the digital human's emotional output is derived from the real user state, close to the user's own instant expression, enhancing authenticity and emotional immersion, and then introducing the audience barrage interactive emotion analysis mechanism. Through barrage emotion classification and emotion distribution modeling within the time window, an influencing factor model of audience emotions is constructed, and then combined with the emotion trajectory prediction results, a second facial dynamic image is generated to reflect the regulatory effect of the audience's overall emotion on the digital human's emotional expression, so that it can perceive and respond to the audience's emotional feedback, thereby realizing dynamic expression with social interaction awareness. Finally, by making a fusion decision on the first facial dynamic image and the second facial dynamic image, the digital human's facial emotions are dynamically balanced between user expression and audience feedback, thereby avoiding conflict and improving naturalness and interaction consistency.

[0043] In summary, the present application can dynamically adjust the facial emotions of digital humans based on audience feedback, thereby improving the authenticity and interactivity of digital human expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is an exemplary flow chart of a method for generating a digital human based on AI interaction information according to some embodiments of the present application;

[0045] Figure 2 is a structural diagram of a digital human image generation unit according to some embodiments of the present application;

[0046] Figure 3This is a structural diagram of a computer terminal device that implements a method for generating a digital human based on AI interactive information according to some embodiments of the present application. DETAILED DESCRIPTION

[0047] This application obtains the user's multimodal interaction information and generates the user's multidimensional identification features based on the multimodal interaction information; generates a digital human virtual image through multidimensional identification features based on a pre-trained AI language model and image generation model; obtains the user's facial collection data in real time, extracts real-time emotional features based on the facial collection data, and generates a first facial dynamic image of the digital human virtual image based on the real-time emotional features; predicts the emotional trajectory based on the user's historical emotional features to obtain an initial emotional prediction trajectory, extracts emotional influencing factors based on historical barrage interaction information and historical emotional features, collects the barrage distribution time window, and generates a second facial dynamic image of the digital human virtual image based on the barrage distribution time window, emotional influencing factors and the initial emotional prediction trajectory; makes a fusion decision based on the first facial dynamic image and the second facial dynamic image, generates a digital human dynamic image and uses it for live broadcast push, and can dynamically adjust the digital human's facial emotions based on audience feedback, thereby improving the authenticity and interactivity of the digital human's expression.

[0048] In order to better understand the above technical solution, the following will be combined with the accompanying drawings and specific implementation methods to describe the above technical solution in detail. Figure 1 , which is an exemplary flow chart of a method for generating a digital human based on AI interaction information according to some embodiments of the present application. The method 100 for generating a digital human based on AI interaction information mainly includes the following steps:

[0049] In step S101 , multimodal interaction information of a user is obtained, and a multidimensional identification feature of the user is generated based on the multimodal interaction information.

[0050] Optionally, in some embodiments, the multimodal interaction information includes: voice modal interaction information, text modal interaction information, image modal interaction information and behavioral modal interaction information, wherein the voice modal interaction information can recognize the user's voice through a voice recognition model, and extract the Mel-frequency cepstral coefficients of the user's voice as the voice modal interaction information; the text modal interaction information can extract semantic vectors related to the user's voice text context, and statistically analyze the emotional characteristics of the language style as the text modal interaction information; the image modal interaction information includes the user's facial embedding vector, facial expression capture information and expression state distribution probability; the behavioral modality includes the user's operation frequency and interaction duration in the live broadcast interface, etc.

[0051] Preferably, in some embodiments, generating a user's multi-dimensional identification features based on the multimodal interaction information specifically includes:

[0052] For the interaction information of any group of modalities in the multimodal interaction information, extracting original feature digital representations through a feature extraction network matching the modality;

[0053] For the original digital feature representation corresponding to any set of modalities, multiple modal features are uniformly mapped to the specified dimensions in the feature space through feature transformation, and all modal features are aligned and fused in the feature space according to the preset dimensions to obtain the user's multidimensional feature vector as the multidimensional identification feature.

[0054] In specific implementation, the original feature digital representation is a digital vector or tensor that can be processed by a computer. It is the first step output result of feature extraction and is the underlying feature expression extracted from the original modal data. For example, the original feature digital representation of the speech modality can adopt a matrix with a dimension of 13 × the number of time frames. The original feature digital representations corresponding to other modalities are extracted using the corresponding feature extraction algorithm. This application does not go into too much detail on this. Then, a normalized linear transformation strategy can be used as the feature transformation method of the original feature digital representation to perform feature space mapping, and all modal features are aligned and fused in the feature space according to the preset dimensions to obtain the user's multidimensional feature vector as a multidimensional identification feature.

[0055] Optionally, in some embodiments, linear multidimensional space allocation is performed based on the information capacity of the interaction information of each modality in the multimodal interaction information. For example, dimensions 0 to 128 of the multidimensional identification feature can be used to store speech modal features, dimensions 129 to 384 of the multidimensional identification feature can be used to store speech modal features, dimensions 385 to 896 of the multidimensional identification feature can be used to store facial modal features, and dimensions 897 to 960 of the multidimensional identification feature can be used to store behavioral modal features.

[0056] In step S102, based on the pre-trained AI language model and image generation model, a digital human virtual image is generated through the multi-dimensional recognition features.

[0057] Preferably, in some embodiments, based on the pre-trained AI language model and image generation model, generating a digital human virtual image through the multi-dimensional recognition features specifically includes: using the user's multi-dimensional recognition features as conditional control vectors, inputting them into the pre-trained language expression model to generate text information containing the user's language habits, and synthesizing speech content with the user's timbre characteristics by the speech synthesis model; inputting the facial modal features in the multi-dimensional recognition features into the image generation model to generate a digital human virtual image, and driving lip movements based on voice information, and changing expressions based on real-time emotional features, to finally construct a dynamic digital human image for live interactive applications.

[0058] In specific implementation, the pre-trained AI language model and image generation model in this application can adopt existing technologies. For example, in one of the embodiments of this application, the pre-trained AI language model can adopt a combination of a text generation model and a speech synthesis (TTS) model, and the image generation model can combine the text-driven image algorithm of Stable Diffusion with the structural control algorithm of ControlNet, thereby integrating text descriptions and facial features to generate a digital human virtual image. Algorithms or scripts in other existing technologies that can achieve speech audio synthesis and virtual image generation can also be applied, and this application does not limit this.

[0059] In step S103, the user's facial data is acquired in real time, emotional features are extracted in real time based on the facial data, and a first dynamic facial image of the digital human avatar is generated based on the real-time emotional features.

[0060] Preferably, in some embodiments, in the process of acquiring the user's facial collection data in real time, the user's facial image data is collected in real time through the user's live broadcast camera, and the facial key point coordinate information is extracted as the user's facial collection data through the facial key point detection model.

[0061] Specifically, the facial key point detection model can adopt the currently publicly available highly robust facial expression analysis tool OpenFace, which has the ability to locate key points in the facial area in RGB images, estimate head posture, track eye movements, and analyze facial action unit (AU) intensity; the OpenFace algorithm can accurately extract 68 three-dimensional facial key points in the user's facial image, including key areas such as the mouth, eyebrows, eyes, nose bridge and facial contours.

[0062] Furthermore, based on the extracted coordinates of facial key points, the system can construct the user's current facial feature vector and input it into the emotion recognition module to extract the real-time emotional state; for example, by extracting the intensity scores of facial action units in OpenFace, such as AU06 (eyebrow raising) and AU12 (smile), combined with rule models or deep learning emotion classification models, facial expressions can be mapped into multi-dimensional emotion vectors, such as happiness, surprise, boredom, etc.; the emotion vector can be used as input information to drive the changes in the digital human's virtual image, realizing the dynamic facial expression of the digital human corresponding to the user's current facial state.

[0063] In addition, to ensure the stability and real-time performance of the acquisition process, the system can set face detection thresholds and tracking cache mechanisms to adapt to face recognition effects under different lighting, angles and backgrounds, ensuring the continuity and robustness of the extracted facial acquisition data; the acquisition and processing process can support a processing rate of no less than 25 frames per second, meeting the real-time and naturalness requirements of digital human live broadcast applications.

[0064] Preferably, in some embodiments, a long short-term memory artificial neural network may be used as an emotion recognition model to perform real-time emotion feature extraction based on the facial collection data.

[0065] It should be noted that the emotion recognition model can adopt a long short-term memory artificial neural network (LSTM), which is suitable for processing facial dynamic data displayed by users in continuous time series, so as to achieve more accurate real-time emotion feature extraction. Specifically, the long short-term memory artificial neural network is used to model and classify the time series features in the user's facial collection data; the facial collection data may include multi-frame continuous facial key point coordinates, action unit intensity, head posture and other information extracted by OpenFace or similar facial key point detection algorithms, and the multi-dimensional time series data can be used as input of the long short-term memory artificial neural network.

[0066] During the training phase, the system constructs a set of facial expression datasets as training samples. Each sample contains a sequence of facial features of multiple consecutive frames and their corresponding emotional labels, such as happiness, surprise, anger, sadness, etc. The long short-term memory artificial neural network retains long-term dependency information through its unique gating mechanism (input gate, forget gate, output gate), effectively capturing subtle changes in the user's facial expressions over time, and improving the ability to recognize complex emotional states such as shyness, boredom, and confusion.

[0067] In actual application, when the system receives real-time facial data streams from the camera, it first converts them into a standardized time series tensor, such as a shape of [time length × feature dimension], and inputs them into a trained long-short-term memory artificial neural network. The model output is an emotion feature vector or probability distribution vector, representing the emotional state and corresponding confidence level at the current time point. For example, the model outputs the following vector:

[0068] [Happy: 0.72, Surprised: 0.10, Sad: 0.08, Angry: 0.05, Calm: 0.05].

[0069] The system can select the current dominant emotion based on the vector, and further use it to drive the dynamic facial expression of the digital human virtual image. Furthermore, in order to enhance the model's ability to capture the temporal sequence of complex expressions, the long short-term memory artificial neural network structure can be used in conjunction with the convolutional neural network (CNN) to form a CNN-LSTM emotion recognition architecture, in which the convolutional neural network is used to extract spatial features in facial images or AU matrices, and the long short-term memory artificial neural network is responsible for processing its temporal evolution process; this combined structure has been proven in existing public research to have high recognition accuracy for micro-expressions and atypical emotional states, and is suitable for real-time digital human emotion recognition tasks in live interactive scenarios.

[0070] Preferably, in some embodiments, generating a first facial dynamic image of a digital human virtual image based on real-time emotional features specifically includes: inputting the real-time emotional features and the static digital human virtual image into an expression driving model, and generating a first facial dynamic image through the expression driving model.

[0071] In specific implementation, the user's emotional feature vector extracted in real time and the preset static digital human virtual image are input into the expression driving model, and the first facial dynamic image matching the user's current emotional state is generated through the model, thereby realizing the dynamic interpretation of the digital human's facial expression in synchronization with the user's emotions. For example, the expression driving model can adopt a technical structure based on generative adversarial networks, expression transfer networks, or three-dimensional expression reconstruction models, preferably using mature deep generative models such as Audio2Face, which have the ability to map expression parameters or emotion vectors into dynamic facial images.

[0072] The input of the model includes: a static digital human virtual image: that is, a basic head image or a three-dimensional head model of the target digital human, which is used as an appearance reference for the generation process; a real-time emotional feature vector: the current user's emotional state extracted by the aforementioned long-short-term memory artificial neural network model, such as the confidence distribution vector or low-dimensional embedding feature of seven types of emotional labels, which represents the category and intensity of facial expression. In order to achieve a continuous and natural transition of expressions, the system can first post-process the input emotional feature vector, such as normalization and smoothing filtering to eliminate mutations; feature mapping, mapping the high-dimensional emotional vector to the facial expression action space, such as the AU encoding space; and temporal modeling, through inter-frame interpolation. Smooth transition changes; in the expression drive stage, the system generates dynamic images in the following ways: the emotional characteristics are input as control signals to the action control branch of the expression drive model; the facial structure, material and texture characteristics of the digital human static image are combined inside the model to simulate and reconstruct the expression changes; finally, a sequence of facial images of continuous frames is output to form a dynamic digital human expression that reflects the current user's emotions, such as smiling, frowning, surprise, etc.; preferably, if the static digital human virtual image is a three-dimensional model, the emotional characteristics can also be mapped to facial blendshape weights or three-dimensional point transformation parameters to drive the deformation of the virtual facial mesh to achieve structural-level dynamic expression synthesis.

[0073] In some specific embodiments of the present application, in order to meet the real-time requirements of live broadcast push, the generation rate of the first facial dynamic image may be no less than 20 frames per second, and the output image size may be configured to 128×128, 256×256 or higher resolution according to application requirements, and encapsulated into the live broadcast frame stream through compression algorithms such as H.264 / H.265 to achieve efficient transmission.

[0074] In the present application, real-time facial data such as facial key points and expression action units are collected, and the emotion recognition model is used to extract the user's current emotional characteristics. The emotion is then mapped with the help of the expression-driven model to generate a first dynamic facial image, thereby realizing the synchronous expression of the user's real emotions at this moment on the digital human.

[0075] In step S104, an emotional trajectory is predicted based on the user's historical emotional characteristics to obtain an initial emotional prediction trajectory, an emotional influencing factor is extracted based on historical barrage interaction information and historical emotional characteristics, a barrage distribution time window is collected, and a second facial dynamic image of the digital human virtual image is generated based on the barrage distribution time window, the emotional influencing factor and the initial emotional prediction trajectory.

[0076] Preferably, in some embodiments, performing emotion trajectory prediction based on the user's historical emotion characteristics to obtain the initial emotion prediction trajectory specifically includes:

[0077] Real-time recording of users’ facial emotional features during live broadcasts to form a time series sample set of historical emotional features;

[0078] A moving average autoregressive model of the emotion features is constructed according to the time series sample set of the historical emotion features, and an initial emotion prediction trajectory is determined based on the moving average autoregressive model of the emotion features.

[0079] Preferably, in some embodiments, the time series of historical emotional features is digitally mapped by recording historical real-time emotional features and corresponding time tags, and then obtaining the dominant emotional type of the real-time emotional features to obtain emotional feature values, and the time series of historical emotional features is composed based on the emotional feature values ​​and corresponding time tags, wherein the initial emotional prediction trajectory is a sequence of emotional feature values ​​for a future period of time determined based on a moving average autoregressive model.

[0080] Optionally, in some embodiments, before extracting the emotion influencing factor based on the historical barrage interaction information and the historical emotion characteristics, the method further includes: storing the barrage information received during the user's live broadcast to obtain the historical barrage interaction information.

[0081] It should be noted that the emotion impact factor described in this application represents the intensity of the effect of external barrage emotions on the user's internal emotional changes. Preferably, in some embodiments, the emotion impact factor is extracted based on historical barrage interaction information and historical emotional features, specifically including:

[0082] Using a natural language processing model to perform sentiment classification on the historical barrage interaction information, statistically analyzing the distribution of barrage emotion types within a set time window, and generating a barrage emotion distribution sequence;

[0083] A time series of historical emotional features is obtained, and an emotional influencing factor is determined based on a sequence correlation between a barrage emotional distribution sequence and the time series of the historical emotional features.

[0084] In specific implementation, the BERT-wwm-ext algorithm can be used as a natural language processing model to perform emotion classification on the historical barrage interaction information, and generate a barrage emotion distribution value based on the distribution of barrage emotion types, wherein the barrage information is divided into time periods according to a set time granularity, such as per second, and mapped based on the threshold range of the barrage emotion distribution in different time periods to obtain the barrage emotion distribution value. Optionally, in some embodiments, the barrage emotion classification types can be calibrated into four emotion distribution types: positive, negative, encouraging, and sarcastic, and then the distribution proportions of different types of emotions are weighted and fused according to the preset emotion weights of different types of emotions to obtain the final barrage emotion distribution value and the corresponding time label, and the barrage emotion distribution sequence is composed according to the time sequence. For example, in some specific embodiments of the present application, the weights of the four emotion distribution types of positive, negative, encouraging, and sarcastic are +1.0, -0.8, +0.8, and -0.6, respectively.

[0085] Preferably, in some embodiments, in the process of determining the emotion influencing factor based on the sequence correlation between the barrage emotion distribution sequence and the time series of the historical emotion features, the differential sequence of the barrage emotion distribution sequence is obtained based on the time label, and after the differential sequence of the barrage emotion distribution sequence and the time series of the historical emotion features are aligned and normalized, the Pearson correlation coefficient between the differential sequence of the barrage emotion distribution sequence and the time series of the historical emotion features is used as the emotion influencing factor.

[0086] In some specific embodiments of the present application, the initial emotion prediction distribution and the emotion impact prediction distribution are calculated based on information from different sources, and the final emotion prediction result is obtained by fusion. Specifically, the initial emotion prediction distribution is generated based on the user's facial emotion characteristics at the target moment and the prediction results of the emotion prediction model, while the emotion impact prediction distribution is calculated within the barrage distribution time window, taking into account the potential impact of the audience's emotions on the digital human virtual image, combining the barrage emotion characteristics and emotion influence factors.

[0087] It should be noted that the barrage distribution time window described in this application refers to a fixed-length time interval containing the statistical distribution of barrage types, which is set for time segment statistics of barrage data when processing barrage information sent by viewers in a live broadcast or video, in order to perform time series analysis and feature extraction. Optionally, in some embodiments, generating a second facial dynamic image of the digital human avatar based on the barrage distribution time window, the emotion influencing factor, and the initial emotion prediction trajectory specifically includes:

[0088] Obtain the initial emotion prediction distribution through the initial emotion prediction trajectory;

[0089] Extracting the emotional features of the barrage based on the barrage distribution time window, and constructing the emotional impact prediction distribution based on the emotional features of the barrage and the emotional impact factors;

[0090] Obtaining a fusion weight of the initial emotion prediction distribution and the emotion impact prediction distribution, and performing weighted fusion on the initial emotion prediction distribution and the emotion impact prediction distribution to determine a final emotion prediction result;

[0091] The emotion prediction result and the static digital human avatar are input into an expression driving model to generate a second dynamic facial image of the digital human avatar.

[0092] It should be noted that the initial emotion prediction distribution described in this application is the predicted probability distribution of the user's facial emotions at the target moment, which reflects the emotional tendency of the user at the target moment. The initial emotion prediction trajectory contains the emotional feature values ​​in the future period of time. The emotional feature values ​​corresponding to the target moment are reverse mapped to obtain the corresponding initial emotion prediction distribution. The emotional distribution vector is composed of the emotional type distribution ratio in the barrage distribution time window closest to the target moment, and the emotional distribution vector is corrected using the emotional influence factor as the proportional coefficient to obtain the final mapped emotional prediction distribution.

[0093] In step S105, a fusion decision is made based on the first dynamic facial image and the second dynamic facial image to generate a dynamic image of a digital human for live broadcast push.

[0094] It should be noted that the first facial dynamic image is a dynamic facial expression image generated based on real-time emotional features, reflecting the facial expression of the user in the current emotional state. The second facial dynamic image is an image generated based on historical emotional features, barrage emotional features and emotional influence factors, taking into account the audience's emotional feedback and the user's emotional trajectory.

[0095] Optionally, in some embodiments, performing a fusion decision based on the first animated facial image and the second animated facial image to generate a dynamic digital human image for live broadcast push specifically includes:

[0096] Acquiring emotional consistency between the first animated facial image and the second animated facial image;

[0097] When the emotion consistency is higher than a preset threshold, extracting motion parameters of the first animated facial image and the second animated facial image respectively, performing weighted fusion on the motion parameters using preset weights, and generating final facial motion data;

[0098] When the emotional consistency is lower than a preset threshold, a fuzzy decision algorithm is used to select a facial dynamic image, and the final facial action data is determined based on the decision result;

[0099] Based on the final facial movement data, a dynamic image of the digital human is generated and broadcast live.

[0100] Preferably, in some embodiments, a CNN+LSTM neural network model can be used to perform emotion classification on the first animated facial image and the second animated facial image to obtain corresponding emotion labels (such as happy, surprised, angry, etc.) and confidence distribution, and then a cosine similarity model can be used to judge the emotion consistency of the emotion labels corresponding to the first animated facial image and the second animated facial image. For example, the inverse of the cosine similarity between the confidence distributions of the emotion labels is used as the emotion consistency.

[0101] Optionally, in some embodiments, when the emotional consistency is higher than a preset threshold, the OpenFace facial motion capture model can be used to extract motion parameters of the first facial dynamic image and the second facial dynamic image, the real-time emotional weight of the first facial dynamic image is set to 0.6, and the emotional impact weight of the second facial dynamic image is set to 0.4, the motion parameters of the first facial dynamic image and the second facial dynamic image are weightedly fused, and the fused motion parameters are input into the FLAME expression-driven model or other existing expression-driven models to generate the final dynamic image frame.

[0102] The following is a specific embodiment of the present application using a fuzzy decision algorithm to select dynamic facial images: a three-input and one-output fuzzy logic reasoning system is built, specifically including:

[0103] Input variables:

[0104] Confidence 1: The emotion recognition confidence of the first animated facial image, which indicates the degree of emotional credibility obtained based on the analysis of real-time user facial data. This application presets the emotion recognition confidence of the first animated facial image to be 0.7. Confidence 2: The emotion recognition confidence of the second animated facial image, which indicates the degree of emotional credibility of the image generated based on the barrage deduction, and is determined based on the cosine similarity between the motion parameters of the second animated facial image and the user's facial motion parameters collected in real time. Audience emotion weight: used to measure the emotional impact intensity of audience interactive content, which can be quantified by the numerical value of the emotion impact factor.

[0105] Output variables:

[0106] Image priority selection value A: The value range is [0, 1] and represents the weighted preference for the final image selection. A value close to 0 indicates a preference for the first animated face image, while A value close to 1 indicates a preference for the second animated face image. Intermediate values ​​indicate image fusion.

[0107] In the process of setting fuzzy rules, the system makes inferences and judgments based on the following typical fuzzy rules:

[0108] If the confidence level 1 is high and the audience emotion weight is low, the image priority selection value A is biased towards 0 (selecting the first image); if the confidence level 2 is high and the audience emotion weight is high, the image priority selection value A is biased towards 1 (selecting the second image); if the difference between the confidence levels 1 and 2 is not large, or the audience emotion weight is medium, the image priority selection value A takes an intermediate value (performing image fusion); in some other embodiments of the present application, high, medium, and low membership functions can be further defined, and the fuzzy reasoning method can be used to complete the inference, which is not limited in the present application.

[0109] In addition, in another aspect of the present application, in some embodiments, the present application provides a digital human generation system based on AI interactive information, the system includes a digital human image generation unit, reference Figure 2 This figure is a schematic diagram of the exemplary hardware and / or software structure of a digital human image generation unit according to some embodiments of the present application. The digital human image generation unit 200 includes: a user information extraction module 201, a virtual image generation module 202, and a dynamic image generation module 203, which are described as follows:

[0110] A user information extraction module 201 is configured to obtain multimodal interaction information of a user and generate a multidimensional identification feature of the user based on the multimodal interaction information;

[0111] A virtual image generation module 202 is configured to generate a digital human virtual image based on the multi-dimensional recognition features based on the pre-trained AI language model and image generation model;

[0112] The dynamic image generation module 203 is used to obtain the user's facial data in real time, extract the real-time emotional features based on the facial data, and generate a first dynamic facial image of the digital human avatar based on the real-time emotional features;

[0113] The dynamic image generation module 203 is further configured to predict an emotion trajectory based on the user's historical emotion characteristics to obtain an initial emotion prediction trajectory, extract an emotion influencing factor based on historical barrage interaction information and historical emotion characteristics, collect a barrage distribution time window, and generate a second dynamic facial image of the digital human avatar based on the barrage distribution time window, the emotion influencing factor, and the initial emotion prediction trajectory;

[0114] The dynamic image generation module 203 is further used to make a fusion decision based on the first dynamic facial image and the second dynamic facial image to generate a dynamic image of a digital human for live broadcast push.

[0115] The above details an example of a method and system for generating a digital human based on AI interactive information provided in an embodiment of the present application. It can be understood that, in order to realize the above functions, the corresponding device includes a hardware structure and / or software module corresponding to executing each function.

[0116] Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function in the application is executed in hardware or in a computer software-driven hardware manner depends on the specific application and design constraints of the technical solution. Therefore, professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0117] In addition, the present application also provides a computer terminal device, which includes a memory and a processor, the memory stores code, and the processor is configured to obtain the code and execute the above-mentioned digital human generation method based on AI interactive information.

[0118] In some embodiments, reference Figure 3 , which is a schematic diagram of the structure of a computer terminal device that implements a method for generating a digital human based on AI interaction information according to some embodiments of the present application. In the above embodiment, a method for generating a digital human based on AI interaction information can be achieved by Figure 3 The computer terminal device 300 shown in FIG. 1 is implemented as shown in FIG. 1 , and the computer terminal device 300 includes at least one communication bus 301 , a communication interface 302 , a processor 303 and a memory 304 .

[0119] The processor 303 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices for controlling the execution of a digital human generation method based on AI interactive information in this application.

[0120] The communication bus 301 may include a path for transmitting information between the aforementioned components.

[0121] Memory 304 may be, but is not limited to, a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, an optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer. Memory 304 may be independent and connected to processor 303 via communication bus 301. Memory 304 may also be integrated with processor 303.

[0122] Memory 304 is used to store program code for executing the solution of the present application, and is controlled by processor 303 for execution. Processor 303 is configured to execute the program code stored in memory 304. The program code may include one or more software modules. In the above embodiment, the generation of the first animated facial image can be implemented by processor 303 and one or more software modules in the program code stored in memory 304.

[0123] The communication interface 302 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), etc.

[0124] Optionally, the computer terminal device 300 may further include a power supply 305 for providing power to various devices or circuits in the real-time computer terminal device.

[0125] In a specific implementation, as an example, a computer terminal device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor herein may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0126] The aforementioned computer terminal device can be a general-purpose computer terminal device or a dedicated computer terminal device. In a specific implementation, the computer terminal device can be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of this application do not limit the type of computer terminal device.

[0127] In addition, in other aspects of the present application, a computer-readable storage medium is provided, which stores at least one computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the above-mentioned digital human generation method based on AI interactive information.

[0128] In summary, the embodiments of the present application disclose a method and system for generating a digital human based on AI interactive information. First, the multimodal interactive information of the user is obtained, and the multidimensional identification features of the user are generated based on the multimodal interactive information; based on the pre-trained AI language model and image generation model, a digital human virtual image is generated through the multidimensional identification features; the user's facial collection data is obtained in real time, real-time emotional features are extracted based on the facial collection data, and a first facial dynamic image of the digital human virtual image is generated based on the real-time emotional features; the emotional trajectory is predicted based on the user's historical emotional features to obtain an initial emotional prediction trajectory, the emotional influencing factor is extracted based on the historical barrage interaction information and historical emotional features, the barrage distribution time window is collected, and the second facial dynamic image of the digital human virtual image is generated based on the barrage distribution time window, the emotional influencing factor and the initial emotional prediction trajectory; a fusion decision is made based on the first facial dynamic image and the second facial dynamic image to generate a digital human dynamic image and use it for live broadcast push, which can dynamically adjust the digital human's facial emotions based on audience feedback, thereby improving the authenticity and interactivity of the digital human's expression.

[0129] The above description is merely an embodiment of the present application. Common knowledge such as the specific technical solutions or features of the solutions is not described in detail herein. It should be noted that those skilled in the art may make various modifications and improvements without departing from the technical solution of the present application, and these modifications and improvements should be considered within the scope of protection of the present application and will not affect the effectiveness of the implementation of the present application or the practical application of the patent.

[0130] The scope of protection claimed by this application shall be determined by the content of the claims. The specific embodiments and other descriptions in the specification may be used to interpret the content of the claims. Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of the invention. Thus, if the modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include such modifications and variations.

Claims

1. A method for generating digital humans based on AI interactive information, characterized in that: include: Acquiring multimodal interaction information of a user, and generating a multidimensional identification feature of the user based on the multimodal interaction information; Based on the pre-trained AI language model and image generation model, a digital human virtual image is generated through the multi-dimensional recognition features; Acquiring user's facial data in real time, extracting real-time emotional features based on the facial data, and generating a first dynamic facial image of the digital human avatar based on the real-time emotional features; Predicting an emotion trajectory based on the user's historical emotion characteristics to obtain an initial emotion prediction trajectory, extracting emotion influencing factors based on historical barrage interaction information and historical emotion characteristics, collecting a barrage distribution time window, and generating a second facial dynamic image of the digital human avatar based on the barrage distribution time window, the emotion influencing factors, and the initial emotion prediction trajectory; Performing a fusion decision based on the first dynamic facial image and the second dynamic facial image to generate a dynamic image of the digital human for live broadcast push; The emotion impact factor represents the intensity of the effect of external barrage emotions on the user's internal emotion changes. The emotion impact factor is extracted based on historical barrage interaction information and historical emotion characteristics. Specifically, Using a natural language processing model to perform sentiment classification on the historical barrage interaction information, statistically analyzing the distribution of barrage emotion types within a set time window, and generating a barrage emotion distribution sequence; A time series of historical emotional features is obtained, and an emotional influencing factor is determined based on a sequence correlation between a barrage emotional distribution sequence and the time series of the historical emotional features.

2. The method according to claim 1, wherein Generating a user's multi-dimensional identification features based on the multimodal interaction information specifically includes: For the interaction information of any group of modalities in the multimodal interaction information, extracting original feature digital representations through a feature extraction network matching the modality; For the original digital feature representation corresponding to any set of modalities, multiple modal features are uniformly mapped to the specified dimensions in the feature space through feature transformation, and all modal features are aligned and fused in the feature space according to the preset dimensions to obtain the user's multidimensional feature vector as the multidimensional identification feature.

3. The method according to claim 1, wherein Emotional trajectory prediction is performed based on the user's historical emotional characteristics. The initial emotional prediction trajectory includes: Real-time recording of users’ facial emotional features during live broadcasts to form a time series sample set of historical emotional features; A moving average autoregressive model of the emotion features is constructed according to the time series sample set of the historical emotion features, and an initial emotion prediction trajectory is determined based on the moving average autoregressive model of the emotion features.

4. The method according to claim 1, wherein Before extracting the emotion influencing factors based on the historical barrage interaction information and the historical emotion features, the method also includes: storing the barrage information received during the user's live broadcast to obtain the historical barrage interaction information.

5. The method according to claim 1, wherein Generating a second facial dynamic image of the digital human avatar based on the bullet comment distribution time window, the emotion influencing factor, and the initial emotion prediction trajectory specifically includes: Obtain the initial emotion prediction distribution through the initial emotion prediction trajectory; Extracting the emotional features of the barrage based on the barrage distribution time window, and constructing the emotional impact prediction distribution based on the emotional features of the barrage and the emotional impact factors; Obtaining a fusion weight of the initial emotion prediction distribution and the emotion impact prediction distribution, and performing weighted fusion on the initial emotion prediction distribution and the emotion impact prediction distribution to determine a final emotion prediction result; The emotion prediction result and the static digital human avatar are input into an expression driving model to generate a second dynamic facial image of the digital human avatar.

6. The method according to claim 1, wherein The steps of making a fusion decision based on the first dynamic facial image and the second dynamic facial image to generate a dynamic digital human image for live broadcasting specifically include: Acquiring emotional consistency between the first animated facial image and the second animated facial image; When the emotion consistency is higher than a preset threshold, extracting motion parameters of the first animated facial image and the second animated facial image respectively, performing weighted fusion on the motion parameters using preset weights, and generating final facial motion data; When the emotional consistency is lower than a preset threshold, a fuzzy decision algorithm is used to select a facial dynamic image, and the final facial action data is determined based on the decision result; Based on the final facial movement data, a dynamic image of the digital human is generated and broadcast live.

7. A digital human generation system based on AI interactive information, comprising a digital human image generation unit, wherein the digital human image generation unit is configured to execute the digital human generation method based on AI interactive information according to any one of claims 1 to 6, characterized in that: The digital human image generation unit includes: A user information extraction module is used to obtain multimodal interaction information of the user and generate a multidimensional identification feature of the user based on the multimodal interaction information; A virtual image generation module, configured to generate a digital human virtual image based on the multi-dimensional recognition features based on a pre-trained AI language model and an image generation model; A dynamic image generation module is used to obtain the user's facial data in real time, extract real-time emotional features based on the facial data, and generate a first dynamic facial image of the digital human avatar based on the real-time emotional features; The dynamic image generation module is further configured to predict an emotion trajectory based on the user's historical emotion characteristics to obtain an initial emotion prediction trajectory, extract an emotion influencing factor based on historical barrage interaction information and historical emotion characteristics, collect a barrage distribution time window, and generate a second dynamic facial image of the digital human avatar based on the barrage distribution time window, the emotion influencing factor, and the initial emotion prediction trajectory; The dynamic image generation module is further used to make a fusion decision based on the first dynamic facial image and the second dynamic facial image to generate a dynamic image of a digital human for live broadcast push.

8. A computer terminal device, characterized in that: The computer terminal device includes a memory and a processor, the memory stores code, and the processor is configured to obtain the code and execute the digital human generation method based on AI interactive information as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing at least one computer program, characterized in that: The computer program is loaded and executed by a processor to implement the operations performed by the method for generating a digital human based on AI interactive information as described in any one of claims 1 to 6.

Images