Digital human emotion modeling method based on deep learning

Abstract

This invention relates to the field of digital human modeling technology and discloses a deep learning-based method for digital human emotion modeling. The method includes acquiring multimodal stimulus data and a current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration, and emotion type; performing emotion judgment based on the multimodal stimulus data to obtain a positive or negative emotion type; performing emotion initialization based on the positive or negative emotion type and the current emotional state to obtain an initial emotion value; performing emotion accumulation simulation based on the stimulus frequency to obtain a dynamic emotion value; performing emotion update based on the dynamic emotion value and the initial emotion value to obtain an initial emotion sequence; and inputting the initial emotion sequence into a pre-trained emotion optimization model to output the final optimal emotion sequence. This method can improve the accuracy of digital human emotion modeling.

Description

Technical Field

[0001] This invention relates to the field of digital human modeling technology, and in particular to a method for modeling digital human emotions based on deep learning. Background Technology

[0002] With the development of artificial intelligence technology, digital humans, as a new type of human-computer interaction interface, are gradually becoming a research hotspot. Digital humans can not only mimic human appearance and behavior, but also simulate human emotional responses through emotion modeling, thereby enhancing the interactive experience with users. Especially in fields such as virtual assistants, customer service, and education and entertainment, digital humans with emotion recognition and expression capabilities can provide a more natural and fluid communication experience. However, to enable digital humans to accurately understand and respond to users' emotional states, an efficient and accurate emotion modeling method is needed.

[0003] In existing technologies, deep learning-based methods for modeling emotions in digital humans involve two main steps: First, a deep neural network model is trained using a large amount of labeled data to understand multiple modalities such as speech, facial expressions, and text. During emotion recognition, structures such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) are used to analyze the user's speech features or facial expression changes to determine their current emotional state. Second, based on the identified emotional state, generative adversarial networks (GANs) or other types of generative models are used to synthesize corresponding facial expressions or language responses, enabling the digital human to adjust its response according to the user's emotions. This method can effectively improve the realism of digital human interactions and the user experience.

[0004] Despite the significant progress made by the aforementioned methods, several challenges and limitations remain. The intensity of emotions in digital humans changes over time. Existing methods cannot model these time-varying emotional shifts, resulting in low accuracy in digital human emotion modeling. Summary of the Invention

[0005] This invention provides a method, system, electronic device, and storage medium for digital human emotion modeling based on deep learning, in order to improve the accuracy of digital human emotion modeling.

[0006] Firstly, in order to solve the above-mentioned technical problems, the present invention provides a deep learning-based digital human emotion modeling method, comprising:

[0007] Acquire multimodal stimulus data and current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration, and emotional type;

[0008] Emotional judgment is performed based on the multimodal stimulus data to obtain the positive or negative emotion type;

[0009] Emotion initialization is performed based on the positive / negative emotion type and the current emotion state to obtain the initial emotion value;

[0010] Based on the stimulation frequency, an emotional accumulation simulation is performed to obtain the dynamic value of the emotion.

[0011] The initial emotion sequence is obtained by updating the emotion based on the dynamic emotion value and the initial emotion value.

[0012] The initial sentiment sequence is input into a pre-trained sentiment optimization model, which outputs the final optimal sentiment sequence.

[0013] In one optional implementation, the step of performing emotion judgment based on the multimodal stimulus data to obtain the positive or negative emotion type includes:

[0014] Numerical encoding is performed on the multimodal stimulus data to obtain stimulus feature vectors;

[0015] The stimulus feature vector is input into a pre-trained sentiment classification model, which outputs positive and negative probabilities.

[0016] When the positive probability is greater than the negative probability, the current stimulus's emotional type is determined to be positive.

[0017] When the positive probability is less than the negative probability, the current stimulus is determined to be a negative emotion.

[0018] The training process of the sentiment classification model includes:

[0019] The trained model is obtained after the loss function of the model meets the conditions, based on training on a large-scale labeled sentiment corpus.

[0020] In one optional implementation, the step of initializing the emotion based on the positive / negative emotion type and the current emotion state to obtain an initial emotion value includes:

[0021] The weight matrix is ​​calculated using the following formula:

[0022] W T =α·W base +(1-α)·S current

[0023] Among them, W T Let W represent the weight matrix. base S represents the pre-training matrix. current This represents the current emotional state, and α represents the decay coefficient.

[0024] An input vector is constructed based on the weight matrix and the positive / negative sentiment type;

[0025] The input vector is fed into a pre-trained sentiment impact value model, which outputs an initial sentiment value.

[0026] The training process of the emotional impact value model includes:

[0027] An emotional impact value model is constructed based on historical emotional vectors. The model is trained, and the training is considered complete when the loss function of the model meets the conditions, thus obtaining the trained emotional impact value model.

[0028] In one optional implementation, the step of simulating emotional accumulation based on the stimulus frequency to obtain an emotional dynamic value includes:

[0029] The emotional dynamic value is calculated using the following formula:

[0030]

[0031] Among them, I dynamic The emotional dynamic value is represented by β, the cumulative coefficient by γ, the cumulative effect index by N, the total number of frequency samples by k, and f. k This indicates the frequency of stimulus number k.

[0032] In one optional implementation, the step of updating the emotion based on the dynamic emotion value and the initial emotion value to obtain an initial emotion sequence includes:

[0033] Emotional intensity is calculated using the following formula:

[0034] I t =I init ·e -λt +I dynamic

[0035] Among them, I t I represents the emotional intensity at time point t. init Let e ​​represent the initial value of emotion, λ represent the natural constant, t represent the natural decay coefficient, and I represent the time point number. dynamic Indicates the emotional dynamic value;

[0036] The initial emotional sequence is obtained by sorting the emotional intensity from smallest to largest according to the time point.

[0037] In one optional implementation, the step of inputting the initial emotion sequence into a pre-trained emotion optimization model and outputting the final optimal emotion sequence includes:

[0038] When the duration of the stimulus exceeds a preset time threshold, lag accumulation is performed according to the emotion sequence function to obtain an initial lag accumulation value;

[0039] The updated sentiment sequence is obtained by weighting the sentiment sequence function and the initial lag cumulative value;

[0040] Based on the updated sentiment sequence, a dual-threshold detection is performed to obtain a discrete sentiment sequence;

[0041] The optimal emotional sequence is obtained by performing gradient optimization based on the discrete emotional sequence.

[0042] In one optional implementation, the step of performing gradient optimization based on the discrete emotion sequence to obtain the optimal emotion sequence includes:

[0043] The gradient of the loss function is calculated using the following formula:

[0044]

[0045] in, Let θ represent the gradient of the loss function, θ represent the hyperparameters of the sentiment optimization model, m represent the number of samples, i represent the sample ID, and h represent the gradient of the loss function. θ (x (i) ) represents the predicted value of the sentiment optimization model for the i-th sample, y (i) Let x represent the value in the discrete sentiment sequence of the i-th sample. (i) This represents the feature vector of the i-th sample;

[0046] The hyperparameters of the sentiment optimization model are updated based on the gradient of the loss function and the learning rate.

[0047] When the loss function of the sentiment optimization model is detected to be less than the preset detection threshold, the optimal sentiment sequence is output.

[0048] Secondly, the present invention provides a deep learning-based digital human emotion modeling system, comprising:

[0049] The data acquisition module is used to acquire multimodal stimulus data and current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration and emotional type;

[0050] The emotion positive / negative module is used to judge emotions based on the multimodal stimulus data and obtain the emotion positive / negative type;

[0051] The initial emotion module is used to initialize the emotion based on the positive / negative emotion type and the current emotion state to obtain an initial emotion value.

[0052] The dynamic emotion module is used to simulate the accumulation of emotions based on the stimulation frequency to obtain dynamic emotion values.

[0053] The emotion sequence module is used to update the emotion based on the emotion dynamic value and the emotion initial value to obtain an initial emotion sequence;

[0054] The sequence optimization module is used to input the initial sentiment sequence into a pre-trained sentiment optimization model and output the final optimal sentiment sequence.

[0055] Thirdly, the present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the deep learning-based digital human emotion modeling method described in any one of the above.

[0056] Fourthly, the present invention also provides a computer-readable storage medium comprising a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the deep learning-based digital human emotion modeling method described above.

[0057] Compared to existing technologies, this invention offers the following advantages: This invention discloses a deep learning-based digital human emotion modeling method. The method includes acquiring multimodal stimulus data and a current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration, and emotion type; performing emotion judgment based on the multimodal stimulus data to obtain a positive or negative emotion type; performing emotion initialization based on the positive or negative emotion type and the current emotional state to obtain an initial emotion value; simulating emotion accumulation based on the stimulus frequency to obtain a dynamic emotion value; updating the emotion based on the dynamic emotion value and the initial emotion value to obtain an initial emotion sequence; and inputting the initial emotion sequence into a pre-trained emotion optimization model to output the final optimal emotion sequence. This method can improve the accuracy of digital human emotion modeling.

[0058] Specifically, this method proposes a sentiment assessment step for multimodal stimulus data. First, the collected multimodal stimulus data is numerically encoded, transforming it into machine-processable stimulus feature vectors. This transformation helps capture and represent the complex information in the original data, laying the foundation for subsequent sentiment analysis.

[0059] Next, the obtained stimulus feature vectors are input into a pre-trained sentiment classification model. This model aims to output positive and negative probabilities based on the input feature vectors, thereby helping to determine the positive or negative type of sentiment. If the positive probability is higher than the negative probability, the current stimulus is judged to have positive sentiment; conversely, if the positive probability is lower than the negative probability, the stimulus is considered to have negative sentiment.

[0060] The main technological improvement of this method lies in the accuracy of emotion recognition. By combining multimodal data and deep learning models, it can more comprehensively and accurately understand the user's emotional state. Compared to traditional methods, it not only considers a single type of input but also integrates multi-dimensional data, thus improving the accuracy of emotion type judgment.

[0061] Furthermore, this invention proposes a method for determining an emotional dynamic value and updating emotions based on this value and the initial emotional state. First, for calculating the emotional dynamic value, a formula is proposed to quantify the emotional changes caused by external stimuli of different frequencies. Here, the "emotional dynamic value" represents the cumulative effect of emotions over time; the "cumulative coefficient" modulates the speed or intensity of this emotional accumulation; the "cumulative effect index" affects the degree to which different stimulus frequencies contribute to emotional accumulation; the "total frequency sampling" is the number of external stimulus frequencies considered; and the "stimulus frequency" refers to each individual external stimulus event. This calculation method is designed to capture and quantify emotional changes caused by external stimuli, providing a mathematical approach to simulate how emotions dynamically change with external stimuli in real-world situations.

[0062] Secondly, for the calculation of emotional intensity, an alternative formula is used to transition from the basic emotional state to the current emotional state. Here, "emotional intensity" represents the emotional level at a certain point in time; "initial emotional value" reflects the basic emotional state in the absence of external stimuli; a natural decay function describes the natural decline of emotion over time; the "natural decay coefficient" determines the rate at which emotional intensity decreases over time; "time point number" is used to identify different time points; and "emotional dynamic value" is the result obtained from previous calculations.

[0063] This step facilitates the transition from a basic emotional state to the current emotional state, incorporating both the natural decay of emotions over time and the dynamic changes in emotions brought about by recent stimuli. Finally, the calculated emotional intensity values ​​at each time point are sorted chronologically to generate an initial emotional sequence. This method allows the system to dynamically adjust its emotional expression to adapt to constantly changing environments and interaction needs, thereby achieving a more natural and harmonious human-computer interaction experience. Furthermore, this strategy enhances the scientific rigor and accuracy of emotion modeling, enabling machines to more subtly and realistically mimic human emotional response patterns. Throughout the process, specific numerical calculations transform abstract emotional changes into quantifiable indicators, improving the accuracy of emotion modeling. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of a deep learning-based digital human emotion modeling method provided in the first embodiment of the present invention;

[0065] Figure 2 This is a schematic diagram of the structure of a deep learning-based digital human emotion modeling system provided in the second embodiment of the present invention. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] With the development of artificial intelligence technology, digital humans, as a new type of human-computer interaction interface, are gradually becoming a research hotspot. Digital humans can not only mimic human appearance and behavior, but also simulate human emotional responses through emotion modeling, thereby enhancing the interactive experience with users. Especially in fields such as virtual assistants, customer service, and education and entertainment, digital humans with emotion recognition and expression capabilities can provide a more natural and fluid communication experience. However, to enable digital humans to accurately understand and respond to users' emotional states, an efficient and accurate emotion modeling method is needed.

[0068] In existing technologies, deep learning-based methods for modeling emotions in digital humans involve two main steps: First, a deep neural network model is trained using a large amount of labeled data to understand multiple modalities such as speech, facial expressions, and text. During emotion recognition, structures such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs) are used to analyze the user's speech features or facial expression changes to determine their current emotional state. Second, based on the identified emotional state, generative adversarial networks (GANs) or other types of generative models are used to synthesize corresponding facial expressions or language responses, enabling the digital human to adjust its response according to the user's emotions. This method can effectively improve the realism of digital human interactions and the user experience.

[0069] Despite the significant progress made by the aforementioned methods, several challenges and limitations remain. The intensity of emotions in digital humans changes over time. Existing methods cannot model these time-varying emotional shifts, resulting in low accuracy in digital human emotion modeling.

[0070] To solve the above problems, refer to Figure 1 The first embodiment of the present invention provides a digital human emotion modeling method based on deep learning, comprising the following steps:

[0071] S11, acquire multimodal stimulus data and current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration and emotional type;

[0072] S12, perform emotion judgment based on the multimodal stimulus data to obtain the positive or negative emotion type;

[0073] S13, perform emotion initialization based on the positive / negative emotion type and the current emotion state to obtain the initial emotion value;

[0074] S14, simulate the accumulation of emotion based on the stimulation frequency to obtain the dynamic value of emotion;

[0075] S15, Update the emotion based on the dynamic emotion value and the initial emotion value to obtain the initial emotion sequence;

[0076] S16, input the initial emotion sequence into the pre-trained emotion optimization model, and output the final optimal emotion sequence.

[0077] In step S11, multimodal stimulus data and current emotional state are acquired, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration and emotional type.

[0078] In one implementation, multimodal stimulus data can be acquired in several ways: visual data can be captured via cameras to show the user's facial expressions and body movements; in virtual reality or augmented reality environments, the user's behavior and reactions can be recorded in real time. Auditory data is collected via microphones to gather the user's voice information, including features such as volume, tone, and rhythm, which helps analyze the user's emotional state. Textual data extracts emotional vocabulary and semantic information from the user's input (such as chat logs and comments) to understand their emotional state. The current emotional state is determined by directly asking the user about their self-reported emotional state. Through these methods, the system can comprehensively understand the user's emotional state and respond accordingly.

[0079] It is worth noting that the parameters are explained as follows: Stimulus intensity reflects the strength or intensity of external stimuli, such as the decibel level of sound or the contrast of an image; Stimulus frequency describes the periodicity of the occurrence of a specific stimulus, such as the frequency of a flashing light source or a repetitive sound pattern; Stimulus duration indicates the length of time a stimulus takes to act, such as the playback time of a piece of music or the display duration of a scene; Emotion type reflects the emotional response evoked by external stimuli, such as pleasure, fear, or anger.

[0080] In step S12, emotion judgment is performed based on the multimodal stimulus data to obtain the positive or negative emotion type.

[0081] In one implementation, the multimodal stimulus data is numerically encoded to obtain a stimulus feature vector; the stimulus feature vector is input into a pre-trained sentiment classification model, which outputs positive and negative probabilities; when the positive probability is greater than the negative probability, the current stimulus's sentiment type is determined to be positive; when the positive probability is less than the negative probability, the current stimulus's sentiment type is determined to be negative; wherein, the training process of the sentiment classification model includes: training based on a large-scale labeled sentiment corpus, and obtaining the trained model after detecting that the model's loss function meets the conditions.

[0082] In one implementation, the sentiment classification model employs a pre-trained multilayer perceptron model. This multilayer perceptron network is trained on a large-scale labeled sentiment corpus. During training, the model learns how to identify sentiment-related patterns from input features. During training, network weights are adjusted by monitoring the performance of the loss function to ensure the model achieves satisfactory performance on a given dataset. Once the loss function meets predetermined conditions, training is complete, resulting in a pre-trained model suitable for practical applications.

[0083] It's worth noting that, for better illustration, here's an example: analyzing a user comment from a social media platform to determine whether the comment expresses a positive or negative sentiment. The specific process is as follows: First, the comment text is preprocessed, such as removing stop words and word segmentation. Then, word embedding methods are used to convert each word into a corresponding vector representation, forming the feature vector of the entire comment. This feature vector is then passed as input to a pre-trained multilayer perceptron network. After a series of nonlinear transformations, the multilayer perceptron network outputs the probabilities of positive and negative sentiment at the output layer. For example, for a comment "The food at this restaurant is delicious," the output probability of positive sentiment is 0.85, and the probability of negative sentiment is 0.15. Since the probability of positive sentiment is greater than the probability of negative sentiment, it is judged as positive sentiment.

[0084] In step S13, emotion initialization is performed based on the positive / negative emotion type and the current emotion state to obtain an initial emotion value.

[0085] In one implementation, the weight matrix is ​​calculated using the following formula:

[0086] W T =α·W base +(1-α)·S current

[0087] Among them, W T Let W represent the weight matrix. base S represents the pre-training matrix. current This represents the current emotional state, and α represents the decay coefficient.

[0088] An input vector is constructed based on the weight matrix and the positive / negative sentiment type;

[0089] The input vector is fed into a pre-trained sentiment impact value model, which outputs an initial sentiment value.

[0090] The training process of the emotional impact value model includes:

[0091] An emotional impact value model is constructed based on historical emotional vectors. The model is trained, and the training is considered complete when the loss function of the model meets the conditions, thus obtaining the trained emotional impact value model.

[0092] It's worth noting that the weight matrix is ​​used to combine the pre-trained matrix and the current sentiment state to generate a new matrix that comprehensively reflects the influence of both. The pre-trained matrix, trained on a large amount of historical sentiment data, contains basic response patterns or weight assignments for different types of emotional stimuli. The current sentiment state represents the sentiment state currently identified by the system, a vector representation inferred from multimodal data. The decay coefficient, a coefficient between 0 and 1, determines the relative contribution of the pre-trained matrix and the current sentiment state to the newly generated weight matrix. When the decay coefficient is close to 1, the system tends to use the pre-trained matrix more; when the decay coefficient is close to 0, it relies more on the current sentiment state.

[0093] In one implementation, the sentiment impact value model is implemented using an LSTM network. This model is built and trained based on historical sentiment vectors (i.e., sentiment states and their changes at different points in the past). During training, the model learns how to extract patterns related to sentiment changes from input features and minimize the difference between predicted and actual values ​​(loss function). Once the model's loss function reaches a predetermined criterion, the model is considered to have completed training and is ready to predict new initial sentiment values.

[0094] In step S14, emotion accumulation simulation is performed based on the stimulation frequency to obtain the dynamic value of emotion.

[0095] In one implementation, the emotional dynamic value is calculated using the following formula:

[0096]

[0097] Among them, I dynamic The emotional dynamic value is represented by β, the cumulative coefficient by γ, the cumulative effect index by N, the total number of frequency samples by k, and f. k This indicates the frequency of stimulus number k.

[0098] It's worth noting that the emotional dynamic value represents the cumulative effect of emotion caused by changes in the frequency of external stimuli. This value can be used to describe the trend of emotion gradually increasing or decreasing over time. The cumulative coefficient modulates the speed or intensity of this emotional accumulation. Different application scenarios require adjustment of this coefficient to better match the actual observed emotional responses. The cumulative effect index affects the degree to which different stimulus frequencies contribute to emotional accumulation. By adjusting this index, the different effects of high-frequency and low-frequency stimuli on emotional states can be controlled. For example, a larger value indicates that high-frequency stimuli will have a greater impact on emotional accumulation. The total number of frequency samples refers to the number of external stimulus frequencies considered. By summing the contributions of all frequencies, this formula can reflect the changing trend of emotional states under the combined effect of multiple frequencies of stimuli over a long period. This is crucial for understanding the emotional evolution of long-term exposure to certain environments.

[0099] In step S15, the emotion is updated based on the emotion dynamic value and the emotion initial value to obtain the initial emotion sequence.

[0100] In one implementation, the intensity of emotion is calculated using the following formula:

[0101] I t =I init ·e -λt +I dynamic

[0102] Among them, I t Indicates the emotional intensity at time point t, i init Let e ​​represent the initial value of emotion, λ represent the natural constant, t represent the natural decay coefficient, and I represent the time point number. dynamic Indicates the emotional dynamic value;

[0103] The initial emotional sequence is obtained by sorting the emotional intensity from smallest to largest according to the time point.

[0104] It's worth noting that the natural constant (Euler's number) is set to 2.7 in this method. The natural decay coefficient determines the rate at which emotional intensity naturally decreases over time. A larger value indicates that the emotional intensity weakens more quickly. The purpose of this formula is to describe how emotional intensity evolves over time. The first part demonstrates the natural decay of emotional intensity over time. This decay reflects the phenomenon that human emotions do not remain at a high level indefinitely, but rather gradually return to a baseline level over time. The second part is the cumulative effect of emotion calculated based on the frequency of external stimuli. It does not directly depend on time but exists as an additional influencing factor on emotional intensity. Even without new stimulus input, previously experienced high-frequency stimuli still have a lasting impact on the current emotional state.

[0105] In step S16, the initial emotion sequence is input into a pre-trained emotion optimization model, and the final optimal emotion sequence is output.

[0106] In one embodiment, when the duration of the stimulus exceeds a preset time threshold, lag accumulation is performed according to the emotion sequence function to obtain an initial lag accumulation value;

[0107] The updated sentiment sequence is obtained by weighting the sentiment sequence function and the initial lag cumulative value;

[0108] Based on the updated sentiment sequence, a dual-threshold detection is performed to obtain a discrete sentiment sequence;

[0109] Gradient optimization based on the discrete sentiment sequence includes:

[0110] The gradient of the loss function is calculated using the following formula:

[0111]

[0112] in, Let θ represent the gradient of the loss function, θ represent the hyperparameters of the sentiment optimization model, m represent the number of samples, i represent the sample ID, and h represent the gradient of the loss function. θ (x (i) ) represents the predicted value of the sentiment optimization model for the i-th sample, y (i) Let x represent the value in the discrete sentiment sequence of the i-th sample. (i) This represents the feature vector of the i-th sample;

[0113] The hyperparameters of the sentiment optimization model are updated based on the gradient of the loss function and the learning rate.

[0114] When the loss function of the sentiment optimization model is detected to be less than the preset detection threshold, the optimal sentiment sequence is output.

[0115] It's worth noting that upper and lower thresholds are set to detect significant changes in emotional state. For example, the upper threshold is set to twice the normal fluctuation, while the lower threshold is negative twice. If the emotional intensity fluctuation exceeds these thresholds within a certain period, a sudden change in emotional state is considered to have occurred. The discrete emotional sequence is obtained by converting continuous emotional intensity values ​​into a series of discrete emotional labels (such as happiness, sadness, etc.) after double threshold detection.

[0116] It's worth noting that the lag effect refers to the delayed or cumulative effect of emotional state changes caused by prolonged exposure to external stimuli (such as visual or auditory stimuli) that act on the system for a duration exceeding a preset threshold. This means that even after the external stimulus stops, the individual's emotional response will not immediately return to normal, but will persist for a period of time, or require an even longer time to recover to baseline levels. Therefore, a time threshold is set to determine whether the external stimulus is long enough to induce a lag effect. If the stimulus duration exceeds this threshold, it is considered to trigger a lag effect, and subsequent steps are taken. When the stimulus duration exceeds the preset threshold, the system performs additional calculations based on the previously calculated emotional sequence function to determine the cumulative emotional intensity caused by prolonged stimulation. This reflects the accumulation of emotional changes resulting from long-term exposure to certain stimuli.

[0117] It's worth noting that gradient descent is used for optimization. Partial derivatives measure the direction and magnitude of the difference between the current model's predicted value and the actual value. Minimizing this gradient allows the model to better fit the training data. Hyperparameters represent the set of all adjustable parameters of the sentiment optimization model, including weights, biases, etc. Furthermore, mapping the residuals back to the input space reveals how to adjust the model parameters corresponding to the input features to reduce error. Different features have different impacts on the final prediction result, so their roles need to be considered separately.

[0118] In summary, this invention discloses a deep learning-based digital human emotion modeling method, aiming to improve the accuracy of digital human emotion simulation through a series of steps. First, the method begins by acquiring multimodal stimulus data, including stimulus intensity, frequency, duration, and emotion type, while simultaneously recording the current emotional state. This process lays the foundation for subsequent emotion analysis. Next, emotion judgment is performed based on the collected multimodal stimulus data to determine the positive or negative type of emotion. Numerical encoding is used to transform the raw data into machine-processable stimulus feature vectors. Then, a pre-trained emotion classification model is used to analyze these feature vectors, outputting probability values ​​for positive or negative emotions. Based on these probability values, the emotional tendency of the current stimulus can be determined as positive or negative.

[0119] Furthermore, the method proposed in this invention performs emotion initialization based on the identified positive / negative emotion types and the current emotion state, obtaining an initial emotion value. In this step, an input vector is constructed by calculating a weight matrix and combining it with the positive / negative emotion types. This input vector is then fed into a pre-trained emotion influence value model to obtain the initial emotion value. This process considers the influence of historical emotion data on the current emotion state, making emotion initialization more accurate.

[0120] Subsequently, the method simulates emotional accumulation based on stimulus frequency, calculating the dynamic value of emotion. This part introduces a formula to quantify emotional changes caused by external stimuli, which includes parameters such as the cumulative coefficient and the cumulative effect index to describe the contribution of stimuli of different frequencies to emotional accumulation. This method can effectively capture the trend of emotion changes over time.

[0121] After obtaining the dynamic emotional values, they are combined with the initial emotional values ​​to update the emotional state and generate an initial emotional sequence. Specifically, a natural decay function is used to describe the decline of emotion over time, and the emotional intensity at each time point is calculated accordingly. Finally, these are arranged in chronological order to form the initial emotional sequence. This step realizes the transition from the basic emotional state to the current emotional state, while also taking into account the dynamic changes brought about by recent stimuli.

[0122] Finally, the initial emotional sequence is input into a pre-trained emotional optimization model. After a series of processes including hysteresis accumulation, weighted averaging, dual threshold detection, and gradient optimization, the final optimal emotional sequence is output. During this process, the gradient of the loss function is calculated and hyperparameters are adjusted to ensure that the model's output closely approximates the actual emotional state. It is worth noting that for prolonged external stimuli, this method also considers the hysteresis effect; even after the external stimulus stops, the individual's emotional response will not immediately return to normal, but will exhibit a certain delay or cumulative effect.

[0123] In summary, this invention provides a digital human emotion modeling method that not only considers a single type of input but also integrates multi-dimensional data such as visual, auditory, and textual data, thereby improving the accuracy of emotion type judgment. Furthermore, through a deep understanding and precise simulation of the patterns of emotion change, the digital human achieves greater accuracy in understanding and responding to user emotions.

[0124] Reference Figure 2 The second embodiment of the present invention provides a deep learning-based digital human emotion modeling system, comprising:

[0125] The data acquisition module is used to acquire multimodal stimulus data and current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration and emotional type;

[0126] The emotion positive / negative module is used to judge emotions based on the multimodal stimulus data and obtain the emotion positive / negative type;

[0127] The initial emotion module is used to initialize the emotion based on the positive / negative emotion type and the current emotion state to obtain an initial emotion value.

[0128] The dynamic emotion module is used to simulate the accumulation of emotions based on the stimulation frequency to obtain dynamic emotion values.

[0129] The emotion sequence module is used to update the emotion based on the emotion dynamic value and the emotion initial value to obtain an initial emotion sequence;

[0130] The sequence optimization module is used to input the initial sentiment sequence into a pre-trained sentiment optimization model and output the final optimal sentiment sequence.

[0131] Preferably, the data acquisition module is used for:

[0132] Acquire multimodal stimulus data and current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration, and emotional type.

[0133] Preferably, the positive / negative emotion module is used for:

[0134] Emotional judgment is performed based on the multimodal stimulus data to obtain positive or negative emotion types, including:

[0135] Numerical encoding is performed on the multimodal stimulus data to obtain stimulus feature vectors;

[0136] The stimulus feature vector is input into a pre-trained sentiment classification model, which outputs positive and negative probabilities.

[0137] When the positive probability is greater than the negative probability, the current stimulus's emotional type is determined to be positive.

[0138] When the positive probability is less than the negative probability, the current stimulus is determined to be a negative emotion.

[0139] The training process of the sentiment classification model includes:

[0140] The trained model is obtained after the loss function of the model meets the conditions, based on training on a large-scale labeled sentiment corpus.

[0141] Preferably, the initial emotion module is used for:

[0142] Emotion initialization is performed based on the positive / negative emotion type and the current emotion state to obtain initial emotion values, including:

[0143] The weight matrix is ​​calculated using the following formula:

[0144] W T =α·W base +(1-α)·S current

[0145] Among them, W T Let W represent the weight matrix. baseS represents the pre-training matrix. current This represents the current emotional state, and α represents the decay coefficient.

[0146] An input vector is constructed based on the weight matrix and the positive / negative sentiment type;

[0147] The input vector is fed into a pre-trained sentiment impact value model, which outputs an initial sentiment value.

[0148] The training process of the emotional impact value model includes:

[0149] An emotional impact value model is constructed based on historical emotional vectors. The model is trained, and the training is considered complete when the loss function of the model meets the conditions, thus obtaining the trained emotional impact value model.

[0150] Preferably, the dynamic emotion module is used for:

[0151] Based on the stimulation frequency, an emotion accumulation simulation is performed to obtain dynamic emotion values, including:

[0152] The emotional dynamic value is calculated using the following formula:

[0153]

[0154] Among them, I dynamic The emotional dynamic value is represented by β, the cumulative coefficient by γ, the cumulative effect index by N, the total number of frequency samples by k, and f. k This indicates the frequency of stimulus number k.

[0155] Preferably, the emotion sequence module is used for:

[0156] Based on the dynamic emotional value and the initial emotional value, an emotional update is performed to obtain an initial emotional sequence, including:

[0157] Emotional intensity is calculated using the following formula:

[0158] I t =I init ·e -λt +I dynamic

[0159] Among them, I t I represents the emotional intensity at time point t. init Let e ​​represent the initial value of emotion, λ represent the natural constant, t represent the natural decay coefficient, and I represent the time point number. dynamic Indicates the emotional dynamic value;

[0160] The initial emotional sequence is obtained by sorting the emotional intensity from smallest to largest according to the time point.

[0161] Preferably, the sequence optimization module is used for:

[0162] The initial sentiment sequence is input into a pre-trained sentiment optimization model, which outputs the final optimal sentiment sequence, including:

[0163] When the duration of the stimulus exceeds a preset time threshold, lag accumulation is performed according to the emotion sequence function to obtain an initial lag accumulation value;

[0164] The updated sentiment sequence is obtained by weighting the sentiment sequence function and the initial lag cumulative value;

[0165] Based on the updated sentiment sequence, a dual-threshold detection is performed to obtain a discrete sentiment sequence;

[0166] Gradient optimization is performed on the discrete sentiment sequence to obtain the optimal sentiment sequence, including:

[0167] The gradient of the loss function is calculated using the following formula:

[0168]

[0169] in, Let θ represent the gradient of the loss function, θ represent the hyperparameters of the sentiment optimization model, m represent the number of samples, i represent the sample ID, and h represent the gradient of the loss function. θ (x (i) ) represents the predicted value of the sentiment optimization model for the i-th sample, y (i) Let x represent the value in the discrete sentiment sequence of the i-th sample. (i) This represents the feature vector of the i-th sample;

[0170] The hyperparameters of the sentiment optimization model are updated based on the gradient of the loss function and the learning rate.

[0171] When the loss function of the sentiment optimization model is detected to be less than the preset detection threshold, the optimal sentiment sequence is output.

[0172] It should be noted that the deep learning-based digital human emotion modeling system provided in this embodiment of the invention is used to execute all the process steps of the deep learning-based digital human emotion modeling method in the above embodiment. The working principles and beneficial effects of the two are one-to-one, so they will not be described again.

[0173] This invention also provides an electronic device. The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, such as a data acquisition program. When the processor executes the computer program, it implements the steps described in the various embodiments of the deep learning-based digital human emotion modeling method, for example... Figure 1The step S11 shown. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the above-described device embodiments, such as the data acquisition module.

[0174] For example, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the electronic device.

[0175] The electronic device may be a desktop computer, laptop, handheld computer, or smart tablet, etc. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above components are merely examples of electronic devices and do not constitute a limitation on the electronic device. It may include more or fewer components than described above, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0176] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting all parts of the electronic device via various interfaces and lines.

[0177] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0178] Wherein, if the modules / units integrated in the electronic device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0179] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0180] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for modeling digital human emotions based on deep learning, characterized in that, include: Acquire multimodal stimulus data and current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration, and emotional type; Emotional judgment is performed based on the multimodal stimulus data to obtain the positive or negative emotion type; Emotion initialization is performed based on the positive / negative emotion type and the current emotion state to obtain the initial emotion value; Based on the stimulation frequency, an emotional accumulation simulation is performed to obtain the dynamic value of the emotion. The initial emotion sequence is obtained by updating the emotion based on the dynamic emotion value and the initial emotion value. The initial sentiment sequence is input into a pre-trained sentiment optimization model, which outputs the final optimal sentiment sequence. The step of initializing the emotion based on the positive / negative emotion type and the current emotion state to obtain the initial emotion value includes: The weight matrix is ​​calculated using the following formula: in, Represents the weight matrix. Represents the pre-trained matrix. Indicates the current emotional state. Indicates the attenuation coefficient; An input vector is constructed based on the weight matrix and the positive / negative sentiment type; The input vector is fed into a pre-trained sentiment impact value model, which outputs an initial sentiment value. The training process of the emotional impact value model includes: A sentiment impact value model is constructed based on historical sentiment vectors. The model is trained, and the training is considered complete after the loss function of the model meets the conditions. The trained sentiment impact value model is obtained. The step of simulating emotional accumulation based on the stimulus frequency to obtain dynamic emotional values ​​includes: The emotional dynamic value is calculated using the following formula: in, Represents the emotional dynamic value. Represents the cumulative coefficient. Indicates the cumulative effect index. Indicates the total number of frequency samples. Indicates the frequency number. express Stimulation frequency; The step of updating the emotion based on the dynamic emotion value and the initial emotion value to obtain the initial emotion sequence includes: Emotional intensity is calculated using the following formula: in, express The intensity of emotions at that time point Indicates the initial value of emotion. Represents the natural constant. Indicates the natural decay coefficient. Number the time points, Indicates the emotional dynamic value; The initial emotional sequence is obtained by sorting the emotional intensity from smallest to largest according to the time point.

2. The deep learning-based digital human emotion modeling method according to claim 1, characterized in that, The step of judging emotions based on the multimodal stimulus data to obtain positive or negative emotion types includes: Numerical encoding is performed on the multimodal stimulus data to obtain stimulus feature vectors; The stimulus feature vector is input into a pre-trained sentiment classification model, which outputs positive and negative probabilities. When the positive probability is greater than the negative probability, the current stimulus's emotional type is determined to be positive. When the positive probability is less than the negative probability, the current stimulus is determined to be a negative emotion. The training process of the sentiment classification model includes: The trained model is obtained after the loss function of the model meets the conditions, based on training on a large-scale labeled sentiment corpus.

3. The deep learning-based digital human emotion modeling method according to claim 1, characterized in that, The step of inputting the initial sentiment sequence into a pre-trained sentiment optimization model and outputting the final optimal sentiment sequence includes: When the duration of the stimulus exceeds a preset time threshold, lag accumulation is performed according to the emotion sequence function to obtain an initial lag accumulation value; The updated sentiment sequence is obtained by weighting the sentiment sequence function and the initial lag cumulative value; Based on the updated sentiment sequence, a dual-threshold detection is performed to obtain a discrete sentiment sequence; The optimal emotional sequence is obtained by performing gradient optimization based on the discrete emotional sequence.

4. The deep learning-based digital human emotion modeling method according to claim 3, characterized in that, The step of performing gradient optimization based on the discrete sentiment sequence to obtain the optimal sentiment sequence includes: The gradient of the loss function is calculated using the following formula: in, Represents the gradient of the loss function. This represents the hyperparameters of the sentiment optimization model. Indicates the number of samples. Indicates the sample number. The sentiment optimization model represents the first... The predicted value for each sample, Indicates the first The values ​​in a discrete sentiment sequence of a sample. Indicates the first Feature vectors of each sample; The hyperparameters of the sentiment optimization model are updated based on the gradient of the loss function and the learning rate. When the loss function of the sentiment optimization model is detected to be less than the preset detection threshold, the optimal sentiment sequence is output.

5. A deep learning-based digital human emotion modeling system, used to implement the deep learning-based digital human emotion modeling method as described in any one of claims 1 to 4, characterized in that, include: The data acquisition module is used to acquire multimodal stimulus data and current emotional state, wherein the multimodal stimulus data includes: stimulus intensity, stimulus frequency, stimulus duration and emotional type; The emotion positive / negative module is used to judge emotions based on the multimodal stimulus data and obtain the emotion positive / negative type; The initial emotion module is used to initialize the emotion based on the positive / negative emotion type and the current emotion state to obtain an initial emotion value. The dynamic emotion module is used to simulate the accumulation of emotions based on the stimulation frequency to obtain dynamic emotion values. The emotion sequence module is used to update the emotion based on the emotion dynamic value and the emotion initial value to obtain an initial emotion sequence; The sequence optimization module is used to input the initial sentiment sequence into a pre-trained sentiment optimization model and output the final optimal sentiment sequence.

6. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the deep learning-based digital human emotion modeling method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the deep learning-based digital human emotion modeling method as described in any one of claims 1 to 4.

Images