Digital human control method and device, equipment and storage medium

Abstract

The invention provides a digital human control method, device and equipment and a storage medium, relates to the field of artificial intelligence, in particular to the technical fields of computer vision, reinforcement learning, digital humans, large models and the like, and can be applied to scenes such as virtual reality and digital humans. According to the specific implementation scheme, emotion features are obtained based on emotion data interacting with the digital person; obtaining short-term memory features and long-term memory features based on the emotion features; fusing the short-term memory feature and the long-term memory feature to obtain a predicted emotional state; generating an interaction strategy of the digital person based on the predicted emotional state; wherein the interaction strategy is used for controlling the emotional response of the digital person.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, reinforcement learning, digital humans, large models and other technical fields, and can be applied to virtual reality, digital humans and other scenarios. Background Art

[0002] In the field of digital interaction, digital humans are gradually becoming important interactive partners in daily life. From virtual assistants, online customer service to digital roles in virtual social networking, the application scope of digital humans is becoming increasingly wide. Most digital humans focus mainly on processing and interacting with content, and lack the ability to deeply understand and respond to user emotions. Summary of the invention

[0003] The present disclosure provides a digital human control method, device, equipment and storage medium.

[0004] According to one aspect of the present disclosure, a method for controlling a digital human is provided, comprising:

[0005] Get emotional features based on the emotional data of interaction with digital humans;

[0006] Based on the emotional features, short-term memory features and long-term memory features are obtained;

[0007] The short-term memory feature and the long-term memory feature are combined to obtain a predicted emotional state;

[0008] An interaction strategy of the digital human is generated based on the predicted emotional state; wherein the interaction strategy is used to control the emotional response of the digital human.

[0009] According to another aspect of the present disclosure, a method for training an emotion interaction model is provided, comprising:

[0010] Inputting sample emotion data of interaction with the digital human into the emotion interaction model to obtain sample emotion features;

[0011] The sample emotion feature is input into the short-term memory network and the long-term memory network of the emotion interaction model to obtain the sample short-term memory feature and the sample long-term memory feature;

[0012] The emotion interaction model is trained based on the short-term memory characteristics of the sample and the long-term memory characteristics of the sample.

[0013] According to another aspect of the present disclosure, there is provided a control device for a digital human, comprising:

[0014] An emotion feature acquisition module, used to obtain emotion features based on the emotion data of the interaction with the digital human;

[0015] A memory feature acquisition module, used to obtain short-term memory features and long-term memory features based on the emotional features;

[0016] A memory feature fusion module, used to fuse the short-term memory feature and the long-term memory feature to obtain a predicted emotional state;

[0017] The interactive strategy generation module is used to generate an interactive strategy for the digital human based on the predicted emotional state; wherein the interactive strategy is used to control the emotional response of the digital human.

[0018] According to another aspect of the present disclosure, there is provided a training device for an emotion interaction model, comprising:

[0019] The emotion feature acquisition module is used to input the sample emotion data of the interaction with the digital human into the emotion interaction model to obtain the sample emotion features;

[0020] A memory feature acquisition module, used for inputting the sample emotion feature into the short-term memory network and the long-term memory network of the emotion interaction model to obtain the sample short-term memory feature and the sample long-term memory feature;

[0021] The training module is used to train the emotion interaction model based on the short-term memory characteristics of the sample and the long-term memory characteristics of the sample.

[0022] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method in the embodiments of the present disclosure.

[0023] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0024] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0025] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0027] Figure 1is a flow chart of a method for controlling a digital human according to an embodiment of the present disclosure;

[0028] Figure 2 is a flow chart of a method for controlling a digital human according to another embodiment of the present disclosure;

[0029] Figure 3 is a flow chart of a method for controlling a digital human according to another embodiment of the present disclosure;

[0030] Figure 4 is a flow chart of a method for controlling a digital human according to another embodiment of the present disclosure;

[0031] Figure 5 is a flow chart of a method for training an emotion interaction model according to an embodiment of the present disclosure;

[0032] Figure 6 is a flow chart of a method for training an emotion interaction model according to another embodiment of the present disclosure;

[0033] Figure 7 is a flow chart of a method for training an emotion interaction model according to another embodiment of the present disclosure;

[0034] Figure 8 is a flow chart of a method for training an emotion interaction model according to another embodiment of the present disclosure;

[0035] Fig. 9 is a structural schematic diagram of a control device for a digital human according to an embodiment of the present disclosure;

[0036] Fig.10 is a structural schematic diagram of a control device for a digital human according to another embodiment of the present disclosure;

[0037] Fig.11 is a structural schematic diagram of a training device for an emotion interaction model according to an embodiment of the present disclosure;

[0038] Fig.12 is a structural schematic diagram of a training device for an emotion interaction model according to another embodiment of the present disclosure;

[0039] Fig.13 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0041] As users' demands for personalized and emotional experiences continue to increase, the simple content interaction of digital humans can no longer meet users' needs in terms of emotional support, psychological counseling, personalized interaction, etc. The interaction between digital humans and users mainly relies on content-based memory and reference. This interaction mode usually uses simple conversation records and content matching algorithms. The system will provide corresponding reference information in future interactions based on the user's past conversation content.

[0042] The following are several features of the digital human interaction system:

[0043] 1. Historical memory based on content:

[0044] The relevant system records the user's conversation content, including the user's questions, answers, preferences and some emotional expressions (such as emoticons). These contents are stored in the database, and the system calls these records in future conversations to achieve simple personalized interactions. For example, if the user has asked a question in the past, the system will provide a quick answer when asked again.

[0045] 2. Keyword matching and simple classification:

[0046] Through keyword matching or rule engines, the relevant system can identify specific content in the conversation and call the corresponding preset response. This method is usually used to deal with common user questions or provide basic personalized suggestions, but it is difficult to cope with complex emotional changes or interactive scenarios with changing contexts.

[0047] 3. Limited personalization:

[0048] Related systems can remember basic information about users (such as name, interests, etc.) and simply use this information to customize greetings or recommendations during interactions. However, this kind of personalization is usually static and lacks dynamic perception and response to the user's emotional state.

[0049] 4. Fixed response mode:

[0050] Related systems usually use predefined fixed response modes to trigger specific responses based on the identified content. Although this method can meet the user's interactive needs to a certain extent, it appears mechanical and lacks emotional resonance, making it difficult to achieve deep emotional interaction.

[0051] 5. Limitations of emotion recognition:

[0052] The emotion recognition capabilities of related systems are relatively basic, and can only classify emotions through emoticons or simple intonation and text content analysis, lacking comprehensive analysis and application of multimodal emotion data. Therefore, related systems cannot truly understand and respond to users' emotional changes during interaction.

[0053] In this framework, the interaction of digital humans is more centered around content, with limited understanding and response to user emotions, and lacks the ability to dynamically adjust and deeply interact. This defect causes users to feel that the interaction is cold and stiff during use, and cannot meet their expectations for more natural and emotional interactions.

[0054] The interactive system based on content memory can achieve personalization and customization to a certain extent. The embodiments of the present disclosure can be improved in one or more of the following aspects:

[0055] 1. Improve emotional perception:

[0056] The relevant systems mainly rely on content and simple keyword matching, and cannot deeply perceive the user's emotional state. Even in simple emotion recognition, the relevant systems can only make limited judgments based on surface features (such as expressions), and cannot accurately understand the user's emotional changes in different situations. This leads to a lack of resonance and response to user emotions in the interaction, making the experience seem mechanical and indifferent.

[0057] 2. Improve emotional continuity and consistency:

[0058] The relevant system only memorizes and processes the content of the conversation, but does not effectively record and manage emotional information. Users may feel emotional disconnection in multiple interactions with digital people. For example, a user shows extreme anxiety in a conversation, but the system completely ignores this emotional change in the next interaction and directly cuts into a new topic, causing users to feel that the interaction lacks care and personalization.

[0059] 3. Rigid response mode:

[0060] The response mode of related systems is usually pre-defined and fixed, lacking flexibility and the ability to dynamically adjust. Even when certain specific emotions or behaviors of users are recognized, the system can only provide preset standardized responses and cannot make more appropriate emotional responses based on specific situations. This rigid response mode makes the interactive experience seem stiff and lacks natural emotional flow.

[0061] 4. One-way interaction:

[0062] The interaction of related systems is often one-way, that is, the system passively waits for users to initiate interaction and responds according to established rules. The lack of active emotional care and interaction makes it impossible for users to feel the presence of digital people as "partners", and the interaction seems cold and boring, making it difficult to maintain long-term user stickiness.

[0063] 5. Management of emotional data:

[0064] The relevant systems lack systematic management and utilization of emotional data, and are unable to achieve the accumulation, analysis and intelligent application of emotional data. Even if the user's emotional state is successfully identified in a specific interaction, this data will not be stored and used for a long time, resulting in each interaction starting from scratch, without the accumulation and evolution of emotional memory.

[0065] 6. Lack of personalization and difficulty in adapting to diverse needs:

[0066] The personalization function of related systems is limited to simple content memory and call, and it is difficult to achieve true emotional personalization. Users' emotions are diverse and complex, and a single memory method cannot adapt to the personalized emotional needs of different users, nor can it provide in-depth emotional support.

[0067] Therefore, the relevant digital human interaction system seems to be unable to meet the growing emotional and personalized needs of users. The digital human controller provided by the disclosed embodiment can deeply perceive, remember and manage the user's emotional state during interaction, providing more natural, intimate and continuous emotional support.

[0068] Figure 1 1 is a flow chart of a method 100 for controlling a digital human according to an embodiment of the present disclosure, the method comprising:

[0069] S110, obtaining emotional features based on the emotional data of interaction with the digital human;

[0070] S120, obtaining a short-term memory feature and a long-term memory feature based on the emotion feature;

[0071] S130, fusing the short-term memory feature and the long-term memory feature to obtain a predicted emotional state;

[0072] S140: Generate an interactive strategy for the digital human based on the predicted emotional state; wherein the interactive strategy is used to control the emotional response of the digital human.

[0073] In the embodiments of the present disclosure, a digital human is usually a virtual character that exists in a digital form in a digital space and has an anthropomorphic or real person's appearance, behavior, and characteristics. It can also be referred to as a virtual image, a digital virtual human, a virtual digital human, a virtual human, a virtual assistant, a digital assistant, etc. The control method of the digital human in the embodiments of the present disclosure can be executed by a control system of the digital human (or referred to as an interactive system of the digital human, an emotional memory and management system, etc.). The interactive system of the digital human can obtain various types of emotional data from the interactive object, such as the interaction process between the user and the digital human. For example, emotional data can include voice, expression, text, physiological and other data generated in the interactive behavior.

[0074] In some examples, voice data can be collected using recording devices such as microphones, expression data of interactive objects can be collected using video recording devices such as cameras, and text data of functional interactive objects can be collected using keyboards, touch screens, etc. From the emotional data of the interaction between the interactive object and the digital human, emotional features can be extracted through the emotional interaction model.

[0075] In the disclosed embodiment, emotion can also be replaced by mood, feeling, etc. Emotion data can include short-term emotion data and long-term emotion data. Short-term emotion data can include voice, expression, text, physiological data, etc. generated in the current or recent conversations between the digital human and the user, such as three conversations. Long-term emotion data can include voice, expression, text, physiological data, etc. generated in historical conversations between the digital human and the user. For example, long-term emotion data within one month. Short-term emotion features can be extracted from short-term emotion data, and long-term emotion features can be extracted from long-term emotion data.

[0076] In the disclosed embodiments, memory may also be referred to as emotional memory. A memory network may be used to obtain short-term memory features based on short-term emotional features, and to obtain long-term memory features based on long-term emotional features. Short-term memory features may represent real-time emotional fluctuations, and long-term memory features may represent long-term emotional trends. By means of memory fusion, such as a fusion algorithm, short-term memory features and long-term memory features may be fused to obtain fused memory features, which may be converted into or directly used as predicted emotional states. The predicted emotional state may represent the current emotional state of the interactive object. For example, the emotional state may be a negative state or a quantified value thereof, or a positive state or a quantified value thereof.

[0077] In the disclosed embodiment, the interaction strategy of the digital human can be generated based on the predicted emotional state. There can be multiple interaction strategies. The interaction strategy can include short-term emotional changes and / or long-term emotional trends of the interactive object, and various emotional responses that the digital human needs to perform when interacting with the interactive object, such as the voice, expression, and text. For example, the user inputs the text "I am so tired today", the interaction strategy of the digital human can be a strategy to comfort the interactive object, and the emotional response of the digital human can be the reply text "You can go and rest for a while". For another example, if the user's expression is recognized as an angry expression, the interaction strategy of the digital human can be a strategy to calm the interactive object, and the emotional response of the digital human can be to adjust the tone and / or expression of the interaction between the digital human and the object.

[0078] According to the embodiments of the present disclosure, based on the fusion of short-term memory features and long-term memory features, the generated digital human interaction strategy is more suitable for the interactive object, which can improve the interaction consistency and accuracy between the digital human and the interactive object.

[0079] In one embodiment, the emotion data includes short-term emotion data and long-term emotion data, and the priority of the long-term emotion data is determined based on at least one of the following: emotion intensity, time weight, modality credibility, memory relevance, and personalized characteristics.

[0080] In the disclosed embodiment, from all the stored historical emotion data, long-term emotion data that is more relevant to the current short-term emotion data may be acquired according to a certain priority.

[0081] 1. Priority influencing factors of sentiment data.

[0082] The priority of sentiment data can be evaluated based on the influencing factors of various dimensions, as shown below:

[0083] (1) Emotional intensity: used to identify the intensity of emotional expression. For example, an angry tone or a very frowning expression may have a higher priority.

[0084] (2) Temporal attributes: For example, recent emotional expressions are more timely and have a higher priority.

[0085] (3) Data source credibility (modality credibility): Some modalities may be more credible in specific scenarios. For example, when faced with ambiguous speech, facial expressions may be more recognizable.

[0086] (4) Memory relevance: Data that matches the user’s historical emotional state or memory more closely has higher priority.

[0087] (5) User personalization preferences (personalized characteristics): Some users may prefer interactions in a specific mode, such as paying more attention to tone of voice rather than facial expressions.

[0088] 2. Dynamic adjustment mechanism: The priority is not fixed and can change dynamically with the situation. For example, when the user's current emotions fluctuate violently, the priority of real-time emotion intensity data increases. For another example, when the user shows anxiety and this emotion has appeared frequently in his historical interactions, the priority of memory data related to this emotion increases.

[0089] 3. Execution strategy:

[0090] During the interaction, the system where the digital human resides can execute one or more of the following data processing strategies based on priority:

[0091] High-priority emotional data is processed first and directly affects the system output (such as tone and expression adjustment).

[0092] Medium and low priority data are used as secondary references to supplement and optimize responses.

[0093] When there is conflicting data (such as conflicting emotions expressed by speech and facial expressions), the data modality with a higher priority is selected first.

[0094] 4. To implement the priority mechanism, please refer to the following steps:

[0095] (1) Calculation of sentiment intensity

[0096] Calculate the intensity value S for each piece of sentiment data. The example is as follows:

[0097] Speech emotion intensity: quantified through features such as volume, pitch fluctuation, and speech speed changes.

[0098] Intensity of facial expression: quantified by the amplitude of facial muscle movements or micro-expression changes.

[0099] Text sentiment strength: The output probability value of sentiment vocabulary scoring or sentiment analysis model.

[0100] (2) Time weight setting

[0101] Combine the time attribute to assign time weight to the sentiment data. For example, the time weight of sentiment data W 时间 It can be calculated by exponential decay function, the formula is as follows: 时间 =e -λt Where t is the time interval of the data and λ is the decay coefficient.

[0102] (3) Comprehensive priority calculation

[0103] The priority P is calculated by comprehensively considering factors such as sentiment intensity, time weight, and modal credibility. An example is as follows:

[0104] P=w1·S+w2·W 时间+w3·Modal credibility

[0105] Among them, w1, w2, and w3 are weight coefficients, reflecting the importance of each factor.

[0106] (4) Dynamically call the most relevant memory

[0107] According to the priority sorting, the historical memory that best matches the current emotion (such as short-term emotional data) is dynamically called. For example, if the current user is "anxious", the system can prioritize the data related to anxiety situations in the historical memory based on the priority of the historical emotional data for reference in the reply.

[0108] According to the embodiments of the present disclosure, priorities can be set by time, emotion intensity, etc., and long-term emotion data that is more relevant to short-term emotion data can be used preferentially to obtain a more appropriate interaction strategy.

[0109] Figure 2 2 is a flow chart of a method 200 for controlling a digital human according to another embodiment of the present disclosure. The method 200 can be used to implement step S110 in the method 100 for controlling a digital human. In one embodiment, the method 200 includes: obtaining an emotion feature based on emotion data of interaction with the digital human, and further includes:

[0110] S210, extracting features of different modalities from multimodal emotion data interacting with the digital human through a multimodal network of an emotion interaction model;

[0111] S220, adjusting the weights of the features of the different modalities through the cross-modal attention mechanism of the multimodal network;

[0112] S230, obtaining the emotion feature based on the fusion of the features of the different modalities and their adjusted weights.

[0113] In the disclosed embodiments, multimodal emotion data may include at least two types of data among text, voice, expression, and physiology. For example, voice data may include voice emotions such as pitch, speech rate, and volume. Expression data may include facial expressions such as micro-expressions and geometric features of key points. Text data may include text content such as emotion keywords extracted from input content through semantic analysis. Physiological data may include physiological signals such as heart rate and galvanic skin response.

[0114] In the disclosed embodiment, the emotion interaction model may include a multimodal network. The emotion interaction model may be part of a digital human or may be separate from a digital human. The training method of the emotion interaction model may refer to the relevant description of the training method of the model in the following embodiment. The emotion features of different modalities can be extracted from the multimodal emotion data through the multimodal network. For example, text features, voice features, expression features, etc. The emotion features of different modalities can be fused through a cross-attention mechanism at the attention layer of the multimodal network. For example, corresponding weights can be set for the features of different modalities, and the features of different modalities can be fused by weight to obtain a comprehensive emotion feature. The emotion feature may include an emotion sequence, an emotion vector, or an emotion matrix. The fusion weights corresponding to the features of different modalities can be adjusted. For example, if you pay more attention to the expression of the interactive object, you can increase the fusion weight of the expression feature. If you pay more attention to the voice of the interactive object, you can increase the fusion weight of the voice feature. Based on the features of different modalities and the adjusted weights, a fusion calculation is performed to obtain the overall emotion feature. For example, the overall emotional feature = F1×w1+F2×w1+F3×w3, where F1, F2 and F3 represent text features, voice features, and expression features respectively; w1, w2 and w3 represent the weights corresponding to the above features.

[0115] Multimodal emotion data may include multimodal short-term emotion data and multimodal long-term emotion data. Short-term emotion features such as short-term emotion sequence features can be extracted based on multimodal short-term emotion data, and long-term emotion features such as long-term emotion sequence features can be extracted based on multimodal long-term emotion data. According to the disclosed embodiment, through the multimodal emotion capture technology, the digital human can fully and delicately understand the emotional state of the object and provide a more accurate emotional response.

[0116] Figure 3 3 is a flow chart of a digital human control method 300 according to another embodiment of the present disclosure, and the method 300 can be used to implement step S120 in the digital human control method 100. In one embodiment, the emotional feature includes a short-term emotional sequence feature and a long-term emotional sequence feature, and the method 300 includes: obtaining a short-term memory feature and a long-term memory feature based on the emotional feature, and further includes:

[0117] S310, inputting the short-term emotion sequence feature into the short-term emotion network of the emotion interaction model to obtain the short-term memory feature;

[0118] S320: Input the long-term emotion sequence feature into the long-term emotion network of the emotion interaction model to obtain the long-term memory feature.

[0119] In the disclosed embodiments, emotional features may include short-term emotional features and long-term emotional features. Short-term emotional features may include short-term emotional sequence features, and long-term emotional features may include long-term emotional sequence features. The emotional interaction model may also include a short-term emotional network and a long-term emotional network. The short-term emotional network and the long-term emotional network may be two independent networks or associated networks. After the short-term emotional sequence features of an interactive object are input into the short-term emotional network, the short-term emotional network may output the short-term memory features of the interactive object. The long-term emotional sequence features of an interactive object are input into the long-term emotional network of the emotional interaction model to obtain the long-term memory features of the interactive object.

[0120] According to the disclosed embodiment, the emotional interaction model can memorize and manage the memory (or emotional history) of the interactive object, so that the digital human and the interactive object can maintain emotional continuity in multiple interactions. This continuity can make the interactive object feel the digital human's attention and understanding of their emotional state, which not only improves the coherence of the interactive experience, but also enhances the interactive object's trust in the digital human.

[0121] Figure 4 4 is a flow chart of a digital human control method 400 according to another embodiment of the present disclosure, and the method 400 can be used to implement step S130 in the digital human control method 100. In one embodiment, the method 400 includes: fusing the short-term memory feature and the long-term memory feature to obtain a predicted emotional state, and further includes:

[0122] S410, based on the emotion data, obtaining short-term memory feature requirements and long-term memory feature requirements;

[0123] S420, obtaining a short-term memory storage feature and a long-term memory storage feature based on the short-term memory feature and the long-term memory feature;

[0124] S430, obtaining a first matching degree between short-term emotion and short-term memory based on short-term memory feature requirements and short-term memory storage features;

[0125] S440, obtaining a second matching degree between the long-term emotion and the long-term memory based on the long-term memory feature requirement and the long-term memory storage feature;

[0126] S450, determining the weights of the short-term memory feature and the long-term memory feature based on the first matching degree and the second matching degree;

[0127] S460: Based on the weights of the short-term memory feature and the long-term memory feature, the short-term memory feature and the long-term memory feature are fused to obtain the predicted emotional state.

[0128] In the disclosed embodiment, not only emotional features but also memory feature requirements can be extracted from the emotional data. The first transformation network in the emotional interaction model generates memory feature requirements based on the emotional data (such as speech, expression and text features). For example, the first transformation network can extract short-term memory feature requirements Q from short-term emotional data. s (It can also be called the feature demand related to short-term memory in emotional data) and extract the long-term memory feature demand Q from the long-term emotional data l (Feature requirements related to long-term memory in sentiment data). Feature requirements can be in the form of vectors or other forms.

[0129] In the embodiment of the present disclosure, the second transformation network in the emotion interaction model is used to transform the emotion interaction model based on the memory feature (such as the short-term memory feature M s and long-term memory features M l ), can be transformed to obtain memory storage features (such as short-term memory storage features K s and long-term memory storage feature K l ).

[0130] Adopt Q s ·K s The first matching degree between the current emotional needs of the interaction object and the short-term memory can be calculated. For example, a higher first matching degree score indicates that the short-term memory is more important in predicting the emotional state. l ·K l The second matching degree between the current emotional needs of the interactive object and the long-term memory can be calculated. For example, a higher second matching degree score indicates that the long-term memory is more important in predicting the emotional state.

[0131] If the predicted emotional state fluctuates dramatically (e.g., emotion suddenly changes from pleasure to anger), the first matching degree Q corresponding to the short-term memory is s ·K s Higher, can increase the short-term memory feature M s The weight w s , mainly referring to short-term memory. If the predicted emotional state is relatively stable, it is necessary to understand the long-term emotional background (such as the long-term anxiety state of the interactive object). The second matching degree Q corresponding to the long-term memory l ·K l Higher, can increase w l .

[0132] By Q s ·K s and Q l ·K l Score, calculate weight w s and w l Finally, the following formula can be used to finally integrate short-term memory and long-term memory: S = ws ·M s +w l ·M l , generating a predicted affective state S, which is used as the basis for subsequent affective prediction, intervention, or memory updating.

[0133] According to the embodiments of the present disclosure, the fusion weights of short-term memory features and long-term memory features can be determined more reasonably through the degree of matching between memory and emotional needs. The two memory features are fused based on the fusion weights to obtain the predicted emotional state of the interactive object. The interactive strategy of the digital human generated based on the predicted emotional state can better meet the needs of the interactive object.

[0134] In one implementation, step S140 generates an interaction strategy of the digital human based on the predicted emotional state, including: inputting the predicted emotional state into an interaction strategy network of the emotional interaction model to obtain the interaction strategy.

[0135] In the disclosed embodiment, the emotional interaction model may include an interaction strategy network. The interaction strategy network may predict the emotional state according to the interaction object and output the interaction strategy of the digital human. The interaction strategy output by the interaction strategy network may include identification information such as the name and number of the interaction strategy, and may also include the specific content of the interaction strategy, such as text, voice, expression, etc. The specific content of the interaction strategy may be obtained according to the identification information of the interaction strategy. Then, the system in which the digital human is located may control the digital human to communicate with the interaction object through text, voice, expression, etc. according to the specific content of the interaction strategy.

[0136] According to the embodiments of the present disclosure, a more suitable interaction strategy for a digital human can be generated through an interaction strategy network, which can improve the interaction consistency and rationality between the digital human and the interaction object.

[0137] In one embodiment, the method further includes: receiving feedback information regarding the emotional response of the digital human; obtaining an emotional change state based on the feedback information through the interactive strategy network; and adjusting the interactive strategy based on the emotional change state through the interactive strategy network.

[0138] In the disclosed embodiment, after the digital human performs an emotional response to the predicted emotional state of the interactive object, the interactive object may perform some feedback actions, such as smiling, changing the tone of voice to cheerful, etc. According to the feedback information corresponding to these feedback actions, for example, the facial recognition result is a smile, the tone of voice recognition result is cheerful, etc. These feedback information can be used to update the last predicted emotional state S and obtain the emotional change state S'. For example, the value of the feedback information can also have a corresponding weight, and the emotional change state S' is calculated together with the short-term memory and the long-term memory. For another example, the feedback information is used to update the short-term emotional change state and / or the long-term emotional change state

[0139] In the disclosed embodiment, these feedback information can also be used to update part or all of the short-term memory data of the previous time, and part or all of the short-term memory data of the previous time can be added to the long-term memory data. Using the new short-term memory data and the new long-term memory data, S110 to S140 and related steps are performed again to obtain the predicted emotional state of this time, and then the interaction strategy of the previous time is adjusted to obtain the interaction strategy of this time. For example, the interaction strategy of the previous time was a soft tone and comforting words, and the interaction strategy of this time is a calm tone and encouraging words.

[0140] According to the embodiments of the present disclosure, based on the emotional change state reflected by the feedback information, the interaction strategy can be flexibly adjusted to improve the rationality and flexibility of the interaction between the digital human and the interactive object.

[0141] Figure 5 FIG. 5 is a flow chart of a method 500 for training an emotion interaction model according to an embodiment of the present disclosure, the method comprising:

[0142] S510, inputting sample emotion data interacting with the digital human into the emotion interaction model to obtain sample emotion features;

[0143] S520, inputting the sample emotion feature into the short-term memory network and the long-term memory network of the emotion interaction model to obtain the sample short-term memory feature and the sample long-term memory feature;

[0144] S530: Training the emotion interaction model based on the short-term memory characteristics of the sample and the long-term memory characteristics of the sample.

[0145] In an embodiment of the present disclosure, the training samples of the emotion interaction model may include sample emotion data and its corresponding memory features (expected values). The sample emotion data may include sample short-term emotion data and sample long-term emotion data. The short-term memory network can extract sample short-term emotion features from the sample short-term emotion data, and the long-term memory network can extract sample long-term emotion features from the sample long-term emotion data. Sample short-term memory features can be obtained based on sample short-term emotion features using a memory network, and sample long-term memory features can be obtained based on sample long-term emotion features. Based on the sample short-term memory features and / or sample long-term memory features, the loss function value can be calculated, and then the emotion interaction model can be trained according to the loss function value. For example, the parameters of the short-term memory network and / or the long-term memory network in the emotion interaction model are adjusted according to the loss function value, and then the training samples are continued to be used for the next round of training until the loss function is reached.

[0146] According to the embodiments of the present disclosure, the emotional interaction model is trained based on the sample short-term memory characteristics and the sample long-term memory characteristics, so that the interaction strategy of the digital human generated by the emotional interaction model can be more suitable for the interaction object, and the interaction consistency and accuracy between the digital human and the interaction object can be improved.

[0147] In one embodiment, the sample emotion data includes sample short-term emotion data and sample long-term emotion data, and the priority of the sample long-term emotion data is determined based on at least one of the following: emotion intensity, time weight, modal credibility, memory relevance, and personalized characteristics. Among them, the priority of the sample long-term emotion data can refer to the relevant description of the priority in the above embodiment. For example, based on the priority of each emotion data in the stored historical emotion data, the long-term emotion data with high priority related to a certain short-term emotion data can be called, and the memory features are respectively marked to form a training sample.

[0148] According to the embodiments of the present disclosure, the priority of emotional data can be set by time, emotional intensity, etc., and long-term emotional data that is more relevant to short-term emotional data can be used to construct training samples, so that the emotional interaction model after training can generate more appropriate interaction strategies.

[0149] Figure 6 6 is a flow chart of a method 600 for training an emotion interaction model according to another embodiment of the present disclosure, and the method 600 can be used to implement step S510 in the method 500 for training an emotion interaction model. In one embodiment, the method 600 includes: inputting sample emotion data of interaction with a digital human into the emotion interaction model to obtain sample emotion features, and further includes:

[0150] S610, inputting the multimodal sample emotion data interacting with the digital human into the multimodal network of the emotion interaction model, and extracting sample features of different modalities;

[0151] S620, adjusting the weights of the sample features of the different modalities through the cross-modal attention mechanism of the multimodal network;

[0152] S630, obtaining the sample emotion feature based on the sample features of the different modalities and the adjusted weights thereof.

[0153] In the disclosed embodiments, the sample emotion data in the training sample may be multimodal sample emotion data. Sample features of different modalities may be extracted from the multimodal sample emotion data through a multimodal network. Sample features of different modalities may be fused through a cross-attention mechanism at the attention layer of the multimodal network. The fusion weights corresponding to sample features of different modalities may be adjusted. Based on the sample features of different modalities and the adjusted weights, a fusion calculation is performed to obtain the overall sample emotion features.

[0154] The multimodal sample emotion data may include multimodal sample short-term emotion data and multimodal sample long-term emotion data. Based on the multimodal sample short-term emotion data, sample short-term emotion features such as sample short-term emotion sequence features may be extracted, and based on the multimodal sample long-term emotion data, sample long-term emotion features such as sample long-term emotion sequence features may be extracted.

[0155] According to the embodiments of the present disclosure, through multimodal emotion capture technology, the digital human can fully and delicately understand the emotional state of the object through the trained emotion interaction model and provide more accurate emotional response.

[0156] Figure 7 It is a flow chart of a training method 700 of an emotion interaction model according to another embodiment of the present disclosure, and the method 700 can be used to implement step S520 in the training method 500 of the emotion interaction model. In one embodiment, the sample emotion feature includes a sample short-term emotion sequence feature and a sample long-term emotion sequence feature, and the method 700 includes: inputting the sample emotion feature into the short-term memory network and the long-term memory network of the emotion interaction model to obtain the sample short-term memory feature and the sample long-term memory feature, and further includes:

[0157] S710, inputting the short-term emotion sequence feature of the sample into the short-term emotion network of the emotion interaction model to obtain the short-term memory feature of the sample;

[0158] S720: Input the long-term emotion sequence features of the sample into the long-term emotion network of the emotion interaction model to obtain the long-term memory features of the sample.

[0159] In an embodiment of the present disclosure, sample emotional features may include sample short-term emotional features and sample long-term emotional features. Sample short-term emotional features may include sample short-term emotional sequence features, and sample long-term emotional features may include sample long-term emotional sequence features. After inputting the sample short-term emotional sequence features into the short-term emotional network, the short-term emotional network may output sample short-term memory features. Inputting the sample long-term emotional sequence features into the long-term emotional network, sample long-term memory features are obtained. According to an embodiment of the present disclosure, the emotional interaction model can memorize and manage the emotional memory of the interactive object, so that the digital human can maintain emotional continuity with the interactive object in multiple interactions through the emotional interaction model.

[0160] Figure 8 8 is a flow chart of a method 800 for training an emotion interaction model according to another embodiment of the present disclosure, and the method 800 can be used to implement step S530 in the method 500 for training an emotion interaction model. In one embodiment, the method 800 includes: training the emotion interaction model based on the sample short-term memory feature and the sample long-term memory feature, and further includes:

[0161] S810, obtaining short-term memory loss based on the short-term memory characteristics of the sample;

[0162] S820, obtaining long-term memory loss based on the long-term memory characteristics of the sample;

[0163] S830, obtaining an overall loss based on the short-term memory loss and the long-term memory loss;

[0164] S840: Training the short-term memory network and the long-term memory network of the emotion interaction model based on the overall loss.

[0165] In an embodiment of the present disclosure, in one method, short-term memory loss can be calculated based on sample short-term memory features and expected short-term memory features, and a short-term emotional network can be trained based on the short-term memory loss. In another method, long-term memory loss can be calculated based on sample long-term memory features and expected short-term memory features, and a long-term emotional network can be trained based on the long-term memory loss. In another method, overall loss can be calculated based on short-term memory loss and long-term memory loss. Short-term emotional networks and / or long-term emotional networks can be trained based on overall loss.

[0166] According to the disclosed embodiment, the emotional interaction model is trained through the loss related to emotional memory, so that the digital human can provide more reasonable and continuous interaction strategies through the emotional interaction model, and then control the digital human to perform more appropriate emotional responses.

[0167] In one embodiment, the training samples of the emotion interaction model include emotion switching samples, and the training weight of the emotion switching samples in the loss function is higher than the training weight of the non-emotion switching samples.

[0168] In the disclosed embodiment, the emotion switching sample can reflect a large emotion change, such as a change from positive to negative, or from negative to positive. Such samples are difficult samples and can be subjected to reinforcement learning and focused training. A higher training weight can be assigned to the switching sample, and a weighted loss function can be used to optimize the model. This can improve the utilization rate of difficult samples, and the trained model can more easily give a suitable interaction strategy for situations similar to difficult samples.

[0169] In one embodiment, the method further includes: fusing the sample short-term memory feature and the sample long-term memory feature to obtain the sample predicted emotional state.

[0170] In the disclosed embodiment, by means of memory fusion, such as a fusion algorithm, the sample short-term memory features and the sample long-term memory features can be fused to obtain the sample fused memory features, which can be converted into or directly used as the sample predicted emotional state. The emotional interaction model can generate an interactive strategy based on the sample predicted emotional state to control the emotional response performed by the digital human. Then, based on the feedback information for the emotional response, the emotional interaction model is trained using reinforcement learning and other methods. In this way, the emotional interaction model that can be trained can generate a more reasonable and coherent interaction strategy.

[0171] In one embodiment, the sample short-term memory feature and the sample long-term memory feature are fused to obtain the sample predicted emotional state, including:

[0172] Based on the sample emotional data, the sample short-term emotional needs and the sample long-term emotional needs are obtained;

[0173] Based on the sample short-term memory feature and the sample long-term memory feature, a sample short-term memory storage feature and a sample long-term memory storage feature are obtained;

[0174] Based on the sample short-term emotional needs and the sample short-term memory storage characteristics, the first matching degree between the sample short-term emotion and the sample short-term memory is obtained;

[0175] Based on the sample long-term emotional needs and the sample long-term memory storage characteristics, a second matching degree between the sample long-term emotion and the sample long-term memory is obtained;

[0176] Based on the first matching degree and the second matching degree, adjusting the weights of the sample short-term memory feature and the sample long-term memory feature;

[0177] Based on the weights of the short-term memory feature of the sample and the long-term memory feature of the sample, the long-term memory feature of the sample and the short-term memory feature of the sample are fused to obtain the predicted emotional state of the sample.

[0178] In the disclosed embodiment, not only sample emotion features but also sample memory feature requirements can be extracted from the sample emotion data. The sample memory feature requirements are generated based on the sample emotion data through the first transformation network in the emotion interaction model. For example, the first transformation network can extract the sample short-term memory feature requirements Q from the sample short-term emotion data. s , extract sample long-term memory feature demand Q from sample long-term sentiment data l Through the second transformation network in the emotional interaction model, based on sample memory features (such as sample short-term memory features M s and sample long-term memory feature M l ), can be transformed to obtain sample memory storage features (such as sample short-term memory storage features K sand sample long-term memory storage feature K l ).

[0179] Adopt Q s ·K s The first matching degree between the current emotional needs of the interaction object and the short-term memory can be calculated. For example, a higher first matching degree score indicates that the short-term memory is more important in predicting the emotional state. l ·K l The second matching degree between the current emotional needs of the interactive object and the long-term memory can be calculated. For example, a higher second matching degree score indicates that the long-term memory is more important in predicting the emotional state.

[0180] If the sample predicts a dramatic change in the emotional state (e.g., the emotion suddenly changes from pleasure to anger), the first matching degree Q corresponding to the sample's short-term memory s ·K s Higher, can increase the sample short-term memory feature M s The weight w s , mainly referring to short-term memory. If the sample predicts a relatively stable emotional state, it is necessary to understand the long-term emotional background (such as the long-term anxiety state of the interactive object). The second matching degree Q corresponding to the sample's long-term memory characteristics l ·K l Higher, can increase w l .

[0181] By Q s ·K s and Q l ·K l Score, calculate weight w s and w l Finally, the following formula can be used to finally fuse the sample short-term memory features and the sample long-term memory features: S = w s ·M s +w l ·M l , generating sample predicted affective states S, which are used as the basis for subsequent affective prediction, intervention, or memory updating.

[0182] According to the disclosed embodiment, the fusion weight of the sample short-term memory feature and the sample long-term memory feature can be more reasonably determined by the matching degree between memory and emotional needs. The emotional interaction model generates the interactive strategy of the digital human by fusing the sample predicted emotional state obtained by fusing the two memory features based on the fusion weight, so as to control the digital human to perform emotional response. Then, the emotional interaction model is further optimized through reinforcement learning, so that the emotional interaction model can generate an interactive strategy that better meets emotional needs.

[0183] In one embodiment, the method further includes: inputting the predicted emotional state of the sample into the interactive strategy network of the emotional interaction model to generate an interactive strategy for the digital person; the interactive strategy is used to control the emotional response of the digital person; receiving feedback information regarding the emotional response of the digital person; and performing reinforcement learning training on the interactive strategy network based on the feedback information.

[0184] In the disclosed embodiment, the interactive strategy network of the emotional interaction model can generate the interactive strategy of the digital human based on the fused sample predicted emotional state to control the digital human to perform emotional response. Then, feedback information can be obtained according to the feedback action performed by the interactive object, and then the interactive strategy network can be reinforced learning training based on the feedback information to further optimize the emotional interaction model, so that the trained emotional interaction model can flexibly give continuous and appropriate interactive strategies according to the feedback of the interactive object.

[0185] In one embodiment, reinforcement learning training is performed on the interactive strategy network based on the feedback information, including: obtaining a change in emotion intensity and a feedback score based on the feedback information; obtaining a reward value of a reward function based on the change in emotion intensity and the feedback score; and optimizing the interactive strategy network using a proximal strategy optimization algorithm based on the reward value.

[0186] In the disclosed embodiment, the change in emotional intensity, feedback score, etc. can be determined based on the feedback information of the interactive object. The reward value of the reward function can be calculated based on the change in emotional intensity and action space, as well as the feedback score and state space. Among them, the action space can include optional actions such as adjusting the tone of voice, providing comforting words, providing encouraging words, singing, etc. The state space can include emotional intensity, fluctuation trend, and modal characteristics. An example formula of a reward function is as follows:

[0187] R = α·(ΔS emotion) + β·(user satisfaction)

[0188] Among them, ΔS emotion represents: the change in the emotional intensity of the interactive object before and after the digital human intervention. User satisfaction can be equal to the user feedback score or calculated based on the user feedback score.

[0189] Then, the Proximal Policy Optimization (PPO) algorithm can be used to optimize the interactive policy network, and the first transformation network, the second transformation network, etc. can also be optimized. An example formula of a PPO algorithm is as follows:

[0190]

[0191] Among them, π θ (a|s) is a probability distribution function, which indicates the probability of executing action a in state s, and θ is the parameter of the policy network; Rt represents the immediate reward. The PPO algorithm maximizes Learn better emotional intervention strategies (interaction strategies) to maximize the improvement of the emotional state of the interactive object and improve user satisfaction.

[0192] The control method of the digital human in the embodiment of the present disclosure can be used in a digital human system (which may be referred to as a digital human emotion memory and management system) that can capture, memorize, manage and dynamically apply user emotions.

[0193] The system may have one or more of the following features:

[0194] 1. Emotional interaction: Users hope that the interaction with digital humans is not only the transmission of information, but also an emotional exchange. Through this system, digital humans should be able to perceive the user's emotional state and respond in accordance with emotional needs.

[0195] 2. Personalized experience: Each user’s emotional responses and psychological needs are unique. Through this system, the digital human should be able to remember the user’s emotional history and refer to these memories in future interactions to provide a more personalized experience.

[0196] 3. Continuous support: In scenarios such as psychological counseling, health management, and social interaction, through this system, the digital human can remember and pay attention to the user's emotional changes, provide continuous emotional support and advice, and help users cope with emotional fluctuations or stress.

[0197] 4. Emotion prediction and prevention: Through the accumulation and analysis of emotional data, the system should be able to predict users’ possible emotional reactions and adjust interaction strategies in advance to prevent the generation or expansion of negative emotions.

[0198] In summary, the digital human emotional memory and management system can provide personalized, emotional and intelligent interactive experience by deeply capturing the user's emotional state, meet the user's needs for emotional communication and emotional support, and thus enhance the practicality and user stickiness of the digital human.

[0199] A digital human emotion memory and management system may include the following modules:

[0200] 1. Dynamic memory and priority management:

[0201] An example of an implementation: The system includes a memory module that can record the user's emotional state in each interaction and set priorities by time and emotional intensity. By combining long-term memory with short-term memory, the system can dynamically call relevant memories in different scenarios to ensure the emotional continuity of the interaction.

[0202] The system can not only statically call up historical records, but also dynamically adjust the priority of memory calls based on the user's current situation. For example, after a user has long shown aversion to a certain topic, the system can automatically prioritize related memories and set the digital person's emotional response behavior to avoid mentioning the topic, thus avoiding triggering similar topics or situations again.

[0203] 2. Emotional prediction and preventive intervention:

[0204] An example of an implementation method: Through the accumulation and analysis of emotional data, the system can predict the trend of user's emotional changes. For example, when it detects that the user's mood is gradually declining during multiple interactions, the system will take intervention measures in advance, such as adjusting the content of the interaction, providing encouragement, or suggesting relaxing activities. The system uses reinforcement learning algorithms to predict emotional data, learn the rules of user's emotional changes, and make appropriate responses before emotional reactions occur.

[0205] Compared with passive emotion recognition, the emotion prediction module enables digital humans to actively participate in emotion management, transforming from reactive interaction to proactive interaction, enhancing users' emotional experience and trust in digital humans.

[0206] 3. Personalized emotional experience optimization and feedback loop:

[0207] An example of an implementation method: The system provides personalized interaction optimization suggestions based on the user's emotional response, and continuously improves the emotional performance of the digital human through a feedback loop. For example, the system can adjust the digital human's tone, expression, and behavior based on the user's feedback on the interaction to ensure that the emotional response is consistent with the user's expectations.

[0208] Different from one-way emotional interaction, the emotional feedback loop of this system allows the user's feedback to directly affect the performance of the digital human, forming a two-way interaction. This dynamic adjustment mechanism significantly improves the level of personalization of the user experience and makes the emotional response of the digital human more intimate and natural.

[0209] An example of steps for one application scenario is as follows:

[0210] Scenario description: A user shows anxiety about a topic during an interaction with a digital human. The system captures the user's nervous facial expression, trembling voice, and negative emotional words in the text. The system records this emotional data and avoids directly mentioning similar topics in future interactions.

[0211] Step 1: Emotional data capture: The user inputs text and interacts with the voice, and the system obtains multimodal emotional data through the camera and microphone.

[0212] Step 2: Sentiment analysis and memory: The sentiment data is input into the fusion algorithm. After analysis, the system records the user’s emotional state and sets priorities.

[0213] Step 3: Prediction and intervention: The system detects that the user's emotional fluctuations tend to be negative in several consecutive interactions, and takes emotional soothing measures in advance, such as soft tone and encouraging words.

[0214] Step 4: Feedback and optimization: The user shows positive feedback to the digital human’s comfort. The system records this information and gives priority to this interaction method in similar situations in the future.

[0215] The following describes the characteristics of the digital human emotion memory and management system of the embodiment of the present disclosure from several aspects such as model structure, loss function, and training samples.

[0216] 1. A more efficient multimodal fusion framework (such as Cross-Attention or Perceiver model) can be used in the model structure.

[0217] Due to the heterogeneity between multimodal data (voice, expression, text), the use of cross-modal attention mechanism can improve the complementarity between modalities. In addition, the cross-modal attention mechanism can better capture the collaborative information of multimodal data by dynamically adjusting the weights between modalities.

[0218] An example of a cross-modal attention mechanism: using voice, emoji, and text as query, key, and value inputs to the attention layer. The formula is as follows:

[0219]

[0220] Among them, Q, K, V can represent feature embeddings of different modalities; d k Can represent the dimension of the key, used for scaling.

[0221] Example of multimodal fusion: Using the features of different modal data F speech, F expression, F text, fusion formula example: F fusion = Attention (F speech, F expression, F text)

[0222] The global features after multimodal fusion are used for classification or regression. In addition, cross-modal attention dynamically adjusts the mutual information weights between modalities, which can enable the model to capture multimodal emotional signals more accurately and improve the ability to handle complex emotional signals (such as modal conflicts).

[0223] Second, more memory mechanisms can be introduced into the model structure to provide a reference for long-term sentiment trend prediction.

[0224] For example, adding short-term and long-term memory modules (or memory networks) to handle real-time sentiment fluctuations and historical sentiment trends, respectively.

[0225] (1) Short-term memory is used for real-time emotional dynamic analysis, such as the emotional fluctuations of the user in the current session or the last three sessions. For example, a recurrent neural network (such as a gated recurrent unit (GRU) or a long short-term memory network (Long-Short Term Memory, LSTM)) is used to process real-time emotional data and extract the correlation between time steps. An example is as follows:

[0226] Input of the short-term memory module: short-term emotion sequence is X s .

[0227] Output of the short-term memory module: short-term memory is M s .

[0228] (2) Long-term memory captures the user's emotional history, such as long-term anxiety tendencies. For example, another GRU module is used to combine the user's long-term emotional history to extract long-term trends. An example is as follows:

[0229] Input of the long-term memory module: long-term sentiment sequence is X l .

[0230] Output of the long-term memory module: long-term memory is M l .

[0231] (3) Memory Fusion:

[0232] An example of a fusion formula: S = w s ·M s +w l ·M l .

[0233] Among them, the weights of short-term memory and long-term memory are w s 、w l The attention mechanism is used to learn and dynamically adjust the weights of short-term memory and long-term memory. The weights are calculated based on the contextual importance of the current sentiment data. For example:

[0234]

[0235] Parameter Description:

[0236] Q s : Feature demands related to short-term memory in affective states.

[0237] Q l : Feature demands related to long-term memory in affective states.

[0238] The above vectors are generated by passing the current emotional data (such as speech, expression and text features) through a transformation network.

[0239] K s : Storage characteristics of short-term memory.

[0240] K l : Storage characteristics of long-term memory.

[0241] The above vectors can be obtained by the output features of the short-term or long-term memory module (such as M s and M l ) is transformed.

[0242] Q s ·K s : The degree of match between current emotional needs and short-term memory (the higher the score, the more important short-term memory is in predicting emotional state).

[0243] Q l ·K l : The degree of match between current emotional needs and long-term memory (higher scores indicate greater importance of long-term memory in predicting emotional states).

[0244] Application process description:

[0245] 1. If the predicted emotional state fluctuates dramatically (e.g., emotion suddenly changes from pleasure to anger), the matching degree of short-term memory Q s ·K s Higher, the system will increase w s , mainly referring to short-term memory.

[0246] 2. If the predicted emotional state is relatively stable, it is necessary to understand the long-term emotional background (such as the user's long-term anxiety state) and the matching degree of long-term memory Q l ·K l Higher, the system will increase w l .

[0247] 3. When the system analyzes the current user's emotional state, s ·K s and Q l ·K l Score, calculate weight w s and w l , and finally integrate short-term and long-term memory: S = w s ·M s +w l ·M l , generating a predicted affective state S, which is used as the basis for subsequent affective prediction, intervention, or memory updating.

[0248] Short-term memory formula:

[0249] M s =GRU(X s , M s (t-1) )

[0250] Long-term memory formula:

[0251] M l =GRU(X l , M l (t-1) )

[0252] Attention weight formula:

[0253]

[0254] Emotional state fusion formula: S = w s ·M s +w l ·M l , where: short-term memory is used for real-time dynamic analysis of emotions, long-term memory is used as a reference for historical trends of emotions, and weight adjustment ensures that it can adapt to the needs of different scenarios. After the improvement, the strategy can more accurately capture short-term fluctuations and long-term trends, and improve the robustness and continuity of emotion prediction. Ensure the adaptability of the model in different situations, for example, real-time interaction gives priority to short-term memory, and emotion trend analysis gives priority to long-term memory.

[0255] 3. Model loss function

[0256] A reward mechanism based on user satisfaction is introduced, and prediction accuracy and intervention effect are jointly included in the optimization objectives.

[0257] Dynamic weight adjustment of multi-task losses: Since the loss functions of short-term fluctuation detection and long-term trend prediction may conflict, the weights can be adjusted dynamically.

[0258] 1. Loss function formula: L 总 =λ t ·L 短期 +(1-λ t )·L 长期 ;

[0259] Where: t is a dynamic weight that changes over time; L 总 Represents the total loss; L 短期 Indicates short-term loss; L 长期 Indicates long-term losses.

[0260] The loss function can be calculated using either classification loss or regression loss, depending on the task type.

[0261] L 短期The calculation method is as follows:

[0262] (1) Classification task:

[0263]

[0264] Among them, y 短期，c Represents: the true emotion category label of the current interaction (such as anxiety); Represents: short-term memory module M s The predicted sentiment category probabilities of the output.

[0265] (2) Regression task:

[0266]

[0267] y 短期 Represents: The true sentiment intensity label.

[0268] L 长期 The calculation method is as follows:

[0269] (1) Classification task:

[0270]

[0271] y 长期，c Represents: Long-term sentiment category labels.

[0272] Represents: The predicted emotion category probability output by the long-term memory module Ml.

[0273] (2) Regression task:

[0274]

[0275] y 长期 Representation: Ground-truth labels for long-term affective states.

[0276] The fused memory S is the weighted output of short-term memory and long-term memory: S = w s ·M s +w l ·M l .

[0277] If the final goal is to predict or intervene in sentiment based on the fused memory output S, then the loss of the fused memory needs to be calculated separately. If the fused memory is only an intermediate state and the final optimization goal is to improve the quality of short-term and long-term memory respectively, there is no need to calculate additional loss for S.

[0278] If you need to calculate the fusion loss, the formula is as follows:

[0279] (1) Classification task:

[0280]

[0281] y c Representation: The true label of the sentiment category.

[0282] Represents: The probability of sentiment category output by the fused memory S.

[0283] (2) Regression task:

[0284]

[0285] Among them, y represents: the true sentiment intensity label.

[0286] 2. Dynamic adjustment rules:

[0287] Adjust the weight based on the task gradient size, and adjust the λ t See the formula:

[0288]

[0289] in, It can represent the adjustment gradient. This formula can balance the optimization of short-term and long-term tasks through dynamic weights and improve the overall performance.

[0290] 4. Add a difficult sample mining mechanism to balance short-term and long-term task samples.

[0291] Emotional switching samples (such as from positive to negative) are difficult to capture, and such samples require reinforcement learning and focused training.

[0292] 1. Sample weight formula:

[0293] Assign higher training weights to switching samples:

[0294]

[0295] ΔS i Indicates: the change value of the sample's sentiment intensity.

[0296] ∈ means: prevent the denominator from being zero.

[0297] 2. Training process:

[0298] Optimize the model using a weighted loss function:

[0299]

[0300] Among them, L i represents the loss value of the i-th sample, w iRepresents the weight of the i-th sample. Emotion switching samples can be trained intensively, which helps improve the model's ability to recognize sudden emotional changes.

[0301] V. Intervention Strategy Training Process

[0302] Optimizing intervention strategies for affective fluctuations using reinforcement learning.

[0303] The effect of the intervention strategy needs to be adjusted in real time according to the changes in user emotions, but traditional supervised learning cannot dynamically optimize the intervention strategy. Reinforcement learning can learn the optimal strategy through the reward function. After selecting intervention measures (such as tone adjustment or text prompts), the model updates the strategy based on user feedback.

[0304] State space: Define state s: including sentiment intensity, fluctuation trend and modal characteristics.

[0305] Action space: Define actions α: such as adjusting tone of voice, providing comforting text, etc.

[0306] An example reward function is as follows:

[0307] R = α·(ΔS emotion) + β·(user satisfaction)

[0308] Among them, ΔS emotion represents: the change in emotion intensity before and after intervention. User satisfaction represents: user feedback score. R: The reward value represents the comprehensive effect of the system in the current emotion intervention behavior, which is used for strategy optimization of reinforcement learning. The model learns a better emotion intervention strategy by maximizing R.

[0309] If ΔS emotion > 0, it means that the intensity of emotion is weakened (for example, anxiety is reduced) and the intervention effect is positive; if ΔS emotion < 0, the opposite is true. α represents the contribution of the change in emotion intensity to the reward value R. If the goal of the system is mainly to reduce the user's negative emotions (such as anxiety and anger), α>β can be set to give priority to the change in emotion intensity.

[0310] β: indicates the contribution of user satisfaction to the reward value R. If the system focuses on the direct satisfaction of user feedback (such as whether the soothing tone is effective), β>α can be set.

[0311] Policy Optimization: Use the PPO algorithm to optimize the policy network

[0312]

[0313] Description: π θ (a|s) is a probability distribution function that represents the probability of executing action a in state s, and θ is the parameter of the policy network.

[0314] s (state) represents: the user's current emotional state, including short-term fluctuations, long-term trends, and emotional features such as voice, expression, and text.

[0315] a (action) means: optional emotional intervention behaviors, such as adjusting tone of voice, providing comforting texts, recommending relaxation activities, etc.

[0316] R t (Instant reward) means: Combine the change in sentiment intensity and user satisfaction to calculate: R t =a·(ΔS sentiment)+β·(user satisfaction).

[0317] The algorithm maximizes Learn better emotion intervention strategies to maximize the improvement of users’ emotional states and improve user satisfaction. Furthermore, the real-time and effectiveness of emotion intervention can be improved.

[0318] The digital human control method of the disclosed embodiment can be used in a digital human interactive system, and can include a digital human emotion memory and management system, and has one or more of the following advantages:

[0319] 1. Deep emotional understanding and resonance:

[0320] The system uses multimodal emotion capture technology to fully and delicately understand the user's emotional state and provide more accurate emotional responses. Not only can it capture surface emotional signals, but the emotional memory and management system can also comprehensively analyze the user's emotional dimensions and enhance the emotional resonance of the interaction.

[0321] 2. Continuity and emotional consistency:

[0322] The system can remember and manage the user's emotional history to ensure emotional consistency in multiple interactions. This continuity allows users to feel the digital human's attention and understanding of their emotional state, enhancing the user's sense of trust and the coherence of the interactive experience.

[0323] 3. Dynamic response and personalized adjustment:

[0324] The system supports dynamic emotional response and personalized adjustment, and can adjust the interaction mode in real time according to the user's emotional changes. The emotional memory and management system makes the interaction more flexible and adaptable through feedback loops and intelligent adjustments to meet the personalized needs of different users.

[0325] 4. Active emotion management and prediction:

[0326] The system not only responds to the user's emotions passively, but can also actively participate in emotion management through emotion prediction technology, and can conduct preventive intervention before emotional problems occur, helping users maintain a positive emotional state. This active management capability can transform digital humans from mere interactors into emotional supporters, providing users with more in-depth emotional services.

[0327] In summary, the digital human emotional memory and management system has significantly improved the level of digital human emotional interaction through innovative emotion capture, memory management, dynamic response and prediction technologies, providing users with a more natural, personalized and emotionally rich interactive experience. It not only solves the shortcomings of traditional interactive systems, but also opens up new application prospects for digital humans in the field of emotional support.

[0328] Fig. 9 is a structural schematic diagram of a digital human control device 900 according to an embodiment of the present disclosure. The device 900 may include:

[0329] The emotion feature acquisition module 910 is used to obtain the emotion feature based on the emotion data of the interaction with the digital human;

[0330] A memory feature acquisition module 920, used to obtain short-term memory features and long-term memory features based on the emotion features;

[0331] A memory feature fusion module 930, used to fuse the short-term memory feature and the long-term memory feature to obtain a predicted emotional state;

[0332] The interactive strategy generation module 940 is used to generate an interactive strategy for the digital human based on the predicted emotional state; wherein the interactive strategy is used to control the emotional response of the digital human.

[0333] In one embodiment, the emotion data includes short-term emotion data and long-term emotion data, and the priority of the long-term emotion data is determined based on at least one of the following: emotion intensity, time weight, modality credibility, memory relevance, and personalized characteristics.

[0334] Fig.10 1 is a schematic diagram of a digital human control device 1000 according to another embodiment of the present disclosure, the device 1000 includes: an emotion feature acquisition module 1010, a memory feature acquisition module 1020, a memory feature fusion module 1030 and an interaction strategy generation module 1040. The functions of the above modules can refer to the functions of the modules of the digital human control device in the above embodiment. In one implementation, the emotion feature acquisition module 1010 may include:

[0335] An extraction submodule 1011 is used to extract features of different modalities from multimodal emotion data interacting with a digital human through a multimodal network of an emotion interaction model;

[0336] A weight adjustment submodule 1012, configured to adjust the weights of the features of the different modalities through a cross-modal attention mechanism of the multimodal network;

[0337] The fusion submodule 1013 is used to obtain the emotion feature based on the features of the different modalities and their adjusted weights.

[0338] In one implementation, the emotion feature includes a short-term emotion sequence feature and a long-term emotion sequence feature, and the memory feature acquisition module 1020 includes:

[0339] The short-term feature acquisition submodule 1021 is used to input the short-term emotion sequence feature into the short-term emotion network of the emotion interaction model to obtain the short-term memory feature;

[0340] The long-term feature acquisition submodule 1022 is used to input the long-term emotion sequence feature into the long-term emotion network of the emotion interaction model to obtain the long-term memory feature.

[0341] In one implementation, the memory feature fusion module 1030 includes:

[0342] The demand acquisition submodule 1031 is used to obtain short-term memory feature requirements and long-term memory feature requirements based on the emotion data;

[0343] A storage feature acquisition submodule 1032, configured to obtain a short-term memory storage feature and a long-term memory storage feature based on the short-term memory feature and the long-term memory feature;

[0344] The short-term matching submodule 1033 is used to obtain a first matching degree between the short-term emotion and the short-term memory based on the short-term memory feature requirement and the short-term memory storage feature;

[0345] The long-term matching submodule 1034 is used to obtain a second matching degree between the long-term emotion and the long-term memory based on the long-term memory feature requirement and the long-term memory storage feature;

[0346] A weight determination submodule 1035, configured to determine the weights of the short-term memory feature and the long-term memory feature based on the first matching degree and the second matching degree;

[0347] The feature fusion submodule 1036 is used to fuse the short-term memory feature and the long-term memory feature to obtain the predicted emotional state based on the weights of the short-term memory feature and the long-term memory feature.

[0348] In one implementation, the interaction strategy generation module 1040 is used to input the predicted emotional state into the interaction strategy network of the emotional interaction model to obtain the interaction strategy.

[0349] In one embodiment, it further includes:

[0350] The receiving module 1050 is used to receive feedback information on the emotional response of the digital human;

[0351] A processing module 1060 is used to obtain an emotion change state based on the feedback information through the interactive strategy network;

[0352] The adjustment module 1070 is used to adjust the interaction strategy based on the emotion change state through the interaction strategy network.

[0353] Fig.11 1 is a schematic diagram of a structure of a training device 1100 for an emotion interaction model according to an embodiment of the present disclosure. The device 1100 may include:

[0354] The emotion feature acquisition module 1110 is used to input the sample emotion data interacting with the digital human into the emotion interaction model to obtain the sample emotion feature;

[0355] A memory feature acquisition module 1120 is used to input the sample emotion feature into the short-term memory network and the long-term memory network of the emotion interaction model to obtain the sample short-term memory feature and the sample long-term memory feature;

[0356] The training module 1130 is used to train the emotion interaction model based on the sample short-term memory characteristics and the sample long-term memory characteristics.

[0357] In one embodiment, the sample emotion data includes sample short-term emotion data and sample long-term emotion data, and the priority of the sample long-term emotion data is determined based on at least one of the following: emotion intensity, time weight, modality credibility, memory relevance, and personalized characteristics.

[0358] Fig.12 1 is a schematic diagram of a training device 1200 for an emotion interaction model according to another embodiment of the present disclosure. The device 1200 includes: an emotion feature acquisition module 1210, a memory feature acquisition module 1220, and a training module 1230. The functions of the above modules can refer to the functions of the modules of the training device for the emotion interaction model in the above embodiment. In one embodiment, the emotion feature acquisition module 1210 may include:

[0359] The extraction submodule 1211 is used to input the multimodal sample emotion data interacting with the digital human into the multimodal network of the emotion interaction model, and extract sample features of different modalities;

[0360] An adjustment submodule 1212, configured to adjust the weights of the sample features of the different modalities through a cross-modal attention mechanism of the multimodal network;

[0361] The fusion submodule 1213 is used to obtain the sample emotion feature based on the sample features of the different modalities and the adjusted weights thereof.

[0362] In one implementation, the sample emotion feature includes a sample short-term emotion sequence feature and a sample long-term emotion sequence feature, and the memory feature acquisition module 1220 includes:

[0363] The short-term feature acquisition submodule 1221 is used to input the short-term emotion sequence feature of the sample into the short-term emotion network of the emotion interaction model to obtain the short-term memory feature of the sample;

[0364] The long-term feature acquisition submodule 1222 is used to input the long-term emotion sequence feature of the sample into the long-term emotion network of the emotion interaction model to obtain the long-term memory feature of the sample.

[0365] In one embodiment, the training module 1230 includes:

[0366] A short-term loss submodule 1231 is used to obtain a short-term memory loss based on the short-term memory feature of the sample;

[0367] The long-term loss submodule 1232 obtains the long-term memory loss based on the long-term memory characteristics of the sample;

[0368] An overall loss submodule 1233, used to obtain an overall loss based on the short-term memory loss and the long-term memory loss;

[0369] The training submodule 1234 is used to train the short-term memory network and the long-term memory network of the emotion interaction model based on the overall loss.

[0370] In one embodiment, the training samples of the emotion interaction model include emotion switching samples, and the training weight of the emotion switching samples in the loss function is higher than the training weight of the non-emotion switching samples.

[0371] In one embodiment, it also includes: a memory feature fusion module 1240, which is used to fuse the sample short-term memory feature and the sample long-term memory feature to obtain the sample predicted emotional state.

[0372] In one implementation, the memory feature fusion module 1240 includes:

[0373] The emotional demand submodule 1241 is used to obtain the sample short-term emotional demand and the sample long-term emotional demand based on the sample emotional data;

[0374] The storage feature submodule 1242 is used to obtain a sample short-term memory storage feature and a sample long-term memory storage feature based on the sample short-term memory feature and the sample long-term memory feature;

[0375] The short-term matching submodule 1243 is used to obtain a first matching degree between the sample short-term emotion and the sample short-term memory based on the sample short-term emotion demand and the sample short-term memory storage characteristics;

[0376] The long-term matching submodule 1244 obtains a second matching degree between the sample long-term emotion and the sample long-term memory based on the sample long-term emotion demand and the sample long-term memory storage characteristics;

[0377] A weight adjustment submodule 1245 adjusts the weights of the sample short-term memory feature and the sample long-term memory feature based on the first matching degree and the second matching degree;

[0378] The feature fusion submodule 1246 fuses the long-term memory feature of the sample with the short-term memory feature of the sample based on the weights of the short-term memory feature of the sample and the long-term memory feature of the sample to obtain the predicted emotional state of the sample.

[0379] In one embodiment, it further includes:

[0380] Interaction strategy generation module 1250, used to input the predicted emotional state of the sample into the interaction strategy network of the emotional interaction model to generate an interaction strategy for the digital human; the interaction strategy is used to control the emotional response of the digital human;

[0381] The receiving module 1260 is used to receive feedback information on the emotional response of the digital person;

[0382] The reinforcement learning training module 1270 is used to perform reinforcement learning training on the interactive strategy network based on the feedback information.

[0383] In one embodiment, the reinforcement learning training module 1270 includes:

[0384] A processing submodule 1271 is used to obtain a change in emotion intensity and a feedback score based on the feedback information;

[0385] A reward function submodule 1272, used to obtain a reward value of a reward function based on the emotion intensity change and the feedback score;

[0386] The optimization submodule 1273 is used to optimize the interactive strategy network based on the reward value using a proximal strategy optimization algorithm.

[0387] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0388] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0389] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0390] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0391] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0392] A number of components in the device 1300 are connected to the I / O interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0393] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as the control method of the digital human and / or the training method of the emotional interaction model. For example, in some embodiments, the control method of the digital human and / or the training method of the emotional interaction model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the control method of the digital human and / or the training method of the emotional interaction model described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to execute the control method of the digital human and / or the training method of the emotional interaction model in any other appropriate manner (for example, by means of firmware).

[0394] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0395] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0396] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0397] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0398] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0399] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0400] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0401] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for controlling a digital human, comprising: Get emotional features based on the emotional data of interaction with digital humans; Obtaining short-term memory features and long-term memory features based on the emotional features; Fusing the short-term memory feature and the long-term memory feature to obtain a predicted emotional state; An interaction strategy of the digital human is generated based on the predicted emotional state; wherein the interaction strategy is used to control the emotional response of the digital human.

2. The method according to claim 1, wherein: The emotional data includes short-term emotional data and long-term emotional data, and the priority of the long-term emotional data is determined based on at least one of the following: emotional intensity, time weight, modal credibility, memory relevance, and personalized characteristics.

3. The method according to claim 1 or 2, wherein: Emotional features are obtained based on the emotional data of interaction with digital humans, including: Extracting features of different modalities from multimodal emotional data of interaction with digital humans through the multimodal network of the emotional interaction model; Adjusting the weights of the features of the different modalities through the cross-modal attention mechanism of the multimodal network; The emotional feature is obtained based on the fusion of the features of the different modalities and their adjusted weights.

4. The method according to claim 3, wherein: The emotional features include short-term emotional sequence features and long-term emotional sequence features. The short-term memory features and long-term memory features are obtained based on the emotional features, including: Inputting the short-term emotion sequence features into the short-term emotion network of the emotion interaction model to obtain the short-term memory features; The long-term emotion sequence features are input into the long-term emotion network of the emotion interaction model to obtain the long-term memory features.

5. The method according to claim 4, wherein: The short-term memory feature and the long-term memory feature are combined to obtain a predicted emotional state, comprising: Based on the emotion data, obtaining short-term memory feature requirements and long-term memory feature requirements; Based on the short-term memory feature and the long-term memory feature, a short-term memory storage feature and a long-term memory storage feature are obtained; Based on the short-term memory feature requirements and short-term memory storage features, the first matching degree between short-term emotions and short-term memory is obtained; Based on the long-term memory feature requirements and long-term memory storage features, the second matching degree between long-term emotions and long-term memory is obtained; Determining weights of the short-term memory feature and the long-term memory feature based on the first matching degree and the second matching degree; Based on the weights of the short-term memory feature and the long-term memory feature, the short-term memory feature and the long-term memory feature are fused to obtain the predicted emotional state.

6. The method according to any one of claims 3 to 5, generating an interaction strategy of a digital human based on the predicted emotional state, comprising: The predicted emotional state is input into the interaction strategy network of the emotional interaction model to obtain the interaction strategy.

7. The method according to claim 6, further comprising: receiving feedback information regarding the emotional response of the digital person; Obtaining the emotional change state based on the feedback information through the interactive strategy network; The interaction strategy is adjusted based on the emotion change state through the interaction strategy network.

8. A method for training an emotional interaction model, comprising: Inputting sample emotion data of interaction with the digital human into the emotion interaction model to obtain sample emotion features; Inputting the sample emotion feature into the short-term memory network and the long-term memory network of the emotion interaction model to obtain the sample short-term memory feature and the sample long-term memory feature; The emotion interaction model is trained based on the sample short-term memory features and the sample long-term memory features.

9. The method according to claim 8, wherein: The sample emotion data includes sample short-term emotion data and sample long-term emotion data, and the priority of the sample long-term emotion data is determined based on at least one of the following: emotion intensity, time weight, modal credibility, memory relevance, and personalized characteristics.

10. The method according to claim 8 or 9, wherein: The sample emotion data of the interaction with the digital human is input into the emotion interaction model to obtain the sample emotion features, including: Inputting the multimodal sample emotion data of the interaction with the digital human into the multimodal network of the emotion interaction model to extract sample features of different modalities; Adjusting the weights of the sample features of the different modalities through the cross-modal attention mechanism of the multimodal network; The sample emotion feature is obtained based on the fusion of the sample features of the different modalities and the adjusted weights thereof.

11. The method according to claim 10, wherein: The sample emotion features include sample short-term emotion sequence features and sample long-term emotion sequence features. The sample emotion features are input into the short-term memory network and the long-term memory network of the emotion interaction model to obtain the sample short-term memory features and the sample long-term memory features, including: Inputting the sample short-term emotion sequence features into the short-term emotion network of the emotion interaction model to obtain the sample short-term memory features; The sample long-term emotion sequence features are input into the long-term emotion network of the emotion interaction model to obtain the sample long-term memory features.

12. The method according to any one of claims 8 to 11, wherein: The emotional interaction model is trained based on the sample short-term memory feature and the sample long-term memory feature, including: Based on the short-term memory characteristics of the sample, short-term memory loss is obtained; Based on the long-term memory characteristics of the sample, obtaining long-term memory loss; Based on the short-term memory loss and the long-term memory loss, an overall loss is obtained; The short-term memory network and the long-term memory network of the emotion interaction model are trained based on the overall loss.

13. The method according to any one of claims 8 to 11, wherein: The training samples of the emotion interaction model include emotion switching samples, and the training weight of the emotion switching samples in the loss function is higher than the training weight of the non-emotion switching samples.

14. The method according to any one of claims 8 to 13, further comprising: The sample short-term memory feature and the sample long-term memory feature are fused to obtain the sample predicted emotional state.

15. The method according to claim 14, wherein: The sample short-term memory feature and the sample long-term memory feature are combined to obtain the sample predicted emotional state, including: Based on the sample emotion data, obtaining the sample short-term emotion demand and the sample long-term emotion demand; Based on the sample short-term memory feature and the sample long-term memory feature, a sample short-term memory storage feature and a sample long-term memory storage feature are obtained; Based on the sample short-term emotional needs and the sample short-term memory storage characteristics, the first matching degree between the sample short-term emotion and the sample short-term memory is obtained; Based on the sample long-term emotional needs and the sample long-term memory storage characteristics, a second matching degree between the sample long-term emotion and the sample long-term memory is obtained; Based on the first matching degree and the second matching degree, adjusting the weights of the sample short-term memory feature and the sample long-term memory feature; Based on the weights of the sample short-term memory feature and the sample long-term memory feature, the sample long-term memory feature and the sample short-term memory feature are fused to obtain the predicted emotional state of the sample.

16. The method according to claim 14 or 15, further comprising: Inputting the predicted emotional state of the sample into the interactive strategy network of the emotional interaction model to generate an interactive strategy for the digital human; The interaction strategy is used to control the emotional response of the digital person; receiving feedback information regarding the emotional response of the digital person; Reinforcement learning training is performed on the interactive strategy network based on the feedback information.

17. The method according to claim 16, wherein: Performing reinforcement learning training on the interactive strategy network based on the feedback information includes: Based on the feedback information, obtaining a change in emotion intensity and a feedback score; Obtaining a reward value of a reward function based on the emotion intensity change and the feedback score; Based on the reward value, the interactive policy network is optimized using a proximal policy optimization algorithm.

18. A control device for a digital human, comprising: An emotion feature acquisition module, used to obtain emotion features based on the emotion data of the interaction with the digital human; A memory feature acquisition module, used to obtain short-term memory features and long-term memory features based on the emotional features; A memory feature fusion module, used to fuse the short-term memory feature and the long-term memory feature to obtain a predicted emotional state; An interactive strategy generation module is used to generate an interactive strategy for the digital human based on the predicted emotional state; wherein the interactive strategy is used to control the emotional response of the digital human.

19. The device according to claim 18, wherein: The emotional data includes short-term emotional data and long-term emotional data, and the priority of the long-term emotional data is determined based on at least one of the following: emotional intensity, time weight, modal credibility, memory relevance, and personalized characteristics.

20. The device according to claim 18 or 19, wherein: The emotion feature acquisition module comprises: An extraction submodule, used to extract features of different modalities from multimodal emotion data interacting with a digital human through a multimodal network of an emotion interaction model; A weight adjustment submodule, used to adjust the weights of the features of the different modalities through the cross-modal attention mechanism of the multimodal network; The fusion submodule is used to obtain the emotional feature based on the features of the different modalities and their adjusted weights.

21. The device according to claim 20, wherein: The emotional features include short-term emotional sequence features and long-term emotional sequence features, and the memory feature acquisition module includes: A short-term feature acquisition submodule, used for inputting the short-term emotion sequence feature into the short-term emotion network of the emotion interaction model to obtain the short-term memory feature; The long-term feature acquisition submodule is used to input the long-term emotion sequence features into the long-term emotion network of the emotion interaction model to obtain the long-term memory features.

22. The device according to claim 21, wherein The memory feature fusion module comprises: A demand acquisition submodule, used to obtain short-term memory feature requirements and long-term memory feature requirements based on the emotion data; A storage feature acquisition submodule, used to obtain a short-term memory storage feature and a long-term memory storage feature based on the short-term memory feature and the long-term memory feature; A short-term matching submodule, used to obtain a first matching degree between short-term emotion and short-term memory based on short-term memory feature requirements and short-term memory storage features; A long-term matching submodule, for obtaining a second matching degree between long-term emotion and long-term memory based on long-term memory feature requirements and long-term memory storage features; A weight determination submodule, used to determine the weights of the short-term memory feature and the long-term memory feature based on the first matching degree and the second matching degree; A feature fusion submodule is used to fuse the short-term memory feature and the long-term memory feature based on the weights of the short-term memory feature and the long-term memory feature to obtain the predicted emotional state.

23. According to the device described in any one of claims 20 to 22, the interaction strategy generation module is used to input the predicted emotional state into the interaction strategy network of the emotional interaction model to obtain the interaction strategy.

24. The apparatus according to claim 23, further comprising: A receiving module, used for receiving feedback information on the emotional response of the digital human; A processing module, configured to obtain an emotion change state based on the feedback information through the interactive strategy network; An adjustment module is used to adjust the interaction strategy based on the emotion change state through the interaction strategy network.

25. A training device for an emotional interaction model, comprising: The emotion feature acquisition module is used to input the sample emotion data of the interaction with the digital human into the emotion interaction model to obtain the sample emotion features; A memory feature acquisition module, used for inputting the sample emotion feature into the short-term memory network and the long-term memory network of the emotion interaction model to obtain the sample short-term memory feature and the sample long-term memory feature; A training module is used to train the emotion interaction model based on the sample short-term memory characteristics and the sample long-term memory characteristics.

26. The device according to claim 25, wherein The sample emotion data includes sample short-term emotion data and sample long-term emotion data, and the priority of the sample long-term emotion data is determined based on at least one of the following: emotion intensity, time weight, modal credibility, memory relevance, and personalized characteristics.

27. The device according to claim 25 or 26, wherein: The emotion feature acquisition module comprises: An extraction submodule, used to input the multimodal sample emotion data interacting with the digital human into the multimodal network of the emotion interaction model, and extract sample features of different modalities; An adjustment submodule, configured to adjust the weights of the sample features of the different modalities through a cross-modal attention mechanism of the multimodal network; The fusion submodule is used to obtain the sample emotion feature based on the sample features of different modalities and their adjusted weight fusion.

28. The device according to claim 27, wherein The sample emotion features include sample short-term emotion sequence features and sample long-term emotion sequence features, and the memory feature acquisition module includes: A short-term feature acquisition submodule, used for inputting the sample short-term emotion sequence feature into the short-term emotion network of the emotion interaction model to obtain the sample short-term memory feature; The long-term feature acquisition submodule is used to input the sample long-term emotion sequence features into the long-term emotion network of the emotion interaction model to obtain the sample long-term memory features.

29. The device according to any one of claims 25 to 28, wherein The training module comprises: A short-term loss submodule, used for obtaining a short-term memory loss based on the short-term memory characteristics of the sample; A long-term loss submodule, which obtains a long-term memory loss based on the long-term memory characteristics of the sample; An overall loss submodule, used for obtaining an overall loss based on the short-term memory loss and the long-term memory loss; A training submodule is used to train the short-term memory network and the long-term memory network of the emotion interaction model based on the overall loss.

30. The device according to any one of claims 25 to 28, wherein: The training samples of the emotion interaction model include emotion switching samples, and the training weight of the emotion switching samples in the loss function is higher than the training weight of the non-emotion switching samples.

31. The apparatus according to any one of claims 25 to 30, further comprising: The memory feature fusion module is used to fuse the sample short-term memory feature and the sample long-term memory feature to obtain the sample predicted emotional state.

32. The device according to claim 31, wherein The memory feature fusion module comprises: An emotional demand submodule, used to obtain sample short-term emotional demand and sample long-term emotional demand based on the sample emotional data; A storage feature submodule, used to obtain a sample short-term memory storage feature and a sample long-term memory storage feature based on the sample short-term memory feature and the sample long-term memory feature; A short-term matching submodule, used to obtain a first matching degree between the sample short-term emotion and the sample short-term memory based on the sample short-term emotion demand and the sample short-term memory storage characteristics; The long-term matching submodule obtains the second matching degree between the sample long-term emotion and the sample long-term memory based on the sample long-term emotional needs and the sample long-term memory storage characteristics; A weight adjustment submodule, adjusting the weights of the sample short-term memory feature and the sample long-term memory feature based on the first matching degree and the second matching degree; The feature fusion submodule fuses the sample long-term memory feature and the sample short-term memory feature based on the weights of the sample short-term memory feature and the sample long-term memory feature to obtain the predicted emotional state of the sample.

33. The apparatus according to claim 31 or 32, further comprising: An interactive strategy generation module, used for inputting the predicted emotional state of the sample into the interactive strategy network of the emotional interaction model to generate an interactive strategy for the digital human; The interaction strategy is used to control the emotional response of the digital person; A receiving module, used for receiving feedback information on the emotional response of the digital human; A reinforcement learning training module is used to perform reinforcement learning training on the interactive strategy network based on the feedback information.

34. The device according to claim 33, wherein The reinforcement learning training module comprises: A processing submodule, used for obtaining a change in sentiment intensity and a feedback score based on the feedback information; A reward function submodule, used to obtain a reward value of a reward function based on the emotion intensity change and the feedback score; An optimization submodule is used to optimize the interactive strategy network based on the reward value using a proximal strategy optimization algorithm.

35. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 17.

36. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-17.

37. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 17.

Images