A Method and System for Emotional Interaction Based on Multi-Attention Mechanism Fusion

Abstract

This invention proposes an emotion interaction method and system based on multi-attention mechanism fusion. The method includes: constructing a contextual semantic vector based on continuous multi-turn user input; concatenating the contextual semantic vector and a real-valued emotion vector to form a joint representation vector; obtaining a current emotion state vector based on the joint representation sequence and the current contextual semantic vector using a graph path-aware attention mechanism of a graph neural network; fusing the current emotion state vector and the current contextual semantic vector using a dual-channel gated attention response input fusion mechanism to obtain a current response control vector for generating a model decoder; executing the response behavior of the current response control vector and collecting the user's feedback emotion state; and introducing a policy optimization network based on reinforcement learning to perform closed-loop policy optimization on the response behavior based on the user's feedback emotion state to obtain the optimal response behavior.

Description

Technical Field

[0001] This invention belongs to the field of multi-attention mechanism fusion, and particularly relates to an emotional interaction method and system based on multi-attention mechanism fusion. Background Technology

[0002] With the continuous advancement of artificial intelligence technology, affective computing has gradually become a key component of human-computer interaction systems, especially in scenarios such as intelligent voice assistants, virtual customer service, mental health support, and companion robots. Emotion recognition and feedback capabilities are considered core competencies for improving user experience and system intelligence. Current affective interaction technologies mainly include two aspects: firstly, emotion recognition, which typically relies on modal inputs such as voice, text, and images, using deep learning models to determine the user's current emotion category or intensity; secondly, emotion-driven interaction generation, such as adjusting tone, intonation, or response content based on the recognized emotional state, thereby achieving more natural communication that closely resembles human interaction habits.

[0003] However, despite the progress made in the static accuracy of emotion recognition, several key issues remain unresolved in real-world interaction scenarios. First, existing methods generally rely on a current input-driven recognition mechanism, judging emotion solely based on the current sentence or round of voice input, neglecting the emotional evolution of users across multiple interactions. In real-world applications, user emotions exhibit clear temporal and gradual changes, often accumulating or subsiding through multiple rounds of communication; this dynamic emotional drift is difficult for traditional models to capture. Second, emotion recognition and interaction decision-making remain separate modules in most systems. The former completes recognition and then passes the label to the latter to generate a response, lacking deep coupling modeling between emotion, semantics, and behavior. This results in systems often only producing templated or simplistic responses, failing to reflect truly empathetic intelligent behavior. Furthermore, current interaction systems generally lack emotion regulation capabilities. Even when a system recognizes a user's negative emotions, such as anger or anxiety, it often only responds to factual or semantic content, failing to effectively guide the user towards a positive shift. A further problem is that the system struggles to assess which response behavior is most helpful in improving user emotions during the interaction process, and lacks an effective strategy optimization mechanism, resulting in highly random interactive feedback and an uncontrollable experience.

[0004] Given the current situation, existing technologies still have significant gaps in building truly dynamic emotional interaction systems with dynamic understanding, empathetic feedback, and adaptive optimization capabilities. Therefore, there is an urgent need for a new technological system capable of comprehensively modeling emotional evolution, dynamically controlling interactive responses, and possessing adaptive feedback optimization capabilities to meet the increasingly complex, personalized, and profound needs of human-computer emotional interaction. Summary of the Invention

[0005] The purpose of this invention is to propose an emotion interaction method and system based on the fusion of multiple attention mechanisms. Through the highly integrated design of emotion modeling and feedback control, a closed-loop interaction system with dynamic recognition and strategy adjustment capabilities is constructed.

[0006] To achieve the above objectives, a first aspect of the present invention provides an emotion interaction method based on multi-attention mechanism fusion, the method comprising the following steps:

[0007] A contextual semantic vector is constructed based on continuous multi-round input from the user. The contextual semantic vector is then input into an encoder to calculate a real-valued sentiment vector. The contextual semantic vector and the real-valued sentiment vector are concatenated to form a joint representation vector and a joint representation sequence. The contextual semantic vector is obtained using a multi-round nested input strategy that is concatenated round by round.

[0008] Based on the joint representation sequence, combined with the current context semantic vector, the current emotion state vector is obtained using the graph path-aware attention mechanism of the graph neural network;

[0009] Based on the current emotional state vector and the current context semantic vector, the response input fusion mechanism of dual-channel gated attention is used to obtain the current response control vector for generating the model decoder; the response control vector is input into the decoder for decoding to obtain the response behavior corresponding to the current response control vector;

[0010] The system executes the response behavior according to the current response control vector and collects the user's feedback emotional state. A policy optimization network based on reinforcement learning is introduced to optimize the response behavior using a closed-loop policy based on the user's feedback emotional state, thereby obtaining the optimal response behavior.

[0011] Further, the step of inputting the context semantic vector into the encoder to calculate the real-valued sentiment vector, concatenating the context semantic vector and the real-valued sentiment vector to form a joint representation vector, and forming a joint representation sequence specifically includes:

[0012] The context semantic vector is input into the encoder, and the feature representation is enhanced through nonlinear transformation. At the same time, the complexity of the encoder is controlled by the emotion intensity constraint term, and the real-valued emotion vector is calculated. The training of the control encoder introduces the soft classification distribution of the context semantic vector, and then aligns the soft classification distribution with the real emotion prior distribution. The loss term of the encoder is obtained through the Kullback-Leibler divergence function.

[0013] The context semantic vector and the real-valued emotion vector are concatenated to form a joint representation vector, and a joint representation sequence is formed.

[0014] Furthermore, based on the joint representation sequence and combined with the current context semantic vector, the current emotion state vector is obtained using the graph path-aware attention mechanism of the graph neural network, specifically including:

[0015] The current emotional state vector is converted into graph nodes, and the weights of directed edges are determined based on the Euclidean distance between any two graph nodes.

[0016] Using the current state node as the query source, an attention mechanism is initiated in the graph to calculate the attention weights for historical nodes; where the current state node is the current emotional state vector.

[0017] The current emotional state vector is aggregated and analyzed using the attention weights to calculate the user's context-aggregated emotional state representation at the current moment.

[0018] Furthermore, the step of converting the current emotion state vector into graph nodes and determining the directed edge weights based on the Euclidean distance between any two graph nodes is specifically calculated as follows:

[0019]

[0020] Where, ω ij h represents the weight of the directed edge. i ,h j It is the current context semantic vector of any two graph nodes, e i ,e j It is the emotion state vector of any two graph nodes; the parameters β1, β2, δ, ρ control the weight of each item; T is the sequence length; i and j are indices;

[0021] in, It is a time interval normalization penalty term that encourages the model to focus on dialogue rounds that are close in time; It is an emotion direction conflict penalty term, which means that when two emotion vectors are in opposite directions, the edge weight between the corresponding two graph nodes is automatically reduced to avoid forming wrong paths in the emotion trajectory graph;

[0022] The attention weights for the attention mechanism initiated in the graph for historical nodes are calculated as follows:

[0023]

[0024] Among them, W q W k Let ω be the projection matrix of the query and the key, and a be the attention vector. it Let be the edge weight from the i-th historical node to the current node t; log(1+∈+ω) it ) represents the path penalty term, where ∈ = 10 -6To ensure numerical stability; the coefficient γ controls the degree of influence of path weights on attention distribution; α ti Attention weights;

[0025] Among them, when the path is unreasonable, ω it Low, resulting in log(1+ω) it The node is small, so attention naturally avoids it.

[0026] Furthermore, the response input fusion mechanism of the dual-channel gated attention is calculated as follows:

[0027]

[0028] Where, r t It is the response control vector; σ(W) r E t +b r ) is the gating weight, which controls the ratio of semantics to emotion; μ· It is the emotional homeostasis residual term, where: This represents the average of historical sentiment over multiple rounds, i.e., the average of E values ​​from the past several rounds. k The average value represents the user's expected sentiment baseline; ResNorm(·) represents the weighted normalized residual function, used to amplify sentiment bias; μ is a control term used to adjust the residual effect.

[0029] Further, the response control vector is input to the decoder for decoding to obtain the response behavior corresponding to the current response control vector, which is calculated as follows:

[0030]

[0031] Among them, f emo (y t ) is an emotion perception module that encodes emotions into the generated text; Let be the loss function of the decoder.

[0032] Furthermore, the introduction of a reinforcement learning-based policy optimization network to perform closed-loop policy optimization on the response behavior based on the user's feedback emotional state, thereby obtaining the optimal response behavior, specifically includes:

[0033] Design a reward function to obtain a reward value, in order to measure whether the behavioral response guides the user toward the desired emotional direction, while avoiding large negative emotional fluctuations;

[0034] The policy network takes the current joint representation vector as input as the state and outputs a policy vector that controls the behavior. The policy vector is optimized using the policy gradient method to make it tend to choose behaviors with higher rewards in order to generate the optimal response behavior.

[0035] Furthermore, the design of the reward function yields the reward value r. t The calculation is as follows:

[0036]

[0037] Among them, e t+1 The emotion vectors in user feedback are extracted by the same emotion encoder. κ1 and κ2 are the emotional target vector that the system expects the user to gradually approach; κ1 and κ2 are hyperparameters used to control the balance weights.

[0038] in, A target state consistency score is used to measure whether the current emotion is close to the target state; ||e t+1 -E t || 2 This is a penalty for emotional fluctuations, designed to limit drastic changes in emotions.

[0039] Furthermore, the emotion perception module is a TextCNN and MLP structure.

[0040] A second aspect of the invention provides an emotion interaction system based on multi-attention mechanism fusion, the system comprising:

[0041] The user input acquisition module is used to construct a contextual semantic vector based on continuous multi-round user input, input the contextual semantic vector into the encoder to calculate the real-valued sentiment vector, concatenate the contextual semantic vector and the real-valued sentiment vector to form a joint representation vector, and form a joint representation sequence; wherein the contextual semantic vector is obtained by a multi-round nested input strategy of sequential concatenation.

[0042] The emotion value analysis module is used to obtain the current emotion state vector based on the joint representation sequence, combined with the current context semantic vector, and using the graph path-aware attention mechanism of the graph neural network.

[0043] The action response module is used to fuse the current emotion state vector and the current context semantic vector through a dual-channel gated attention response input fusion mechanism to obtain the current response control vector for generating the model decoder; the response control vector is input into the decoder for decoding to obtain the response behavior corresponding to the current response control vector;

[0044] The strategy optimization module is used to execute the response behavior of the current response control vector and collect the user's feedback emotional state; a reinforcement learning-based strategy optimization network is introduced to perform closed-loop strategy optimization on the response behavior based on the user's feedback emotional state to obtain the optimal response behavior.

[0045] The beneficial technical effects of the present invention are at least as follows:

[0046] This invention addresses the shortcomings of existing emotional interaction technologies in multi-turn interaction scenarios, such as lack of dynamic modeling, module fragmentation, and uncontrollable feedback. It proposes an emotional interaction method and system that integrates a multi-attention mechanism. Through a highly integrated design of emotion modeling and feedback control, a closed-loop interaction system with dynamic recognition and strategy adjustment capabilities is constructed. The main innovations of this invention are as follows: First, by introducing a modeling mechanism for the emotional evolution path in multi-turn inputs, this invention achieves dynamic representation and tracking of the user's emotional state, effectively overcoming the information gap problem caused by traditional emotion recognition methods that only recognize the current round of input. Second, based on emotion recognition, this invention further designs a module that integrates the current emotional state into the interaction strategy control process, enabling the system response to no longer be based solely on semantic logic generation, but to provide accurate, stylized, and empathetic feedback to changes in the user's emotions. Finally, to overcome the singularity and randomness of traditional interaction response feedback strategies, this invention introduces a regulation strategy learning method based on a feedback optimization mechanism. This allows the system to continuously learn which interactive behaviors are most helpful in improving the user's emotional state during multi-turn dialogues, thereby achieving true emotional guidance and empathetic optimization. Overall, this invention establishes an effective dynamic coupling mechanism among emotion state modeling, response control, and strategy learning, forming an emotional interaction technology system with complete adaptive capabilities. Attached Figure Description

[0047] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0048] Figure 1 This is a flowchart of the emotion interaction method based on the fusion of multiple attention mechanisms of the present invention. Detailed Implementation

[0049] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0050] like Figure 1 As shown in the embodiment of the present invention, the emotion interaction method based on multi-attention mechanism fusion includes:

[0051] S1. Construct a context semantic vector based on continuous multi-round input from the user, input the context semantic vector into the encoder to calculate the real-valued sentiment vector, concatenate the context semantic vector and the real-valued sentiment vector to form a joint representation vector, and form a joint representation sequence; wherein the context semantic vector is obtained by a multi-round nested input strategy of sequential concatenation.

[0052] Specifically, the main goal of this step is to process the user's input x1, x2, ..., x in multiple consecutive rounds. t In the process, a joint representation vector z that simultaneously reflects linguistic semantics and emotional features is extracted. i This provides structured and emotion-sensitive graph nodes for subsequent emotion trajectory mapping and attention fusion. Since the subsequent system must model the evolution path of emotions, this step designs a joint modeling mechanism that can capture the intensity of emotion expression, the ambiguity of emotion categories, and contextual consistency, ensuring that each round of z... i It can be truly used to model emotional states in evolution, rather than isolated emotional labels or static sentence vectors.

[0053] For the user input x in the i-th round i This invention first uses a language model fine-tuned for the dialogue scenario (such as RoBERTa) to extract the contextual semantic vector h. i This represents the content and language structure of the current statement. Since subsequent emotional trajectories require contextual continuity, this invention employs a multi-round nested input strategy, where each round is sequentially connected. Specifically, when extracting h... i At that time, input x from the previous round i-1 With the current round input x i Concatenate the input model to make h i It can reflect the logic and contextual relationships between sentences. This strategy can be regarded as nested contextual language modeling, which is more suitable for multi-turn interaction scenarios than single-turn independent modeling.

[0054] Based on h i This invention designs a lightweight linear emotion encoder with enhanced emotion discrimination capabilities to compute a real-valued emotion vector e. i This encoder not only considers h i It also introduces a special emotion intensity constraint to encourage the system to learn sentences expressing strong emotions during training, with its e i The magnitude is larger and the direction is clearer, thus enhancing the separability of edge weights in the trajectory mapping stage. This emotion vector is calculated using the following structure:

[0055] e i =tanh(W e h i +b e )+λ·NormPenalty(h i(1)

[0056] in, It is a linear mapping matrix, b e h is the bias term. i It is a semantic vector, and tanh(·) is a hyperbolic tangent activation function used for emotion range normalization.

[0057] This is the emotion intensity regularization term, where α is the adjustment coefficient and λ is the regularization intensity weight, used to increase e. i Vector separability when expressing strong emotions.

[0058] Furthermore, to enhance the expressive power of the output of this step regarding emotional ambiguity, this invention further introduces a distributed emotional consistency constraint based on KL divergence. This invention provides a constraint for each emotional vector e. i An auxiliary output is introduced during training: its soft classification distribution. Compare it with the prior distribution of true emotion q i Alignment is performed using weakly supervised annotations or dictionary matching, constrained by the following loss term:

[0059]

[0060] Among them, W p For classification mapping matrix, From h i The predicted sentiment distribution derived from projection, q i It is a weakly supervised label distribution generated from the dictionary, and KL(·||·) is the Kullback-Leibler divergence function. This term helps e i It should gravitate towards meaningful distributions, enhancing its ability to express vague emotions rather than extreme classifications.

[0061] Finally, the semantic vector h i With emotion vector e i The vectors are concatenated to form a joint representation vector z. i =[h i ||e i ]. z i As input to graph nodes, it includes not only the meaning of the text content of the current round of dialogue (h) i It also possesses emotional vectors (e) that exhibit differentiation and directionality after regularization enhancement. i This makes the edge weight calculation in the next step of constructing the emotion trajectory graph more physically meaningful and dynamically separable, especially in scenarios where emotions evolve slowly or reach a critical point. The entire z... i The construction process is a structured representation that serves trajectory modeling, rather than a simple splicing operation. iThe construction method takes into account the modulus length control, semantic consistency and emotional ambiguity expression, ensuring that each node is not only a feature representation, but also a behavioral state unit in the trajectory analysis. In the subsequent graph attention mechanism, the precise focus on key nodes is achieved through the vector difference, intensity difference and direction difference between nodes. The final output is the joint feature sequence Z = {z1, z2,..., z t}, which will be input as the nodes of the emotional trajectory graph in Step 2 and used for edge construction and attention focusing.

[0062] S2. According to the joint representation sequence, combined with the current context semantic vector, use the graph path-aware attention mechanism of the graph neural network to obtain the current emotional state vector.

[0063] Specifically, this step aims to structure the semantic-emotional joint representation sequence Z = {z1, z2,..., z t} output in Step 1 into an emotional trajectory graph that can dynamically reflect the user's emotional evolution trend, and focus on the historical states in the graph structure through the attention mechanism, so as to output the aggregated emotional state vector E t . This representation not only needs to accurately reflect the current input language content and emotional characteristics, but also needs to fully perceive the historical emotional influence path, so that the responses generated by the subsequent system have continuity and adjustability. This step is the core of the emotional evolution modeling of the entire system, and its construction quality determines the upper limit of the subsequent strategy adjustment and empathy generation capabilities.

[0064] Based on z i = [h i || e i output in Step 1, each round of input x i is converted into a semantic-emotional joint vector, h i is the semantic feature generated by the language model, and e i is the emotional vector calculated based on the weak supervision mechanism. The present invention constructs these vectors into graph nodes and designs an asymmetric edge structure based on bidirectional path attention weights to reflect the non-linear propagation of emotions in multi-round interactions.

[0065] Furthermore, for any two graph nodes z i and z j (i < j), the present invention defines the directed edge weight ω[[ID=...]] ij between them, which not only depends on the Euclidean distance between semantics and emotions, but also considers the time position difference penalty term and the emotional conflict direction loss term, forming the following structure:

[0066]

[0067] Among them, h i , hj It is a semantic vector, e i ,e j These are emotion vectors, all from step one; the first and second terms are standard semantic-emotion difference weighted averages; the third term... The fourth term is a time interval normalization penalty term, which encourages the model to focus on dialogue turns that are close in time; This is an emotion direction conflict penalty term. Its design purpose is to automatically reduce the edge weight between two emotion vectors when their directions are opposite (i.e., their inner product is negative) to avoid forming incorrect paths in the emotion trajectory graph. The parameters β1, β2, δ, and ρ control the weight of each term. For example, setting β1 = 1.0, β2 = 1.5, δ = 0.3, and ρ = 2.0 emphasizes that emotions and their directionality are more important.

[0068] Understandably, this edge weight definition not only integrates multi-dimensional information (semantics / emotion / time / emotional direction), but also improves the construction quality of emotional trajectories by punishing unreasonable connections. This is a significantly different approach to emotion evolution modeling from traditional modeling based on Transformer or RNN structures.

[0069] Furthermore, after constructing the graph structure, this invention uses the current round's state node z t To query the source, an attention-focusing process is initiated on historical nodes in the graph. To ensure that the attention mechanism can simultaneously consider node content similarity and path structure rationality, this invention introduces a graph path-aware attention mechanism. Unlike conventional self-attention which is based solely on content vector similarity, this mechanism explicitly incorporates logarithmic edge weights extracted from the edge weight function as path suppression terms into the attention query term, thereby effectively avoiding the shift of attention distribution along emotional conflict paths.

[0070] The attention calculation structure is as follows:

[0071]

[0072] Among them, W q W k Let ω be the projection matrix of the query and the key, and a be the attention vector. it Let be the edge weight from the i-th historical node to the current node t; log(1+∈+ω) it ) represents the path penalty term, where ∈ = 10 -6 To ensure numerical stability; the coefficient γ controls the degree of influence of path weights on attention distribution; when the path is unreasonable (e.g., long duration, semantic fragmentation, or opposite emotions), ω... it Low, resulting in log(1+ω) it The node is small, thus naturally drawing attention away from it. ti For attention weights.

[0073] Unlike Transformer, which allocates attention based purely on content similarity, this mechanism provides a structural solution to the problem of repeated fluctuations in user emotions during dialogue in this scenario, and can prevent the system from being misled by incorrect old emotional states.

[0074] Using the attention weight α mentioned above ti This invention calculates the user's context-aggregated emotional state representation E at the current moment. t :

[0075]

[0076] This vector incorporates the current input z. t The relationship between the system and all historical states in terms of content and path serves as a control signal when the system generates a response in the next stage, determining whether the system should use a comforting tone, an explanatory tone, or a neutral feedback method.

[0077] S3. Based on the current emotional state vector and the current context semantic vector, the response input fusion mechanism of dual-channel gated attention is used to obtain the current response control vector for generating the model decoder; the response control vector is input into the decoder for decoding to obtain the response behavior corresponding to the current response control vector.

[0078] Specifically, this step follows the current emotional state vector E output in step two. t And combined with the current semantic representation h from step one t Used to generate system responses y with semantic accuracy and emotional empathy. t In this patented solution, the core objective of this step is not simply to complete a semantically coherent dialogue output, but rather to proactively adjust the response style, tone, and expression strategies during the generation process by introducing an emotion-driven generation and regulation mechanism to adapt them to the current user's emotional state. This capability is a key implementation link in realizing an emotional interaction system with feedback regulation capabilities, undertaking the core task of the entire system's shift from emotion perception to behavior control.

[0079] This invention proposes a response-input fusion mechanism based on dual-channel gated attention to construct a response representation r for generating models. t This mechanism not only combines h t With E t Furthermore, an emotion steady-state disturbance factor is introduced to address the issue of unstable system response under user emotion fluctuations. The overall calculation structure is as follows:

[0080]

[0081] Where, r tIt is the response control vector ultimately used to generate the model decoder; σ(W) r E t +b r () is the gating weight, which controls the ratio of semantics to emotion integration; This is a specially designed emotion steady-state residual term in this step, where: This represents the average of historical sentiment over multiple rounds, i.e., the average of E values ​​from the past several rounds. k The average value represents the user's expected sentiment baseline; ResNorm(·) represents the weighted normalized residual function, used to amplify sentiment bias; μ is a control term (e.g., μ = 0.7), used to adjust the residual effect; σ(W r E t +b r ) is the gating factor.

[0082] The core innovation of this approach lies in addressing the issue of style instability that can occur when a user's emotions shift suddenly from smiling to calm or anger in real-world scenarios. By incorporating a signal indicating the degree of the user's emotional shift into the generated input, the system can proactively adjust its style (e.g., responding to angry transitions with a soothing tone), which is particularly effective in psychological counseling interactions and virtual companionship scenarios.

[0083] The response generation module uses a decoder (such as TransformerDecoder or LLM) to receive r t Vector generation system responds to text y t To guide the generated text to have a clear emotional expression style, the system introduces a target emotion-guided loss function during the training phase, aiming to optimize the controllability of the generated content at the emotional expression level.

[0084]

[0085] Among them, f emo (y t ) is an emotion-aware module that encodes emotions into the generated text (such as the TextCNN+MLP structure);

[0086] Furthermore, the goal is to ensure that the emotions expressed in the generated text are consistent with the emotions the system understands in the user;

[0087] In particular, this invention does not require the output emotion to be aligned only with E. t Instead, it aligns with the dynamic sentiment trend vector—that is, E. t The integration with its steady-state residuals allows the generated content to proactively adapt to either mitigation or responsive divergence.

[0088] The key innovation of this step lies not only in fusing semantic and emotional signals through a gating mechanism, but also in introducing the modeling concept of historical emotional steady-state residuals. This enables the system to model the deviation between the current emotion and the stable baseline, thereby proactively adjusting behavioral output during generation. Unlike traditional style control methods that rely solely on labels to guide generation, this invention constructs a continuous adjustment variable, supporting more nuanced and realistic human-computer interaction behavior modeling. It demonstrates significant effectiveness and practicality in emotional interaction tasks such as multi-round anxiety regulation and anger management.

[0089] S4. Execute the response behavior of the current response control vector and collect the user's feedback emotional state; introduce a policy optimization network based on reinforcement learning, and optimize the response behavior in a closed loop based on the user's feedback emotional state to obtain the optimal response behavior.

[0090] Specifically, this step aims to analyze the response behavior y generated by the system in step three. t The strategy optimization involves the system, after outputting a response, determining whether the previous response had a positive regulatory effect based on the user's next round of feedback (especially changes in emotional state), and updating the behavior generation strategy accordingly. More importantly, this step goes beyond the modeling layer, explicitly applying the strategy optimization results directly to the actual system's response control module, thus achieving a complete closed loop from strategy optimization to actual interactive behavior control, enabling the entire system to possess realistic emotional feedback regulation capabilities.

[0091] This step requires input including the current system state. t =[h t ||E t ], where h t E is the semantic vector input by the current user (from step one). t The current aggregated emotion state vector (from step two) together constitute a comprehensive expression of the semantic intent and emotional context of the current dialogue; generate response y. t From step three, the natural language response is gated by emotion; the user's feedback on this behavior is conveyed through their next input x. t+1 In summary, this invention extracts the feedback emotional state vector e using the same emotion encoder. t+1 .

[0092] Furthermore, the system uses feedback emotional state e t+1 To determine the response y t The actual effect. This invention designs a reward function r t To measure behavior y t Whether to guide users towards a desired emotional state (such as calmness or peace) while avoiding significant negative emotional fluctuations. The reward function is defined as follows:

[0093]

[0094] Among them, e t+1 The emotion vectors in user feedback are extracted by the same emotion encoder. The system expects users to gradually approach an emotional target vector, such as a calm or positive state. The first term is the target state consistency score, which measures whether the current emotion is close to the target state. The second term is the emotion fluctuation penalty term, which limits drastic changes in emotion. Parameters such as κ1=1.2 and κ2=0.8 control the balance weight.

[0095] Furthermore, this structure ensures that the system aims to guide users toward positive emotions without overstimulating them and causing emotional backlash, making it particularly suitable for scenarios such as anxiety intervention and psychological support.

[0096] Policy Network π θ With s t The input is a policy vector or category distribution (such as soothing, explaining, encouraging, etc.) generated from the output control behavior. This invention uses a policy gradient method to optimize the policy, making it tend to choose higher-reward behaviors.

[0097]

[0098] Where, π θ : Policy network, outputs the behavior control variables in the current state; y t The generated response behavior comes from the system behavior generation module; r t Rewards calculated based on user feedback, used to adjust π. θ The strategy network continuously optimizes its behavioral decision-making mechanism, enabling the system to gradually learn to output more emotionally driven responses in similar situations.

[0099] Furthermore, to achieve a genuine correlation between the strategy model and the behavioral execution of the emotional interaction system, the system incorporates the output of the strategy network as part of the behavior control interface during the deployment phase, directly influencing the response generation method. Specifically, this includes:

[0100] If the policy network output is a style tag (such as mild), the system selects a predefined template under that style for language fill.

[0101] If the output is a control vector (such as a mood modulation embedding), then the vector is injected into the generative model (such as a Transformer decoder) to affect its decoding process;

[0102] This control mechanism is embedded in the emotion gating structure of step three, truly forming a closed-loop pathway of strategy → control vector → response generation;

[0103] System response action y t Therefore, it is not only reasonable in content, but also in style and emotionally appropriate.

[0104] This invention also provides an emotion interaction system based on multi-attention mechanism fusion, the system comprising:

[0105] The user input acquisition module is used to construct a contextual semantic vector based on continuous multi-round user input, input the contextual semantic vector into the encoder to calculate the real-valued sentiment vector, concatenate the contextual semantic vector and the real-valued sentiment vector to form a joint representation vector, and form a joint representation sequence; wherein the contextual semantic vector is obtained by a multi-round nested input strategy of sequential concatenation.

[0106] The emotion value analysis module is used to obtain the current emotion state vector based on the joint representation sequence, combined with the current context semantic vector, and using the graph path-aware attention mechanism of the graph neural network.

[0107] The action response module is used to fuse the current emotion state vector and the current context semantic vector through a dual-channel gated attention response input fusion mechanism to obtain the current response control vector for generating the model decoder; the response control vector is input into the decoder for decoding to obtain the response behavior corresponding to the current response control vector;

[0108] The strategy optimization module is used to execute the response behavior of the current response control vector and collect the user's feedback emotional state; a reinforcement learning-based strategy optimization network is introduced to perform closed-loop strategy optimization on the response behavior based on the user's feedback emotional state to obtain the optimal response behavior.

[0109] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0110] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0111] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An emotion interaction method based on multi-attention mechanism fusion, characterized in that, The method includes the following steps: A contextual semantic vector is constructed based on continuous multi-round input from the user. The contextual semantic vector is then input into an encoder to calculate a real-valued sentiment vector. The contextual semantic vector and the real-valued sentiment vector are concatenated to form a joint representation vector and a joint representation sequence. The contextual semantic vector is obtained using a multi-round nested input strategy that is concatenated round by round. Based on the joint representation sequence, combined with the current context semantic vector, the current emotion state vector is obtained using the graph path-aware attention mechanism of the graph neural network; Based on the current emotional state vector and the current context semantic vector, the response input fusion mechanism of dual-channel gated attention is used to obtain the current response control vector for generating the model decoder; the response control vector is input into the decoder for decoding to obtain the response behavior corresponding to the current response control vector; The system executes the response behavior of the current response control vector and collects the user's feedback emotional state; it then introduces a policy optimization network based on reinforcement learning to perform closed-loop policy optimization on the response behavior based on the user's feedback emotional state, thereby obtaining the optimal response behavior. Specifically, based on the joint representation sequence and combined with the current context semantic vector, the current emotion state vector is obtained using the graph path-aware attention mechanism of a graph neural network, including: The current emotional state vector is converted into graph nodes, and the weights of directed edges are determined based on the Euclidean distance between any two graph nodes. Using the current state node as the query source, an attention mechanism is initiated in the graph to calculate the attention weights for historical nodes; where the current state node is the current emotional state vector. The current emotional state vector is aggregated and analyzed using the attention weights to calculate the user's context-aggregated emotional state representation at the current moment. The process of converting the current emotional state vector into graph nodes and determining the directed edge weights based on the Euclidean distance between any two graph nodes is specifically calculated as follows: ； in, For directed edge weights, , It is the current context semantic vector of any two graph nodes. , It is the emotion state vector of any two graph nodes; parameters , , , The weight of each item is controlled; T is the sequence length; i and j are indices; in, It is a time interval normalization penalty term that encourages the model to focus on dialogue rounds that are close in time; It is an emotion direction conflict penalty term, which means that when two emotion vectors are in opposite directions, the edge weight between the corresponding two graph nodes is automatically reduced to avoid forming wrong paths in the emotion trajectory graph; The attention weights for the attention mechanism initiated in the graph for historical nodes are calculated as follows: ； in, , To query the projection matrix of the key, This is the attention vector; For the first Each historical node points to the current node. The right of the border; For path penalty terms, where Ensure numerical stability; coefficients The extent to which control path weights influence attention distribution; Attention weights; Among them, when the path is unreasonable Low, leading to The node is small, so attention naturally avoids that node.

2. The emotion interaction method based on multi-attention mechanism fusion according to claim 1, characterized in that, The step of inputting the context semantic vector into the encoder to calculate the real-valued sentiment vector, concatenating the context semantic vector and the real-valued sentiment vector to form a joint representation vector, and forming a joint representation sequence specifically includes: The context semantic vector is input into the encoder, and the feature representation is enhanced through nonlinear transformation. At the same time, the complexity of the encoder is controlled by the emotion intensity constraint term, and the real-valued emotion vector is calculated. The training of the control encoder introduces the soft classification distribution of the context semantic vector, and then aligns the soft classification distribution with the real emotion prior distribution. The loss term of the encoder is obtained through the Kullback-Leibler divergence function. The context semantic vector and the real-valued emotion vector are concatenated to form a joint representation vector, and a joint representation sequence is formed.

3. The emotion interaction method based on multi-attention mechanism fusion according to claim 1, characterized in that, The response input fusion mechanism of the dual-channel gated attention is calculated as follows: ； in, It is the response control vector; It is a gating weight that controls the ratio of semantics to emotion. It is the emotional homeostasis residual term, where: This represents the average of sentiment over multiple historical rounds, i.e., the sentiment over several past rounds. The average value represents the user's expected sentiment benchmark; This represents the weighted normalized residual function, used to amplify sentiment bias; It is a control item used to adjust the effect of residuals.

4. The emotion interaction method based on multi-attention mechanism fusion according to claim 3, characterized in that, The response control vector is input into the decoder for decoding to obtain the response behavior corresponding to the current response control vector, which is calculated as follows: ； in, It is an emotion perception module that performs emotion encoding on the generated text; Let be the loss function of the decoder.

5. The emotion interaction method based on multi-attention mechanism fusion according to claim 1, characterized in that, The introduction of a reinforcement learning-based policy optimization network, which optimizes the response behavior using a closed-loop policy based on the user's feedback emotional state to obtain the optimal response behavior, specifically includes: Design a reward function to obtain a reward value, in order to measure whether the behavioral response guides the user toward the desired emotional direction, while avoiding large negative emotional fluctuations; The policy network takes the current joint representation vector as input as the state and outputs a policy vector that controls the behavior. The policy vector is optimized using the policy gradient method to make it tend to choose behaviors with higher rewards in order to generate the optimal response behavior.

6. The emotion interaction method based on multi-attention mechanism fusion according to claim 5, characterized in that, The design of the reward function yields the reward value. The calculation is as follows: ； in, The emotion vectors in user feedback are extracted by the same emotion encoder. This represents the emotional target vector that the system expects the user to gradually approach. and These are hyperparameters used to control the balancing weights; in, A target state consistency score is assigned to measure whether the current emotion is close to the target state. This is a penalty for emotional fluctuations, designed to limit drastic changes in emotions.

7. The emotion interaction method based on multi-attention mechanism fusion according to claim 4, characterized in that, The emotion perception module uses a TextCNN and MLP structure.

8. A system for implementing the emotion interaction method based on multi-attention mechanism fusion as described in claim 1, characterized in that, The system includes: The user input acquisition module is used to construct a contextual semantic vector based on continuous multi-round user input, input the contextual semantic vector into the encoder to calculate the real-valued sentiment vector, concatenate the contextual semantic vector and the real-valued sentiment vector to form a joint representation vector, and form a joint representation sequence; wherein the contextual semantic vector is obtained by a multi-round nested input strategy of sequential concatenation. The emotion value analysis module is used to obtain the current emotion state vector based on the joint representation sequence, combined with the current context semantic vector, and using the graph path-aware attention mechanism of the graph neural network. The action response module is used to fuse the current emotion state vector and the current context semantic vector through a dual-channel gated attention response input fusion mechanism to obtain the current response control vector for generating the model decoder; the response control vector is input into the decoder for decoding to obtain the response behavior corresponding to the current response control vector; The strategy optimization module is used to execute the response behavior of the current response control vector and collect the user's feedback emotional state; a reinforcement learning-based strategy optimization network is introduced to perform closed-loop strategy optimization on the response behavior based on the user's feedback emotional state to obtain the optimal response behavior.

Images