Multimodal human-computer interaction methods with empathy mechanisms

Abstract

This invention relates to a multimodal human-computer interaction method with an empathy mechanism. It extracts the user's emotional features in real-time from multiple modalities during the interaction process, obtaining emotional features corresponding to each modality. These extracted emotional features are then fused to form a multimodal emotional feature vector representing the user's current emotion. This vector is further processed to obtain the user's current emotion. Finally, the obtained current emotion is input into a dialogue model to predict the user's future emotion, and empathetic interaction content matching that future emotion is output to the user. This method not only accurately identifies the user's current emotion based on a multimodal approach but also predicts it and outputs empathetic interaction content matching the user's future emotion, thus humanizing the human-computer interaction process and enhancing its application capabilities in complex emotional interaction scenarios.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction, and more particularly to a multimodal human-computer interaction method with an empathy mechanism. Background Technology

[0002] With the development of artificial intelligence and human-computer interaction technologies, multimodal human-computer interaction has become an important means to improve the responsiveness and interactive experience of intelligent systems. Multimodal human-computer interaction integrates multiple input methods (or modalities) such as voice, facial expressions, gestures, and touch, enabling the system to perceive and understand user needs from multiple dimensions.

[0003] However, existing multimodal human-computer interaction methods have shortcomings:

[0004] First, existing multimodal human-computer interaction methods mainly focus on the accuracy and efficiency of completing interaction tasks, such as simply focusing on accurately identifying the user's emotions as "happy" or "sad" (or feelings). They lack a deep understanding of the user's emotions and intentions, resulting in an overly mechanical human-computer interaction process. They lack the ability to empathize with the user's emotions and cannot subtly perceive and empathize with the user's emotional state during the interaction process. They are unable to provide interactive content such as comfort or support based on the user's emotions (such as anxiety or frustration), and the interaction effect is not human enough.

[0005] Secondly, since users' emotional states usually change dynamically during human-computer interaction, especially during continuous interaction, existing multimodal human-computer interaction methods cannot adjust in a timely manner according to the fluctuations in users' emotions. This reduces the naturalness and flexibility of the human-computer interaction process and greatly reduces the experience of the human-computer interaction process. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a multimodal human-computer interaction method with an empathy mechanism, addressing the aforementioned shortcomings of the prior art. This multimodal human-computer interaction method can not only identify the user's emotional state, but also output empathetic interaction content to the user during the interaction process, thereby humanizing the human-computer interaction process and improving the user experience.

[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is: a multimodal human-computer interaction method with an empathy mechanism, characterized by comprising the following steps:

[0008] Step 1: Perform multimodal real-time extraction of the user's emotional features during the human-computer interaction process to obtain the emotional features corresponding to each modality; wherein, the multimodality includes the user's facial expression image signal, audio signal and dialogue text semantic signal;

[0009] Step 2: The extracted emotional features are fused to form a multimodal emotional feature vector representing the user's current emotion; wherein, in this multimodal emotional feature vector, each modality has its corresponding emotional feature vector;

[0010] Step 3: Process the multimodal sentiment feature vector to obtain the user's sentiment at the current moment;

[0011] Step 4: Input the user's current emotion into the dialogue model to predict the user's future emotion, and output empathetic interaction content that matches the user's future emotion to the user.

[0012] Improved, in the multimodal human-computer interaction method with empathy mechanism, the process of fusing various emotional features to form the multimodal emotional feature vector in step 2 includes the following steps a1 to a3:

[0013] Step a1: The feature-level fusion method is used to fuse the emotional features extracted from facial expression image signals and the emotional features extracted from user audio signals to obtain the user's non-verbal emotional features.

[0014] Step a2: Extract semantic features from the semantic signals of the user's dialogue text, and combine the semantic features with the sentiment dictionary to generate sentiment tags, thereby obtaining the user's language sentiment text features;

[0015] Step a3 involves fusing the obtained non-verbal emotional features and verbal emotional text features of the user to obtain the multimodal emotional feature vector.

[0016] Furthermore, in the multimodal human-computer interaction method with empathy mechanism, step 4, the process of processing the multimodal emotion feature vector to predict the user's future emotions, includes the following steps b1 to b3:

[0017] Step b1: The emotional feature vector extracted from the user's facial image signal and the emotional feature vector extracted from the user's audio signal are concatenated to obtain an aggregated semantic vector.

[0018] Step b2: Input the aggregated semantic vector into the fully connected layer of the convolutional neural network for processing, and then use the polarity consistency cross-entropy loss function to perform sentiment classification training on the semantic vector after processing by the fully connected layer.

[0019] Step b3: The trained dialogue model is used to process the input multimodal sentiment feature vector, and the output of the dialogue model is used as the predicted sentiment of the user at future time.

[0020] Further improvements are made to the multimodal human-computer interaction method with empathy mechanism. In step 3, the process of obtaining the user's current emotion by processing the emotional feature vector corresponding to the semantic signal modality of the user's dialogue text includes the following steps c1 to c10:

[0021] Step c1: Perform word embedding processing on the received user's dialogue text semantic signal; wherein, the received user's dialogue text semantic signal is denoted as U=[CLS,u1,u2,…,u N ], CLS represents the aggregate semantic information of the entire statement, u i Let u represent the i-th token, and use word embedding and position embedding to embed each token u respectively. i The mapping is represented as a vector, 1≤i≤N, where N represents the total number of tokens; token u i Word embedding representation as token u i Position embedding is represented as d is the dimension of the word vector;

[0022] Step c2, embed the speaker identification vector into each token representation; wherein, the speaker identification vector is embedded into token u i The text is marked as

[0023] Step c3: Based on the word embedding, position embedding, and speaker identification vector embedding obtained from the embedding, the final semantic embedding of each token is obtained; where the i-th token u i The final semantic embedding tag is s i ;

[0024]

[0025] Step c4: Encode the semantic context information of each token using a multi-layered stacked Transformer encoder, and calculate the hidden vector and final representation of each token; where:

[0026]

[0027] Among them, h i Indicates token u i The hidden vectors, where LayerNorm represents the layer normalization used to standardize the input features. Let MHAtt represent the input features of the i-th token at layer i-1 in the Transformer network, and MHAtt represent the multi-head self-attention mechanism. This represents the i-th token in the (i-1)-th layer. This represents the output of the i-th token after passing through the multi-head self-attention mechanism. Indicates residual connection, Let represent the final output vector of the i-th token in the l-th layer of the Transformer network; FNN represents a two-layer feedforward neural network with ReLU activation function; L represents the total number of layers in the Transformer network.

[0028] Step c5: Generate a contextual representation that represents the semantic context for emotion recognition; wherein, the contextual representation is denoted as Ω:

[0029] Ω=[ω1,ω2,…,ω N ];

[0030] Where, ω N It is the final feature vector of the Nth token in the sequence, which is also the final representation of the Nth sentence. The main information of the final representation of each sentence is stored in the sentence beginning vector.

[0031] Step c6: Based on the generated contextual representation, a self-attention mechanism is used to capture global emotional information during each round of human-computer dialogue; wherein:

[0032] c i =selfAtt([CLS]) i );

[0033] m i =Wc i +b;

[0034] Among them, c i For the i-th token u i The sentiment context vector, selfAtt represents the self-attention mechanism, m i For the sentiment context vector c i Mapped to a learnable sentiment representation; W and b are learnable parameters, [CLS] i For the i-th token u located in the human-computer dialogue process i The corresponding sentence's initial vector;

[0035] Step c7 involves using an LSTM network to learn the inherent logical order of the human-computer dialogue and cache contextual cues. The formula for the LSTM network to learn the inherent logical order of the dialogue is as follows:

[0036]

[0037] in, Let be the output vector of the t-th round of human-computer dialogue. Let k be the working memory for the t-th round of human-computer dialogue, and k represent the index of the context. Let be the output vector of the (t-1)th round of human-computer dialogue. The working memory for the (t-1)th round of human-computer dialogue;

[0038] Step c8: In each round of human-computer interaction dialogue, an attention mechanism is used to retrieve contextual cues related to the current user's utterance, and contextual information matching the user's utterance is obtained through weighted summation and calculation; wherein, the calculated contextual information matching the user's utterance is labeled as...

[0039]

[0040] in, The attention weights are calculated using the Softmax function, representing the nth element g in global memory. n Its importance in the current search; This represents the utterance entered by the user in the (t-1)th round of human-computer dialogue. Expression of speech With the nth element g in global memory n The correlation score between them This indicates that for element g n and words Perform a dot product operation;

[0041] Step c9 involves combining the current user utterance with the retrieved context information to form an updated semantic context vector, and then processing this vector to obtain the user sentiment of the received user dialogue text; wherein, the updated semantic context vector is labeled as... The processed user sentiment representation of the received user dialogue text is labeled as o. γ :

[0042]

[0043] in, This represents the sentiment vector within the entire human-computer interaction dialogue context. User sentiment representation (o) is the emotion vector representing the statements input by the speaker throughout the entire human-computer interaction dialogue. γ From the emotional vector and emotional vector Cascaded to obtain; A semantic vector representing contextual information describes a comprehensive representation of the semantic background in the current dialogue; The semantic vector of the current speaker's statement is represented by: M represents the model's pre-trained parameters, which contain general knowledge related to emotion reasoning and are used to reason about the input content; T represents the rules or template selection mechanism used to constrain the generation of empathic responses to ensure semantic consistency and emotion matching; Reason(·) represents the inference function, which calculates the semantic or emotion vector by combining the context or speaker information with the model parameters and rules.

[0044] Step c10: Use the Softmax function to classify the user sentiment representation of the received user dialogue text to obtain the user's sentiment category.

[0045] Furthermore, in the multimodal human-computer interaction method with empathy mechanism, in step 4, the process by which the agent outputs empathic interaction content that matches the user's future emotions to the user includes the following steps d1 to d5:

[0046] Step d1 involves performing a linear transformation on the predicted user's future emotions to obtain emotional information representing the user's future emotions; wherein the predicted user's future emotions are labeled as emo, and the emotional information corresponding to the user's future emotions emo is labeled as emo'.

[0047] Step d2 involves combining the obtained emotional information of the user at future time moments with the corresponding decoder input tag embedding to form the decoder's input sequence; wherein the embedding tags of the corresponding decoder input tags are [tag1, tag2, ..., tag1]. v-1 The input sequence of the decoder is labeled E. γ :

[0048] E γ =[tag0,tag1,tag2,…,tag v-1 ], tag0=emo';

[0049] Where v is the length of the generated response, and tag v-1 This represents the input sequence E of the decoder at the (v-1)th time step in generating the response. γ The embedding representations in the text contain information transformed from multimodal emotion recognition results and embeddings of decoder input tags, which can provide the decoder with comprehensive context and features, meeting the input requirements of the Transformer decoder, so that it can extract context vectors from the dialogue context through the multi-head cross-attention MHCAtt method, thereby generating high-quality and empathetic responses to achieve the goal of generating empathetic responses;

[0050] Step d3: Pre-build a response generator to generate empathetic interactive content; the response generator is as follows:

[0051] Φ = Y + W·MHCAtt(Y,Cont);

[0052]

[0053] Where Φ is the text representation adjusted by the cross-attention mechanism, Y is the text vector before adjustment, W are the learnable model parameters, MHCActt represents the multi-head cross-attention mechanism, and Cont represents the dialogue context. Φ represents the text representation after layer normalization, and FFN represents the feedforward neural network. It is the adjusted text vector;

[0054] Step d4: The response generator generates a probability distribution function for the next token on the vocabulary based on the adjusted text vector, and calculates the loss function for generating the response by optimizing the probability distribution function based on maximum likelihood estimation; wherein, the generated next token is labeled y. v , token y v The probability distribution function is denoted as The loss function for generating a response is defined as L gen L gen =-log[p(y v |y1,y2,…,y v-1 The next token sequence number generated here corresponds one-to-one with the time step sequence number of the generated response.

[0055] Step d5: Based on the calculated loss function for generating the response, the final joint loss function is obtained; where the joint loss function is denoted as L. g :

[0056] L g =β1·L emp +β2·L gen ;

[0057] Among them, L emp For empathy for loss, L gen The loss for generating the response, β1 is the corresponding adjustment loss L. emp The weights are parameters, and β2 is the corresponding adjustment loss L. gen The parameters of the weights.

[0058] Improved in this invention, the multimodal human-computer interaction method with empathy mechanism further includes: a process for generating a semantic discriminator; wherein the generation process of the semantic discriminator is as follows: steps e1 to e5:

[0059] Step e1 involves encoding the generated response and the gold standard response using an LSTM (Long Short-Term Memory) network, obtaining the response vector for the generated response and the response vector for the gold standard response, respectively; wherein the response vector for the generated response is labeled z. gen The response vector corresponding to the gold standard response is labeled z. gold ;

[0060] Step e2: The hidden vectors are processed through a convolutional layer to obtain convolutional features; where the convolutional features are labeled θ. hidden :

[0061] θ hidden =ReLU(z) hidden *W θ +b θ );W θ ∈R v×d b θ ∈R v×d ;

[0062] Where, θ hidden It is obtained from the hidden vector z through convolution operation hidden The extracted features are represented by ReLU, where ReLU is the activation function, * indicates a convolution operation, and W... θ and b θ These are all learnable parameters in a convolutional filter;

[0063] Step e3: Combine the obtained convolutional features with the semantic feedback vector and the dialogue context vector to obtain the semantic classification result; where:

[0064]

[0065] Among them, Z sem (z hidden ) represents the output of the semantic classifier, that is, the semantic classification result of the hidden representation; z θ This is the semantic feedback vector used to provide contextual information, i.e., the last hidden state of the LSTM encoder; W is the dialogue context vector. z and b z These are all parameters of the semantic discriminator model;

[0066] Step e4: Adjust the parameters of the semantic discriminator model to make the output distributions of the semantic discriminator model for positive and negative samples as separate as possible; where:

[0067]

[0068] in, Let n represent the loss function of the semantic discriminator, and n represent the total number of rounds of human-computer dialogue. This represents the semantic classification result of the generated response. This represents the semantic classification result of the true label, where μ is the gradient penalty coefficient. Z represents the semantic discriminator sem For the intermediate variable z t 'gradient, express The L2 norm of the gradient, where δ is a random value. Indicates the use of Z to ensure sem Gradient penalty term satisfying 1-Lipschitz constraint;

[0069] Step e5: Construct an empathy discriminator that recognizes user emotions, identifying sentiment words in the generated response, the golden response, and user feedback, so that the semantic discriminator can determine whether the generated response has empathy by recognizing these sentiment words; wherein:

[0070]

[0071] in, Let n represent the loss function of the empathy discriminator, and n represent the total number of rounds of human-computer dialogue. This indicates the empathy classification result for the generated response. This represents the empathic classification result based on the true label, where μ is the gradient penalty coefficient representing the gradient penalty strength. Z represents sem For z t 'gradient, express The L2 norm of the gradient, where δ is a random value. Indicates the use of Z to ensure emp Gradient penalty term satisfying 1-Lipschitz constraint.

[0072] Compared with the prior art, the advantages of the present invention are as follows: The multimodal human-computer interaction method with empathy mechanism of the present invention extracts the emotional features of the user in the human-computer interaction process in real time in multiple modes to obtain the emotional features corresponding to each mode, and fuses and processes the extracted emotional features to form a multimodal emotional feature vector representing the user's current emotion. Then, the multimodal emotional feature vector is processed to obtain the user's emotion at the current moment. Finally, the obtained user's emotion at the current moment is input into the dialogue model of the intelligent agent to predict the user's emotion at the future moment, and outputs the empathic interaction content that matches the user's emotion at the future moment to the user. In this way, not only can the emotional features of users during human-computer interaction be extracted from multimodal sources such as facial expression image signals, audio signals, and semantic signals of dialogue text, improving the recognition rate of users' current emotions, but it can also predict users' future emotions based on the recognized current emotions and output empathetic interactive content that matches the user's future emotions to the user. This makes the human-computer interaction process more personalized and adaptive, more emotionally resonant with users, alleviates user stress, and humanizes the human-computer interaction process, thereby improving the experience of the human-computer interaction process and enhancing the application capabilities for complex emotional interaction scenarios. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the multimodal human-computer interaction method with empathy mechanism in an embodiment of the present invention. Detailed Implementation

[0074] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0075] This embodiment provides a multimodal human-computer interaction method with an empathy mechanism, applicable to human-computer interaction systems formed by users and intelligent interactive devices. For example, the intelligent interactive device can be an interactive terminal device or an intelligent agent. An intelligent agent is an entity with autonomy, adaptability, responsiveness, and proactiveness; it can interact with the environment through processes such as perception, learning, decision-making, and execution to achieve specific goals. Intelligent agents have broad application prospects in fields such as artificial intelligence, automation, and robotics.

[0076] Specifically, see Figure 1 As shown, the multimodal human-computer interaction method with empathy mechanism in this embodiment includes the following steps 1 to 4:

[0077] Step 1: The intelligent interactive device performs multimodal real-time extraction of the user's emotional features during the human-computer interaction process, obtaining the emotional features corresponding to each modality. The multimodality includes the user's facial expression image signal, audio signal, and dialogue text semantic signal. In other words, after performing real-time extraction of emotional features for the user's facial expression image signal, audio signal, and dialogue text semantic signal, the emotional features corresponding to each model will be obtained.

[0078] For example, user audio signals can be extracted as needed using short-time Fourier transform and Mel frequency cepstral coefficients to obtain emotional features corresponding to the user audio signals. These emotional features can then be segmented into multiple segments with the same duration. All the segments obtained can then be input into a 3D ResNet-101 network for feature extraction.

[0079] The 2D ResNet-18 network is used to extract features from the user's facial expression image signal to obtain the emotional features corresponding to the user's facial expression image signal. Then, the emotional features are weighted by the spatial attention sub-network and the temporal attention network.

[0080] Step 2: The extracted emotional features are fused to form a multimodal emotional feature vector representing the user's current emotion; wherein, in this multimodal emotional feature vector, each modality has its corresponding emotional feature vector;

[0081] Step 3: Process the multimodal sentiment feature vector to obtain the user's sentiment at the current moment;

[0082] Step 4: Input the user's current emotion into the dialogue model to predict the user's future emotion, and output empathetic interaction content that matches the user's future emotion to the user.

[0083] In this embodiment, the "dialogue model" is a text generation model based on a Generative Adversarial Network (GAN) architecture. It decodes the sentiment and semantic features of the input text through generative adversarial methods, thereby achieving real-time generation of dialogue text. Specifically, the dialogue model consists of a generator and a discriminator. The generator is responsible for generating dialogue text that conforms to the contextual semantic logic and possesses empathy based on the user's input semantic and sentiment features. The discriminator is responsible for evaluating the sentiment consistency and semantic rationality of the generated text and passing the feedback information to the generator for optimization.

[0084] This dialogue model, through generative adversarial training, continuously improves the emotional accuracy and linguistic naturalness of generated dialogues, thereby generating high-quality, empathetic dialogue text that meets user needs in real time. Its core function is to predict the user's future emotional state based on their current input and generate empathetic content that matches that emotion. The construction process of this dialogue model is as follows:

[0085] First, based on a pre-trained Transformer and combined with multiple functional modules, the sentiment perception module encodes the input user multimodal information to generate the current sentiment feature vector; wherein, the sentiment feature vector contains multi-dimensional sentiment (or emotion) information such as positivity, negativity and empathy;

[0086] Then, through the sentiment prediction mechanism, the user's emotional state at future moments is predicted by combining the current sentiment feature vector, the context semantic vector, and the sentiment change trend in historical dialogues.

[0087] Finally, a generation constraint module is added to the generation stage of the dialogue model. By controlling the decoding process, this module ensures that the generated content is semantically consistent with the context and emotionally matches the predicted future emotional state of the user. The training data for the dialogue model comes from a high-quality multi-turn empathic dialogue dataset, covering various emotional scenarios and user intentions, to ensure that the model possesses powerful emotion recognition, prediction, and generation capabilities.

[0088] Specifically, in step 2 of this embodiment, the process of fusing the various emotional features to form the multimodal emotional feature vector includes the following steps a1 to a3:

[0089] Step a1: A feature-level fusion method is used to fuse the emotional features extracted from facial expression image signals and the emotional features extracted from user audio signals to obtain the user's non-verbal emotional features. The feature-level fusion method is a mature and conventional technique in this field, and the processing procedure of the feature-level fusion method will not be described in detail here.

[0090] Step a2: Extract semantic features from the semantic signal of the user's dialogue text, and combine these semantic features with a sentiment dictionary to generate sentiment tags, thus obtaining the user's language sentiment text features; wherein, in this embodiment, the sentiment tag generation process is as follows:

[0091] First, the dialogue model extracts semantic features from the user's dialogue text; these extracted features are high-dimensional vectors generated by encoding through the pre-trained language model XLNet, which can represent the semantic information of the text.

[0092] Then, the dialogue model uses entries in the sentiment dictionary to match semantic features. The sentiment dictionary contains a large number of sentiment-related keywords or phrases, and each entry is assigned a corresponding sentiment category. The dialogue model calculates the similarity between the semantic feature vector of the user text and the sentiment entry vector in the dictionary, selects the entry with the highest matching degree, and combines the sentiment category corresponding to the entry to assign one or more sentiment tags to the user text.

[0093] Finally, based on the intensity of the emotional signals appearing in the semantic features and the contextual information, a sentiment label suitable for user input is generated, and this sentiment label is used as the final representation of the user's language emotional text features.

[0094] Step a3 involves fusing the obtained nonverbal emotional features and verbal emotional text features of the user to obtain the multimodal emotional feature vector. In this embodiment, since the semantic signal content of the obtained user dialogue text and the real-time user audio signal and facial expression image signal cannot be perfectly aligned temporally, a decision-level fusion method is employed. The specific process of this decision-level fusion method is as follows:

[0095] First, a pre-trained language model is used to extract the sentiment vector of the linguistic sentiment text features, and a sentiment classifier is used to output the sentiment category and its confidence distribution. Specifically, for non-linguistic sentiment features, a speech analysis model is used to extract the sentiment vector of speech intonation and output the corresponding sentiment category and confidence distribution. In addition, an expression recognition model is used to extract the sentiment vector of facial expressions and output the corresponding sentiment category and confidence distribution.

[0096] Then, a dynamic weight allocation strategy is constructed to adjust the weights according to the reliability of modal information. For example, when the semantics of user dialogue text are clear, the weight of verbal emotion is increased, and when the user's audio or facial expression image signal is blurry, the weight of nonverbal emotion is reduced.

[0097] Finally, the final multimodal sentiment category is calculated by weighted summation and fusion of the sentiment confidence distributions of the user's audio signal and facial expression image signal. The weight allocation and fusion calculation dynamically adapt to the quality of different modalities; the fused result is output as the sentiment category with the highest confidence, ensuring the integration of sentiment information from both verbal and non-verbal modalities, thus improving the accuracy and robustness of sentiment prediction.

[0098] Furthermore, in the multimodal human-computer interaction method with empathy mechanism in this embodiment, step 4, the process of processing the multimodal emotion feature vector to predict the user's future emotions, includes the following steps b1 to b3:

[0099] Step b1: The emotional feature vector extracted from the user's facial image signal and the emotional feature vector extracted from the user's audio signal are concatenated to obtain an aggregated semantic vector.

[0100] In this embodiment, the process of concatenating the two emotional feature vectors includes the following steps: First, facial image feature vectors are extracted using a convolutional neural network, and audio feature vectors are extracted using a pre-trained Wav2Vec model to ensure that the dimensions of the two feature vectors are fixed; then, the extracted facial image feature vectors and audio feature vectors are directly concatenated in dimensional order, that is, the audio features are appended to the facial features to generate a new aggregate vector, thus realizing the process of concatenating the two emotional feature vectors.

[0101] Step b2 involves inputting the aggregated semantic vector into a fully connected layer for processing. Then, a cross-entropy loss function with consistency constraints is used to train sentiment classification on the processed semantic vector. The fully connected layer performs feature integration and non-linear transformation on the concatenated aggregated vector, while reweighting features between modalities to enhance the interactivity of multimodal features. In this embodiment, the fully connected layer structure includes a weight matrix and a bias vector.

[0102] In this embodiment, the process of inputting the aggregated semantic vector into the fully connected layer is as follows: First, the concatenated aggregated vector is input into the fully connected layer, where matrix multiplication and bias addition operations are performed to linearly transform the concatenated features. Then, the ReLU activation function is used to perform nonlinear processing on the result of the linear transformation to capture the complex relationships between modalities. The dimension of the output feature vector is adjusted to a fixed value. This dimensionality reduction operation eliminates redundant features while preserving key information. Finally, a batch normalization layer is added after the fully connected layer to normalize the distribution of the output vector, thereby accelerating the model convergence speed and preventing overfitting.

[0103] In this embodiment, the cross-entropy loss function with consistency constraints is a loss function that combines cross-entropy loss and consistency constraints. It aims to ensure the classification performance of each modality's output results in the multimodal sentiment prediction task, while maintaining the consistency of sentiment polarity of the prediction results between modalities. Its basic principle is to reduce the conflict between different modalities in sentiment prediction by optimizing the classification task for each modality separately.

[0104] Specifically, the process of training sentiment classification on the semantic vectors processed by the fully connected layer using the cross-entropy loss function with consistency constraints is as follows:

[0105] First, the cross-entropy loss part optimizes the output prediction probability and the true label for each modality, measuring the difference between the class distribution predicted by the model and the true distribution;

[0106] Secondly, the consistency constraint part calculates the similarity of the predicted probability distributions of different modalities to ensure that the sentiment polarity of each modality remains consistent. The consistency constraint is achieved by calculating the KL divergence or cosine similarity between modalities. The KL divergence guides the model optimization by measuring the difference between the predicted distributions of two modalities, while the cosine similarity measures the similarity between modalities by calculating the cosine value of the angle between the vectors.

[0107] In practical calculations, cross-entropy loss is used to improve the classification performance of each modality, while consistency constraints are embedded into the total loss function in a weighted form to balance the optimization objectives of classification task and inter-modality consistency. The construction of the total loss function includes summing the cross-entropy losses for each modality and calculating the consistency loss for the prediction distributions of all modality combinations and summing them in a weighted manner. The weight parameters are used to adjust the influence ratio between classification performance and consistency constraints. Through this combined design, polarity consistency cross-entropy loss can simultaneously improve the classification ability of individual modalities and the synergistic effect between multiple modalities, effectively reducing the probability of prediction conflicts between modalities and improving the overall accuracy and robustness of the multimodal sentiment analysis model.

[0108] Step b3: The trained dialogue model is used to process the input multimodal sentiment feature vector, and the output of the dialogue model is used as the predicted sentiment of the user at future time.

[0109] It should be noted that in the multimodal human-computer interaction method with empathy mechanism in this embodiment, the process of obtaining the user's current emotion by processing the emotional feature vector corresponding to the semantic signal modality of the user's dialogue text in step 3 includes the following steps c1 to c10:

[0110] Step c1: Perform word embedding processing on the received user's dialogue text semantic signal; wherein, the received user's dialogue text semantic signal is denoted as U=[CLS,u1,u2,…,u N ], CLS represents the aggregate semantic information of the entire statement, u i Let u represent the i-th token, and use word embedding and position embedding to embed each token u respectively. i The mapping is represented as a vector, 1≤i≤N, where N represents the total number of tokens; token u i Word embedding representation as token u i Position embedding is represented as d is the dimension of the word vector;

[0111] Step c2, embed the speaker identification vector into each token representation; wherein, the speaker identification vector is embedded into token u i The text is marked as

[0112] Step c3: Based on the word embedding, position embedding, and speaker identification vector embedding obtained from the embedding, the final semantic embedding of each token is obtained; where the i-th token u i The final semantic embedding tag is s i ;

[0113]

[0114] Step c4: Encode the semantic context information of each token using a multi-layered stacked Transformer encoder, and calculate the hidden vector and final representation of each token; where:

[0115]

[0116] Among them, h i Indicates token u i The hidden vectors, where LayerNorm represents the layer normalization used to standardize the input features. Let MHAtt represent the input features of the i-th token at layer i-1 in the Transformer network, and MHAtt represent the multi-head self-attention mechanism. This represents the i-th token in the (i-1)-th layer. This represents the output of the i-th token after passing through the multi-head self-attention mechanism. Indicates residual connection, Let represent the final output vector of the i-th token in the l-th layer of the Transformer network; FNN represents a two-layer feedforward neural network with ReLU activation function; L represents the total number of layers in the Transformer network.

[0117] It should be noted that in the Transformer architecture, after processing by multiple stacked Transformer encoders, each input token generates a corresponding context encoding vector, called the "final representation" of that token. This "final representation" retains the token's own characteristics and captures its relationship with other tokens in the sequence through a self-attention mechanism, thus possessing the ability to express global semantic contextual information;

[0118] Step c5: Generate a contextual representation that represents the semantic context for emotion recognition; wherein, the contextual representation is denoted as Ω:

[0119] Ω=[ω1,ω2,…,ω N ];

[0120] Where, ω N It is the final feature vector of the Nth token in the sequence, which is also the final representation of the Nth sentence. The main information of the final representation of each sentence is stored in the sentence beginning vector.

[0121] Step c6: Based on the generated contextual representation, a self-attention mechanism is used to capture global emotional information during each round of human-computer dialogue; wherein:

[0122] c i =selfAtt([CLS]) i );

[0123] m i =Wc i +b;

[0124] Among them, c i For the i-th token u i The sentiment context vector, selfAtt represents the self-attention mechanism, m i For the sentiment context vector c i Mapped to a learnable sentiment representation; W and b are learnable parameters, [CLS] i For the i-th token u located in the human-computer dialogue process i The vector corresponding to the beginning of a sentence;

[0125] Step c7 involves learning the inherent logical order of the human-computer dialogue using an LSTM (Long Short-Term Memory) network and caching contextual cues. The formula for the LSTM network to learn the inherent logical order of the dialogue is as follows:

[0126]

[0127] in, Let be the output vector of the t-th round of human-computer dialogue. Let k be the working memory for the t-th round of human-computer dialogue, and k represent the index of the context. Let be the output vector of the (t-1)th round of human-computer dialogue. The working memory for the (t-1)th round of human-computer dialogue;

[0128] Step c8: In each round of human-computer interaction dialogue, an attention mechanism is used to retrieve contextual cues related to the current user's utterance, and contextual information matching the user's utterance is obtained through weighted summation and calculation; wherein, the calculated contextual information matching the user's utterance is labeled as...

[0129]

[0130] in, The attention weights are calculated using the Softmax function, representing the nth element g in global memory. n Its importance in the current search; This represents the utterance entered by the user in the (t-1)th round of human-computer dialogue. Expression of speech With the nth element g in global memory n The correlation score between them This indicates that for element g n and words Perform a dot product operation;

[0131] Step c9 involves combining the current user utterance with the retrieved context information to form an updated semantic context vector, and then processing this vector to obtain the user sentiment of the received user dialogue text; wherein, the updated semantic context vector is labeled as... The processed user sentiment representation of the received user dialogue text is labeled as o. γ :

[0132]

[0133] in, This represents the sentiment vector within the entire human-computer interaction dialogue context. User sentiment representation (o) is the emotion vector representing the statements input by the speaker throughout the entire human-computer interaction dialogue. γ From the emotional vector and emotional vector Cascaded to obtain; A semantic vector representing contextual information describes a comprehensive representation of the semantic background in the current dialogue; The semantic vector represents the current speaker's statement, describing the semantic content or emotional features of the input sentence from the current interlocutor (e.g., user or agent); M represents the model's pre-trained parameters, which contain general knowledge related to emotion reasoning and are used to reason about the input content; T represents the rules or template selection mechanism used to constrain the generation of empathic responses to ensure semantic consistency and emotion matching; Reason(·) represents the inference function, a conventional technique in this field, which calculates the semantic or emotion vector by combining context or speaker information with model parameters and rules; that is, Indicates based on M and T infer the corresponding sentiment vector from the entire human-computer interaction dialogue context; Indicates based on M and T infer the corresponding sentiment vector from the semantic vector of the current speaker's statement;

[0134] Step c10: Use the Softmax function to classify the user sentiment representation of the received user dialogue text to obtain the user's sentiment category.

[0135] Furthermore, regarding the multimodal human-computer interaction method with empathy mechanism in this embodiment, in step 4, the process of the agent outputting empathic interaction content that matches the user's future emotions to the user includes the following steps d1 to d5:

[0136] Step d1 involves performing a linear transformation on the predicted user's future emotions to obtain emotional information representing the user's future emotions; wherein the predicted user's future emotions are labeled as emo, and the emotional information corresponding to the user's future emotions emo is labeled as emo'.

[0137] It should be noted that, in this embodiment, the process of linearly transforming the predicted user's future emotions is as follows:

[0138] First, determine the minimum value of the predicted sentiment score emo. min and maximum value emo max And the minimum value of the target sentiment information emo′. min and the maximum value emo′ max ;

[0139] Secondly, the predicted sentiment score emo and its minimum value emo are calculated. min The difference between the target and target emotional information ranges is used to obtain a new emotional information value, emo′. The formula for calculating the new emotional information value emo′ is as follows:

[0140]

[0141] Using the above formula for calculating the new emotional information value emo′, the predicted emotional score emo can be mapped to the target emotional information range, thereby generating the new emotional information value emo′, which better represents the user's emotional state at future moments.

[0142] The core of this process is to map the input sentiment score to the target sentiment information range according to a certain proportional relationship, so that the transformed sentiment information can effectively represent the user's sentiment state at future moments.

[0143] Step d2 involves combining the obtained emotional information of the user at future time moments with the corresponding decoder input tag embedding to form the decoder's input sequence; wherein the embedding tags of the corresponding decoder input tags are [tag1, tag2, ..., tag1]. v-1 The input sequence of the decoder is labeled E. γ :

[0144] E γ =[tag0,tag1,tag2,…,tag v-1 ], tag0=emo';

[0145] Where v is the length of the generated response, and tag v-1 This represents the input sequence E of the decoder at the (v-1)th time step in generating the response. γ The embedding representations in the text contain the information after the transformation of the multimodal emotion recognition results and the embeddings of the decoder input labels. They can provide the decoder with comprehensive context and features, which meets the requirements of the Transformer decoder for input, so that it can extract context vectors from the dialogue context through the multi-head cross-attention MHCATt method, and then generate high-quality and empathetic responses to achieve the goal of generating empathetic responses.

[0146] The input sequence of the decoder contains two parts of information: one part is the sentiment information obtained through sentiment prediction, and the other part is the embedded input tags. This ensures that the decoder not only takes into account the sentiment information, but also combines other contextual information to generate appropriate output.

[0147] Step d3: Pre-build a response generator to generate empathetic interactive content; the response generator is as follows:

[0148] Φ = Y + W·MHCAtt(Y,Cont);

[0149]

[0150] Where Φ is the text representation adjusted by the cross-attention mechanism, Y is the text vector before adjustment, W are the learnable model parameters, MHCActt represents the multi-head cross-attention mechanism, and Cont represents the dialogue context. Φ represents the text representation after layer normalization, and FFN represents the feedforward neural network. It is the adjusted text vector;

[0151] Step d4: The response generator generates a probability distribution function for the next token on the vocabulary based on the adjusted text vector, and calculates the loss function for generating the response by optimizing the probability distribution function based on maximum likelihood estimation; wherein, the generated next token is labeled y. v , token y v The probability distribution function is denoted as The loss function for generating a response is defined as L gen L gen =-log[p(y v |y1,y2,…,y v-1 The next token sequence number generated here corresponds one-to-one with the time step sequence number of the generated response.

[0152] Step d5: Based on the calculated loss function for generating the response, the final joint loss function is obtained; where the joint loss function is denoted as L. g :

[0153] L g =β1·L emp +β2·L gen ;

[0154] Among them, L emp For the empathy loss function, L gen The loss function for generating the response, β1 is the corresponding adjustment loss function L. emp The weights are parameters, and β2 is the corresponding adjustment loss function L. gen The parameters of the weights.

[0155] Of course, based on actual human-computer interaction needs, this embodiment of the multimodal human-computer interaction method with empathy mechanism also includes a process for generating a semantic discriminator. The generation process of the semantic discriminator is as follows: steps e1 to e5:

[0156] Step e1: Generate responses and gold standard responses using LSTM encoding, obtaining the response vectors for the generated responses and the gold standard responses, respectively; wherein, the response vector for the generated responses is labeled z. gen The response vector corresponding to the gold standard response is labeled z. gold ;

[0157] Step e2: The hidden vectors are processed through a convolutional layer to obtain convolutional features; where the convolutional features are labeled θ. hidden :

[0158] θ hidden =ReLU(z) hidden *W θ +b θ );W θ ∈R v×db θ ∈R v×d ;

[0159] Where, θ hidden It is obtained from the hidden vector z through convolution operation hidden The extracted features are represented by ReLU, where ReLU is the activation function, * indicates a convolution operation, and W... θ and b θ These are all learnable parameters in a convolutional filter;

[0160] Step e3: Combine the obtained convolutional features with the semantic feedback vector and the dialogue context vector to obtain the semantic classification result; where:

[0161]

[0162] Among them, Z sem (z hidden ) represents the output of the semantic classifier, that is, the semantic classification result of the hidden representation; z θ This is the semantic feedback vector used to provide contextual information, i.e., the last hidden state of the LSTM encoder; W is the dialogue context vector. z and b z These are all parameters of the semantic discriminator model;

[0163] It should be noted that the dialogue model in this embodiment is built on a generative adversarial network (GAN), and the semantic feedback vector is calculated by a semantic discriminator. In this architecture, the generator is responsible for generating the dialogue response, and the semantic discriminator is used to determine whether the generated response meets the expected semantic and emotional requirements. Specifically, the semantic discriminator analyzes the dialogue response generated by the generator to calculate a vector representing the current semantic and emotional state of the dialogue; this vector is the "semantic feedback vector." The semantic feedback vector reflects the semantic information in the dialogue and provides feedback to the generator on the quality of the response, thereby further optimizing the dialogue generation process. This semantic feedback vector, combined with other contextual information, is ultimately used to generate dialogue content that better meets semantic requirements.

[0164] Step e4: Adjust the parameters of the semantic discriminator model to make the output distributions of the semantic discriminator model for positive and negative samples as separate as possible; where:

[0165]

[0166] in, Z represents the semantic discriminator sem The loss function, where n represents the total number of rounds of human-computer dialogue. This represents the semantic classification result of the generated response. This represents the semantic classification result of the true label, where μ is the gradient penalty coefficient. Z represents the semantic discriminator sem For the intermediate variable z t 'gradient, express The L2 norm of the gradient, where δ is a random value. Indicates the use of Z to ensure sem Gradient penalty term satisfying 1-Lipschitz constraint;

[0167] It should be noted that the intermediate variable z t 'It is obtained by interpolating between generated samples and real samples through linear combination;

[0168] Furthermore, in this embodiment, the positive samples refer to genuine dialogue responses that conform to the expected semantics, while the negative samples refer to dialogue responses generated by the generator that may not conform to the expected semantics. The task of the semantic discriminator is to distinguish between these positive and negative samples. By adjusting the model parameters, it aims to separate the output distribution of positive samples (i.e., genuine responses) and negative samples (i.e., generated responses) as much as possible, thereby improving the generator's ability to generate more natural and semantically consistent responses.

[0169] Step e5: Construct an empathy discriminator that recognizes user emotions, identifying sentiment words in the generated response, the golden response, and user feedback, so that the semantic discriminator can determine whether the generated response has empathy by recognizing these sentiment words; wherein:

[0170]

[0171] in, Z represents the empathy discriminator sem The loss function, where n represents the total number of rounds of human-computer dialogue. This indicates the empathy classification result for the generated response. This represents the empathic classification result based on the true label, where μ is the gradient penalty coefficient representing the gradient penalty strength. Z represents the empathy discriminator sem For the intermediate variable z t 'gradient, express The L2 norm of the gradient, where δ is a random value. Indicates the use of Z to ensure emp Gradient penalty term satisfying 1-Lipschitz constraint (a conventional constraint in this field).

[0172] Although preferred embodiments of the present invention have been described in detail above, it should be clearly understood that various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal human-computer interaction method with an empathy mechanism, characterized in that, Includes the following steps: Step 1: Perform multimodal real-time extraction of the user's emotional features during the human-computer interaction process to obtain the emotional features corresponding to each modality; wherein, the multimodality includes the user's facial expression image signal, audio signal and dialogue text semantic signal; Step 2: The extracted emotional features are fused to form a multimodal emotional feature vector representing the user's current emotion; wherein, in this multimodal emotional feature vector, each modality has its corresponding emotional feature vector; Step 3: Process the multimodal sentiment feature vector to obtain the user's sentiment at the current moment; Step 4 involves inputting the user's current emotional state into the dialogue model to predict the user's future emotional state and outputting empathetic interaction content that matches the user's future emotional state. The dialogue model consists of a generator and a discriminator. The generator is responsible for generating dialogue text that conforms to the contextual semantic logic and possesses empathy based on the user's input semantics and emotional features. The discriminator is responsible for evaluating the emotional consistency and semantic rationality of the generated text and passing feedback information to the generator for optimization. The construction process of this dialogue model includes the following steps: Step 41: Based on the pre-trained Transformer and combined with multiple functional modules, the sentiment perception module encodes the input user multimodal information to generate the sentiment feature vector at the current moment; wherein, the sentiment feature vector contains positive, negative and empathy information; Step 42: Through the sentiment prediction mechanism, combining the current sentiment feature vector, the context semantic vector, and the sentiment change trend in historical dialogues, predict the user's sentiment state at future moments. Step 43: In the generation stage, the dialogue model incorporates a generation constraint module to control the decoding process, ensuring that the generated content is semantically consistent with the context and emotionally matches the predicted future emotional state of the user. The training data for the dialogue model comes from a multi-turn empathetic dialogue dataset that covers various emotional scenarios and user intentions.

2. The multimodal human-computer interaction method with empathy mechanism according to claim 1, characterized in that, In step 2, the process of fusing the various emotional features to form the multimodal emotional feature vector includes the following steps a1 to a3: Step a1: The feature-level fusion method is used to fuse the emotional features extracted from facial expression image signals and the emotional features extracted from user audio signals to obtain the user's non-verbal emotional features. Step a2: Extract semantic features from the semantic signals of the user's dialogue text, and combine the semantic features with the sentiment dictionary to generate sentiment tags, thereby obtaining the user's language sentiment text features; Step a3 involves fusing the obtained non-verbal emotional features and verbal emotional text features of the user to obtain the multimodal emotional feature vector.

3. The multimodal human-computer interaction method with empathy mechanism according to claim 2, characterized in that, In step 4, the process of processing the multimodal sentiment feature vector to predict the user's future sentiment includes the following steps b1 to b3: Step b1: The emotional feature vector extracted from the user's facial image signal and the emotional feature vector extracted from the user's audio signal are concatenated to obtain an aggregated semantic vector. Step b2: Input the aggregated semantic vector into the fully connected layer of the convolutional neural network for processing, and then use the polarity consistency cross-entropy loss function to perform sentiment classification training on the semantic vector after processing by the fully connected layer. Step b3: The trained dialogue model is used to process the input multimodal sentiment feature vector, and the output of the dialogue model is used as the predicted sentiment of the user at future time.

4. The multimodal human-computer interaction method with empathy mechanism according to claim 3, characterized in that, In step 3, the process of obtaining the user's current emotion by processing the emotion feature vector corresponding to the semantic signal modality of the user's dialogue text includes the following steps c1 to c10: Step c1: Perform word embedding processing on the received user's dialogue text semantic signal; wherein, the received user's dialogue text semantic signal is denoted as U=[CLS,u1,u2,…,u N ], CLS represents the aggregate semantic information of the entire statement, u i Let u represent the i-th token, and use word embedding and position embedding to embed each token u respectively. i The mapping is represented as a vector, 1≤i≤N, where N represents the total number of tokens; token u i Word embedding representation as token u i Position embedding is represented as d is the dimension of the word vector; Step c2, embed the speaker identification vector into each token representation; wherein, the speaker identification vector is embedded into token u i The text is marked as Step c3: Based on the word embedding, position embedding, and speaker identification vector embedding obtained from the embedding, the final semantic embedding of each token is obtained; where the i-th token u i The final semantic embedding tag is s i ; Step c4: Encode the semantic context information of each token using a multi-layered stacked Transformer encoder, and calculate the hidden vector and final representation of each token; where: Among them, h i Indicates token u i The hidden vectors, where LayerNorm represents the layer normalization used to standardize the input features. Let MHAtt represent the input features of the i-th token at layer i-1 in the Transformer network, and MHAtt represent the multi-head self-attention mechanism. This represents the i-th token in the (i-1)-th layer. This represents the output of the i-th token after passing through the multi-head self-attention mechanism. Indicates residual connection, Let represent the final output vector of the i-th token in the l-th layer of the Transformer network; FNN represents a two-layer feedforward neural network with ReLU activation function; L represents the total number of layers in the Transformer network. Step c5: Generate a contextual representation that represents the semantic context for emotion recognition; wherein, the contextual representation is denoted as Ω: Ω=[ω1,ω2,…,ω N ]; Where, ω N It is the final feature vector of the Nth token in the sequence; Step c6: Based on the generated contextual representation, a self-attention mechanism is used to capture global emotional information during each round of human-computer dialogue; wherein: w i =selfAtt([CLS] i ); m i =Wc i +b; Among them, c i For the i-th token u i The sentiment context vector, selfAtt represents the self-attention mechanism, m i For the sentiment context vector c i Mapped to a learnable sentiment representation; W and b are learnable parameters, [CLS] i For the i-th token u located in the human-computer dialogue process i The vector corresponding to the beginning of a sentence; Step c7 involves using an LSTM network to learn the inherent logical order of the human-computer dialogue and cache contextual cues. The formula for the LSTM network to learn the inherent logical order of the dialogue is as follows: in, Let be the output vector of the t-th round of human-computer dialogue. Let k be the working memory for the t-th round of human-computer dialogue, and k represent the index of the context. Let be the output vector of the (t-1)th round of human-computer dialogue. The working memory for the (t-1)th round of human-computer dialogue; Step c8: In each round of human-computer interaction dialogue, an attention mechanism is used to retrieve contextual cues related to the current user's utterance, and contextual information matching the user's utterance is obtained through weighted summation and calculation; wherein, the calculated contextual information matching the user's utterance is labeled as... in, The attention weights are calculated using the Softmax function, representing the nth element g in global memory. n Its importance in the current search; This represents the utterance entered by the user in the (t-1)th round of human-computer dialogue. Expression of speech With the nth element g in global memory n The correlation score between them This indicates that for element g n and words Perform a dot product operation; Step c9 involves combining the current user utterance with the retrieved context information to form an updated semantic context vector, and then processing this vector to obtain the user sentiment of the received user dialogue text; wherein, the updated semantic context vector is labeled as... The processed user sentiment representation of the received user dialogue text is labeled as o. γ : in, This represents the sentiment vector within the entire human-computer interaction dialogue context. User sentiment representation (o) is the emotion vector representing the statements input by the speaker throughout the entire human-computer interaction dialogue. γ From the emotional vector and emotional vector Cascaded to obtain; A semantic vector representing contextual information; The semantic vector of the current speaker's statement is represented; M represents the pre-training parameters of the model; T represents the rules or template selection mechanism used to constrain the generation of empathic responses to ensure semantic consistency and sentiment matching; Reason(·) represents the inference function. Step c10: Use the Softmax function to classify the user sentiment representation of the received user dialogue text to obtain the user's sentiment category.

5. The multimodal human-computer interaction method with empathy mechanism according to claim 4, characterized in that, In step 4, the process of outputting empathetic interaction content that matches the user's future emotions to the user includes the following steps d1 to d5: Step d1 involves performing a linear transformation on the predicted user's future emotions to obtain emotional information representing the user's future emotions; wherein the predicted user's future emotions are labeled as emo, and the emotional information corresponding to the user's future emotions emo is labeled as emo'. Step d2 involves combining the obtained emotional information of the user at future time moments with the corresponding decoder input tag embedding to form the decoder's input sequence; wherein the embedding tags of the corresponding decoder input tags are [tag1, tag2, ..., tag1]. v-1 The input sequence of the decoder is labeled E. γ : IS γ =[tag0,tag1,tag2,...,tag v-1 ],tag0=emo': Where v is the length of the generated response, and tag v-1 This represents the input sequence E of the decoder at the (v-1)th time step in generating the response. γ Embedded representation in; Step d3: Pre-build a response generator to generate empathetic interactive content; the response generator is as follows: Φ = Y + W·MHCAtt(Y,Cont); Where Φ is the text representation adjusted by the cross-attention mechanism, Y is the text vector before adjustment, W are the learnable model parameters, MHCActt represents the multi-head cross-attention mechanism, and Cont represents the dialogue context. Φ represents the text representation after layer normalization, and FFN represents the feedforward neural network. It is the adjusted text vector; Step d4: The response generator generates a probability distribution function for the next token on the vocabulary based on the adjusted text vector, and calculates the loss function for generating the response by optimizing the probability distribution function based on maximum likelihood estimation; wherein, the generated next token is labeled y. v , token y v The probability distribution function is denoted as The loss function for generating a response is defined as L gen L gen =-log[p(y v |y1,y2,…,y v-1 The generated next token sequence number corresponds one-to-one with the time step sequence number of the generated response. Step d5: Based on the calculated loss function for generating the response, the final joint loss function is obtained; where the joint loss function is denoted as L. g : L g =β1·L emp +β2·L gen ; Among them, L emp For the empathy loss function, L gen The loss function for generating the response, β1 is the corresponding adjustment loss function L. emp The weights are parameters, and β2 is the corresponding adjustment loss function L. gen The parameters of the weights.

6. The multimodal human-computer interaction method with empathy mechanism according to claim 5, characterized in that, It also includes the process of generating a semantic discriminator.

Images