Intelligent agent common-situation response method based on brain-computer coupling under guidance of emotion of learner

Abstract

The invention relates to the field of cross fusion of artificial intelligence and brain science, and particularly discloses a brain-computer coupling-based agent common-situation response method under the guidance of the emotion of a learner, and the method comprises the steps: firstly, collecting the multi-modal emotion data of the learner oriented to the agent common-situation response; secondly, performing emotion analysis on the acquired multi-modal data of the learner under brain-computer coupling by adopting a multi-modal data emotional state judgment method; and designing a multi-modal common-situation response generation framework based on the emotion recognition result and the dialogue content, constructing a common-situation response model based on multi-modal representation, and carrying out joint learning training on the common-situation response model to form an agent common-situation response model based on a large language model multi-agent system under the guidance of the learner emotion. The emotional state of the learner is accurately identified, and the dialogue content of the intelligent agent is dynamically adjusted based on the emotional state, so that the emotional ability cultivation of the learner is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of the cross-integration of artificial intelligence and brain science, and specifically to an intelligent agent empathic response method based on brain-computer coupling under the guidance of learners' emotions. Background Art

[0002] In the current educational environment, learners not only need to master subject knowledge, but also need to have social emotional skills and empathy in order to achieve better learning results in increasingly complex learning tasks. Empathy, as the ability of an individual to understand, perceive and respond appropriately to the emotions of others in social interactions, is one of the core elements of collaborative learning. Studies have shown that empathy can effectively promote trust, cooperation and understanding between individuals and others, thereby enhancing learners' depth of thinking and teamwork ability. Therefore, how to cultivate learners' empathy has become the key to improving learning outcomes.

[0003] However, the current focus of human-computer collaborative education is mainly on knowledge transfer, and less on how to help learners develop social emotional abilities, especially empathy. Although traditional intelligent agents can perform certain emotional recognition and feedback in their interactions with learners, they are usually limited to single-modal emotional expression and lack sufficient emotional understanding. More importantly, existing intelligent agents often cannot accurately identify learners' emotional states in different situations, nor can they generate appropriate empathetic feedback based on their emotional states, resulting in their failure to effectively promote the cultivation of learners' empathy in the collaborative learning process. In addition, with the advancement of artificial intelligence and brain-computer interface technology, intelligent agents should be able to perceive and feedback multimodal emotional information, including visual, auditory and even physiological signals, in their interactions with learners. This provides a new direction for the design of empathetic intelligent agents, but how to effectively combine these technologies to generate dialogue content that matches learners' emotional states is still a difficulty in current technology. Summary of the invention

[0004] In response to the above-mentioned problems in the prior art, the present invention proposes an empathic response method of an intelligent agent based on brain-computer coupling under the guidance of learners' emotions, which can accurately identify the learners' emotional state and dynamically adjust the dialogue content of the intelligent agent based on the emotional state, effectively improving the learners' emotional cognitive ability in the collaborative learning process and promoting the development of their social and emotional intelligence.

[0005] To achieve the above objectives, the present invention proposes an intelligent agent empathy response method based on brain-computer coupling under the guidance of learner emotions, and the specific steps are as follows: S1. Collect multimodal emotional data of learners for agent-oriented empathic response, where the collected multimodal emotional data includes learners' brain data for emotional empathy and dialogue data for agent collaboration; S2. Use the emotion state discrimination method of multimodal data to perform emotion analysis on the multimodal data of learners collected under brain-computer coupling. The emotion analysis is performed from three parts: facial emotion analysis of learners based on brain-computer coupling learning method, emotion recognition of learners' dialogue text based on second-order interactive attention mechanism, and emotion state discrimination of learners integrating brain-vision-language multimodal data; S3. Design a multimodal empathy response generation framework based on emotion recognition results and conversation content, construct an empathy response model based on multimodal representation, and conduct joint learning and training of the empathy response model to form an intelligent agent empathy response model based on a large language model multi-agent system under the guidance of learners' emotions.

[0006] Preferably, in S1, the specific steps of collecting learner brain data for emotional empathy are: S11. Collect learners’ brain data in a laboratory environment and let them watch different emotional images in the dataset to obtain EEG signals. S12. After EEG signal acquisition, invalid segments and artifacts were removed, and the frequency band of 1 Hz-75 Hz was filtered using a Butterworth filter.

[0007] Preferably, in S1, the specific steps of collecting dialogue data for agent collaboration are as follows: after completing the staged classroom learning task, the learner conducts self-perception of the learning content and learning emotions, reports to the agent, and engages in dialogue and communication with the agent, and records the dialogue context data set as , the dialogue situation is recorded as , the intelligent agent model understands the emotions in the conversation ;in, Indicates Speeches, including words, situations Indicated by The situation description consists of words, where w q Provides contextual background information.

[0008] Preferably, in S2, the specific steps of analyzing learner's facial emotions based on brain-computer coupling learning method are as follows: S211, use EEGNet and CNNNet to perform preliminary feature extraction on facial expression images and EEG signals; Among them, for facial expression images in the visual field, an improved convolutional neural network CNNNet is used to extract preliminary features and preliminary representation of the visual field : ; In the formula, is the improved convolutional neural network CNNNet, V is a given visual image, For the initial representation of the visual field; For EEG signals, the feature extractor of EEGNet is used to perform preliminary feature extraction and preliminary representation of cognitive domains : ; In the formula, EEGNet is a compact convolutional neural network designed for EEG brain-computer interface. R is a given EEG signal. It is a preliminary representation of the cognitive domain; S212, construct a cognitive-visual learning framework, project the obtained preliminary representation to the public channel and the private channel, and encode the public channel and the private channel representation; Among them, the public channel adopts the encoding function of shared parameters , learning to capture the common information between cognitive and visual fields. After training, the common channel extracts the shared features of cognitive and visual fields. Given and , then the common representation of cognitive and visual domain modalities is: ; ; In the formula, is an encoding function based on a simple fully connected neural layer, for shared parameters in cognitive and visual domains; The private representation is output by the private channels in the cognitive and visual domains, capturing the private information related to the cognitive and visual domains. and , then the private representation is: ; ; In the formula, is the private encoding function of the private channel in the visual field, is the private encoding function of the private channel in the cognitive domain, This is achieved through fully connected neural layers. is the parameter of the private encoding function of the private channel in the visual field, are the parameters of the private encoding function of the private channel of the cognitive domain; S213, after obtaining the public representation and the private representation, performing a splicing operation on the obtained public representation and the private representation, and performing an emotion recognition task; Among them, the concatenated representations of the public and private representations in the visual and cognitive domains are defined as: ; ; In the formula, It is the spliced ​​representation of the visual field. It is a representation after splicing of cognitive domains; The expressions of emotion recognition tasks in the visual domain and cognitive domain are: ; ; In the formula, and is the predicted label of the image and EEG signal, using KNN as the function Model.

[0009] Preferably, in S2, the specific steps of emotion recognition of learner dialogue text based on the second-order interactive attention mechanism are as follows: S221, constructing explicit information pairs and implicit information pairs based on the learners' learning emotions and learning experience reports as inputs, and sending them to the information association module; S222, encode the dialogue information, regard the dialogue speech and situation between the learner and the agent as explicit information, regard the reasoning knowledge as implicit information, and process the explicit information and implicit information to obtain the semantic representation of the dialogue speech representation, the situation representation and the reasoning knowledge; S223, using the information association module to capture important associated words in explicit and implicit information, constructing an associated matrix, first-order interactive attention, second-order interactive attention, and storing associated words in memory, to obtain an associated representation, the expression of which is: ; In the formula, For association representation, For related information, , is the number of associated words in the memory, For encoder; S224, input the dialogue speech representation, situation representation, association representation and reasoning knowledge representation into the aggregation network to obtain the emotion representation, and use the emotion representation to predict the emotion probability respectively; wherein the expression of the emotion probability is: ; ; ; ; In the formula, , represents the emotional probability for the dialogue utterances, represents the sentiment probability for the context, represents the sentiment probability for the association, Represent sentiment probability for inference knowledge, is the softmax function, is the amount of emotion, and represents an aggregate network with the same structure but different parameters, Speaking for the dialogue, To express the situation, For association representation, for reasoning knowledge representation; S225, multiply the emotion probability as the final emotion probability, and use the log-likelihood loss function to calculate the emotion probability and the true label Perform parameter optimization; its expression is: ; ,in, The loss function for training and optimizing the model is calculated by calculating the logarithmic difference between the predicted emotion probability and the true emotion label.

[0010] Preferably, in S2, the specific steps of identifying the learner's emotional state by integrating brain-vision-language multimodal data are as follows: S231. Receive sentiment prediction results from the second-order attention mechanism , sentiment prediction tags based on visual images and emotion prediction labels based on EEG signals , and concatenate them into a vector ; S232, the spliced ​​input Through multiple fully connected layers MLP processing, each layer is transformed by weight matrix and activation function to generate the final sentiment prediction result, which is expressed as: ; The MLP is a multi-layer perceptron, comprising multiple fully connected layers and activation functions.

[0011] Preferably, in S3, the multimodal empathy response generation framework introduces three models, namely, a perception and retrieval model PRM, a generation model GM, and a retrieval enhancement model RAM, to process multimodal data; the multimodal empathy response generation framework uses ViT-G / 14, Qformer, and linear layers to encode images, adopts a DALL·E2 image decoder for image decoding, and uses GPT-4 for language modeling.

[0012] Preferably, in S3, the specific steps of constructing the empathy response model based on multimodal representation are as follows: S31. Perform multimodal input perception and feature fusion: The conversation content between the learner and the agent is used as multimodal input and converted into a feature vector that can be solved by a large language model. The image is encoded through a pre-trained visual encoder and converted into a feature vector through BLIP and MLP. Fusion with text features; each text tag is embedded as a semantic information vector of the text ; S32, perform multimodal output and generation: perform operations of expanding the vocabulary, text generation, and image generation to obtain a loss function optimization model, and perform retrieval enhanced image generation operations.

[0013] Preferably, in S32, the loss function expressions obtained are: ; ; ; In the formula, for language modeling loss, Generate loss for the image, is the image retrieval loss, are the features of the input text and image, Represented as generated textual and visual tokens, , is the embedding representation of the image and represents the visual features, is the contrast loss from text to image, is the image-to-text contrast loss, is the hidden state in the generation task, is the query feature, is a frozen text encoder, is the description of the image, It’s emotional information.

[0014] Preferably, in S3, the empathy response model joint learning training adopts an end-to-end model training method, uses an adapter fine-tuning method for joint fine-tuning, synchronously updates a limited number of parameters in the LLM, and simultaneously updates the input linear projection layer and feature mapping module to obtain a final overall loss function; the final loss function includes a language modeling loss , image generation loss and image retrieval loss , then the overall loss function is expressed as: ; In the formula, is a hyperparameter. For perception and retrieval models, ; For the generative model, .

[0015] Therefore, the present invention proposes an intelligent agent empathy response method based on brain-computer coupling under the guidance of learner emotions, and its beneficial effects are as follows: (1) This paper proposes an empathetic response method for an intelligent agent based on brain-computer coupling, which enhances the learner's empathy ability by accurately identifying the learner's emotional state and dynamically adjusting the conversation content of the intelligent agent based on the emotional state.

[0016] (2) The present invention combines electroencephalogram (EEG) signals and multimodal emotion recognition technology to obtain learners' emotional feedback in real time through brain-computer coupling, and generates more empathetic dialogue content on this basis, thereby improving learners' emotional cognitive ability in the collaborative learning process and promoting the development of their social and emotional intelligence.

[0017] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is the main flow chart of the empathic response method of an intelligent agent based on brain-computer coupling under the guidance of learner emotions of the present invention; Figure 2 It is a cognitive-visual learning framework diagram of the intelligent agent empathy response method based on brain-computer coupling under the guidance of learner emotions of the present invention; Figure 3 It is a learner dialogue text emotion recognition model diagram based on the second-order interactive attention mechanism of the intelligent agent empathy response method based on brain-computer coupling under the learner emotion guidance of the present invention; Figure 4 It is a multimodal empathy response generation framework diagram of the intelligent agent empathy response method based on brain-computer coupling under the learner's emotion guidance of the present invention. DETAILED DESCRIPTION

[0019] In order to make the technical solutions, advantages and purposes of the present invention clearer, the technical solutions of the embodiments of the present invention are clearly and completely described below. The described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of this application.

[0020] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0021] like Figure 1As shown, the present invention provides an intelligent agent empathy response method based on brain-computer coupling under the guidance of learner emotions, and the specific steps are as follows: S1. Collect multimodal emotional data of learners for agent-oriented empathic response, where the collected multimodal emotional data includes learners' brain data for emotional empathy and dialogue data for agent collaboration; The specific steps for collecting learner brain data for emotional empathy are as follows: S11. Collect learners’ brain data in a laboratory environment and let them watch different emotional images in the dataset to obtain EEG signals. S12. After EEG signal acquisition, invalid segments and artifacts were removed, and the frequency band of 1 Hz-75 Hz was filtered using a Butterworth filter.

[0022] Among them, EEG signals were collected through the NeuroScan64-lead EEG cap, which contains 62 scalp electrodes and two reference electrodes, placed according to the international 10-20 standard, with a sampling rate of 1kHz; the images are divided into seven emotional types: happiness, sadness, anger, surprise, fear, disgust, and neutral. Each facial emotion image is displayed for 0.5 seconds, and there is a 10-second buffer between facial images; to reduce interference, irrelevant equipment was turned off during the experiment to ensure a clean environment. The experiment lasted about 13 minutes.

[0023] The specific steps of conversation data collection for agent collaboration are as follows: after completing the staged classroom learning tasks, learners conduct self-perception of learning content and learning emotions, report them to the agent, have conversations and exchanges with the agent, and record the conversation context data set as , the dialogue situation is recorded as , the agent model understands the emotion A in the conversation; where, represents the i-th speech, including words, and the situation Q represents the situation description consisting of n words, where w q It provides contextual information to help understand the context of the conversation and generate appropriate responses accordingly.

[0024] S2. Use the emotion state discrimination method of multimodal data to perform emotion analysis on the multimodal data of learners collected under brain-computer coupling. The emotion analysis is performed from three parts: facial emotion analysis of learners based on brain-computer coupling learning method, emotion recognition of learners' dialogue text based on second-order interactive attention mechanism, and emotion state discrimination of learners integrating brain-vision-language multimodal data; The specific steps of learner facial emotion analysis based on brain-computer coupling learning method are as follows: S211, Extracting preliminary representations of facial expression images and EEG signals: Use EEGNet and CNNNet to perform preliminary feature extraction on facial expression images and EEG signals; Among them, for facial expression images in the visual field, an improved convolutional neural network CNNNet is used to extract preliminary features and preliminary representation of the visual field : ; In the formula, is the improved convolutional neural network CNNNet, V is a given visual image, For the initial representation of the visual field; For EEG signals, the feature extractor of EEGNet is used to perform preliminary feature extraction and preliminary representation of cognitive domains : ; In the formula, EEGNet is a compact convolutional neural network designed for EEG brain-computer interface. R is a given EEG signal. It is a preliminary representation of the cognitive domain; Among them, EEG signals and facial expression images are two different data modalities; CNNNet is composed of three convolutional modules, each of which includes a convolutional layer, a normalization layer, a nonlinear activation layer, and a maximum pooling layer. The output of the third convolutional module is used as a preliminary representation of the visual field to reflect the learner's facial emotional response; for EEG signals in the cognitive field, EEGNet is used as a feature extractor. EEGNet (GR) is a compact convolutional neural network designed for EEG brain-computer interface, which includes standard convolutional layers, deep convolutional layers, and separable convolutional layers. The output of the third convolutional module is used as a preliminary feature representation of the cognitive field to capture the changes in learners' EEG activity; S212, construct a cognitive-visual learning framework, project the obtained preliminary representation to the public channel and the private channel, and encode the public channel and the private channel representation; Among them, the public channel adopts the encoding function of shared parameters , learning to capture the common information between cognitive and visual fields. After training, the common channel extracts the shared features of cognitive and visual fields. Given and , then the common representation of cognitive and visual domain modalities is: ; ; In the formula, is an encoding function based on a simple fully connected neural layer, for shared parameters in cognitive and visual domains; The private representation is output by the private channels in the cognitive and visual domains, capturing the private information related to the cognitive and visual domains. and , then the private representation is: ; ; In the formula, is the private encoding function of the private channel in the visual field, is the private encoding function of the private channel in the cognitive domain, This is achieved through fully connected neural layers. is the parameter of the private encoding function of the private channel in the visual field, are the parameters of the private encoding function of the private channel of the cognitive domain; like Figure 2 As shown in the figure, the cognitive domain and the visual domain have independent private channels respectively, and the commonalities between the cognitive domain and the visual domain are captured through the public channel. The public features of the cognitive-visual domain refer to the emotional response patterns (emotion activation intensity, emotion category consistency, etc.) that can be shared with EEG signals by observing facial expression images. The private features of the cognitive domain refer to the unique features of EEG, which reflect the way the brain processes emotions and individual differences (brain wave frequency differences, brain wave response intensity differences, etc.). The private features of the visual domain refer to the individual differences and details unique to facial expressions (facial morphological features, micro-expression features).

[0025] S213, after obtaining the public representation and the private representation, performing a splicing operation on the obtained public representation and the private representation, and performing an emotion recognition task; Among them, the concatenated representations of the public and private representations in the visual and cognitive domains are defined as: ; ; In the formula, It is the spliced ​​representation of the visual field. It is a representation after splicing of cognitive domains; The expressions of emotion recognition tasks in the visual domain and cognitive domain are: ; ; In the formula, and is the predicted label of the image and EEG signal, using KNN as the function Model.

[0026] like Figure 3As shown in the figure, a learner dialogue text emotion recognition model based on the second-order interactive attention mechanism is proposed. In a real classroom environment, the second-order interactive attention mechanism is used to analyze the explicit and implicit connotations of the dialogue text to improve the accuracy of learners' emotions. After completing the staged learning tasks, learners self-perceive the learning content and learning emotions, and report them to the intelligent agent through text or emoticon pictures for emotional communication and feedback. This process regards the dialogue content as explicit information, while the implicit information includes the learner's emotional state and situational reasoning. On this basis, the dialogue content is encoded from the perspectives of explicit information and implicit information.

[0027] The specific steps of emotion recognition of learner dialogue text based on the second-order interactive attention mechanism are as follows: S221: Construct explicit information pairs and implicit information pairs based on the learner's learning emotions and learning experience reports as input, and send them to the information association module to understand the learner's current speech on learning emotions and learning experience reports. ; Constructed explicit information pairs: ; Implicit information about: ; In the formula, , Represents memory, used to store associated words, and is initialized to empty; when When , the dialogue and memory are empty; Among them, explicit information pairs are used to capture the important connections between the current speech and the context, conversation history, and memory, while implicit information pairs are used to capture the implicit important connections between the current speech and the context, conversation history, and memory.

[0028] S222, encode the dialogue information, regard the dialogue speech and situation between the learner and the agent as explicit information, regard the reasoning knowledge as implicit information, and process the explicit information and implicit information to obtain the semantic representation of the dialogue speech representation, the situation representation and the reasoning knowledge; The processing of explicit information is as follows: special start markers are added before the dialogue speech and the context. , get the word sequence and situation description ; For each round of speeches and situation description Encode; Use a speech-grade encoder Generate sentence representation: ; In the formula, is the word embedding, is the position embedding, is the state embedding, which is used to distinguish the speaker (learner or agent) from the listener (agent or learner); For the entire conversation Using a Conversation Level Encoder To encode: ; In the formula, , ,in and Respectively represent the length of the i-th speech and the total number of dialogue speeches, represents the hidden layer size, Represents a splicing operation; For situation word sequences , using the encoder Let’s learn about context representation: ; In the formula, and Represent the word embedding and position embedding of the context respectively, , is the number of words the situation contains; The process of processing implicit information is: generating reasoning knowledge through the COMET model and , add before the inference knowledge Marking and encoding the inference knowledge; the inference knowledge It contains five types of texts: event impact uEffect, personal emotional reaction uReact, personal intention uIntent, personal need uNeed and personal wish uWant. The expression obtained after processing the implicit information is: ; In the formula, A semantic representation that represents reasoning knowledge, That is, for the i-th round of speeches and Situation The inference knowledge of type ∈ uEffect, uReact, uIntent, uNeed, uWant, ⊕ represents the splicing operation; S223, using the information association module to capture important associated words in explicit and implicit information, and performing the steps of constructing an associated matrix, first-order interactive attention, second-order interactive attention, and storing associated words in memory to obtain an associated representation, so as to more coherently and comprehensively understand the dialogue between the learner and the intelligent agent; Among them, the conjunctions exist in both sentences (speech or situation) at the same time, so the conjunctions are selected bidirectionally in the two sentences. The specific steps of constructing the conjunction matrix are as follows: Based on the context of speech selection (i.e. ), choose the conjunctions in the speech based on the context (i.e. ), in order to capture rich features, a multi-head correlation matrix is ​​constructed: , represents the association score from situation words to utterance words; Represents the correlation score between speech words and situation words. The expression of the multi-head correlation matrix is: ; ; In the formula, , is the Sigmoid function, is the number of longs, Represents the first Header index. , Represents the weight matrix and word vector used in the multi-head attention mechanism; The steps for performing first-order interactive attention are as follows: Identify the key words in the context and select the context words that are highly related to the spoken words as the key words: ; ; in, is the average function, is the filter function, Indicates the number of selected keywords. ; The steps for performing second-order interactive attention are as follows: Select the speech words with the highest correlation with the key context words as important correlation words: ; in, and They are the representation and score of spoken words related to situation words, respectively. is the number of associated words; The steps to store associative memory are as follows: For explicit information, choose keywords between the speech and the situation , the link between speech and dialogue history and speech and memory Related words between: ; Implicit memory: ; When the iteration reaches the last sentence, the explicit information is memorized and implicit information memory Combined as the final associated information : ; in, , ; Based on the memory of associated words, the associated information is learned through the encoder, and the expression of the associated information is:

[0029] In the formula, For association representation, For related information, , is the number of associated words in the memory, For encoder; S224, input the dialogue speech representation, situation representation, association representation and reasoning knowledge representation into the aggregation network to obtain the emotion representation, and use the emotion representation to predict the emotion probability respectively; wherein the expression of the emotion probability is: ; ; ; ; In the formula, , represents the emotional probability for the dialogue utterances, represents the sentiment probability for the context, represents the sentiment probability for the association, Represent sentiment probability for inference knowledge, is the softmax function, is the amount of emotion, and represents an aggregate network with the same structure but different parameters, Speaking for the dialogue, To express the situation, For association representation, for reasoning knowledge representation; S225, multiply the emotion probability as the final emotion probability, and use the log-likelihood loss function to calculate the emotion probability and the true label Perform parameter optimization; its expression is: ; ,in, The loss function for training and optimizing the model is calculated by calculating the logarithmic difference between the predicted emotion probability and the true emotion label.

[0030] The specific steps of identifying learners’ emotional states by integrating brain-vision-language multimodal data are as follows: S231. Receive sentiment prediction results from the second-order attention mechanism , sentiment prediction tags based on visual images and emotion prediction labels based on EEG signals , and concatenate them into a vector ; S232, the spliced ​​input Through multiple fully connected layers MLP processing, each layer is transformed by weight matrix and activation function to generate the final sentiment prediction result, which is expressed as: ; Among them, MLP is a multi-layer perceptron, which contains multiple fully connected layers and activation functions S3. Design a multimodal empathy response generation framework based on emotion recognition results and conversation content, construct an empathy response model based on multimodal representation, and conduct joint learning and training of the empathy response model to form an intelligent agent empathy response model based on a large language model multi-agent system under the guidance of learners' emotions.

[0031] like Figure 4 As shown in the figure, the multimodal empathy response generation framework has the ability to perceive learners' emotions and generate text replies and emoticons based on the learners' emotion recognition results based on EEG and text data and the content of the conversation (including text input and emoticon input). Through different image generation strategies, the multimodal empathy response generation framework is derived based on this framework, including the perception and retrieval model (PRM), the generation model (GM), and the retrieval enhancement model (RAM); the multimodal empathy response generation framework uses ViT-G / 14, Qformer, and linear layers to encode images, uses GPT-4 for language modeling, and uses DALL·E 2 for image decoder.

[0032] The specific steps of constructing an empathy response model based on multimodal representation are as follows: S31. Perform multimodal input perception and feature fusion: The conversation content between the learner and the agent is used as multimodal input and converted into a feature vector that can be solved by a large language model. The image is encoded through a pre-trained visual encoder and converted into a feature vector through BLIP and MLP. Fusion with text features; each text tag is embedded as a semantic information vector of the text ; S32, perform multimodal output and generation: perform operations of expanding the vocabulary, text generation, and image generation to obtain a loss function optimization model, and perform retrieval enhanced image generation operations.

[0033] The specific steps for expanding the vocabulary are: Add visual markers set , the visual markers are divided into two groups, tags are used for image retrieval, and then The tag is used for image generation, and the image information is used as part of the generated content. Its expression is: ; ; ; In the formula, For perception and retrieval models and retrieval enhancement models, Used to generate models and retrieve enhanced models; The specific steps for text generation are: After receiving multimodal input, generate a joint sequence of textual and visual tags , and the generated markup is represented as ,in, , then the loss function Defined as: ; in, and are the features of the input text and image, The image embedding represents the visual features for alignment with the text embedding, and the loss function optimizes the model by maximizing the generated probability.

[0034] In image retrieval, the agent empathy response model will correspond to The hidden state Align to retrieval space via contrastive learning and use cosine similarity to measure the similarity of projected vectors Image retrieval loss , combining text to image ( ) and Image to Text ( ), optimizing the mapping between images and text, image retrieval loss for: ; in, Used to optimize the loss of the retrieval projection layer; The specific steps for image generation are: In the image generation stage, the generated image is generated through the DALL·E 2 decoder and the loss between the generated image and the actual image is minimized.

[0035] Loss Function It is expressed as: ; in, is the hidden state in the generation task, is the query feature, is a frozen text encoder, is the description of the image, It is emotional information; The steps for performing the retrieval enhanced image generation operation are: Retrieve an image as a latent representation To enhance the generation process. In the image generation process, we still use As a condition, this approach continues to generate on the retrieved images, which can expand the diversity of images while maintaining image quality.

[0036] The specific steps of joint learning and training of the empathy response model are as follows: The model is trained in an end-to-end manner, and the adapter fine-tuning method is used for joint fine-tuning. A limited number of parameters in the LLM are updated simultaneously, and the input linear projection layer and feature mapping module are updated at the same time to obtain the final overall loss function; the final loss function includes the language modeling loss , image generation loss and image retrieval loss , then the overall loss function is expressed as: ; In the formula, is a hyperparameter. For perception and retrieval models, ; For the generative model, .

[0037] Through this multi-task joint training, the agent empathy response model can generate reasonable responses with visual elements in empathy-rich conversations, improving the diversity and emotional understanding capabilities of the agent's empathy responses and conversations.

[0038] Therefore, the empathic response method of an intelligent agent based on brain-computer coupling under the guidance of learner emotions provided by the present invention can enhance the learner's empathy ability by accurately identifying the learner's emotional state and dynamically adjusting the dialogue content of the intelligent agent based on the emotional state. It combines electroencephalogram (EEG) signals and multimodal emotion recognition technology to obtain learner's emotional feedback in real time through brain-computer coupling, and generates more empathetic dialogue content on this basis, thereby improving learners' emotional cognitive ability in the collaborative learning process and promoting the development of their social and emotional intelligence.

[0039] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. An intelligent agent empathy response method based on brain-computer coupling under the guidance of learner emotions, characterized in that: Here are the steps: S1. Collect multimodal emotional data of learners for agent-oriented empathic response, where the collected multimodal emotional data includes learners' brain data for emotional empathy and dialogue data for agent collaboration; S2. Use the emotion state discrimination method of multimodal data to perform emotion analysis on the multimodal data of learners collected under brain-computer coupling. The emotion analysis is performed from three parts: facial emotion analysis of learners based on brain-computer coupling learning method, emotion recognition of learners' dialogue text based on second-order interactive attention mechanism, and emotion state discrimination of learners integrating brain-vision-language multimodal data; S3. Design a multimodal empathy response generation framework based on emotion recognition results and conversation content, construct an empathy response model based on multimodal representation, and conduct joint learning and training of the empathy response model to form an intelligent agent empathy response model based on a large language model multi-agent system under the guidance of learners' emotions.

2. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 1, characterized in that: In S1, the specific steps for collecting learner brain data for emotional empathy are: S11. Collect learners’ brain data in a laboratory environment and let them watch different emotional images in the dataset to obtain EEG signals. S12. After EEG signal acquisition, invalid segments and artifacts were removed, and the frequency band of 1 Hz-75 Hz was filtered using a Butterworth filter.

3. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 1, characterized in that: In S1, the specific steps of conversation data collection for agent collaboration are as follows: after completing the staged classroom learning tasks, learners conduct self-perception of learning content and learning emotions, report them to the agent, have conversations and exchanges with the agent, and record the conversation context data set as , the dialogue situation is recorded as , the intelligent agent model understands the emotions in the conversation ;in, Indicates Speeches, including words, situations Indicated by The situation description consists of words, where w q Provides contextual background information.

4. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 1, characterized in that: In S2, the specific steps of learner facial emotion analysis based on the brain-computer coupling learning method are as follows: S211, use EEGNet and CNNNet to perform preliminary feature extraction on facial expression images and EEG signals; Among them, for facial expression images in the visual field, an improved convolutional neural network CNNNet is used to extract preliminary features and preliminary representation of the visual field : ； In the formula, For the improved convolutional neural network CNNNet, For a given visual image, For the initial representation of the visual field; For EEG signals, the feature extractor of EEGNet is used to perform preliminary feature extraction and preliminary representation of cognitive domains : ； In the formula, EEGNet is a compact convolutional neural network designed for EEG brain-computer interface. R is a given EEG signal. It is a preliminary representation of the cognitive domain; S212, construct a cognitive-visual learning framework, project the obtained preliminary representation to the public channel and the private channel, and encode the public channel and the private channel representation; Among them, the public channel adopts the encoding function of shared parameters , learning to capture the common information between cognitive and visual fields. After training, the common channel extracts the shared features of cognitive and visual fields. Given and , then the common representation of cognitive and visual domain modalities is: ； ； In the formula, is an encoding function based on a simple fully connected neural layer, for shared parameters in cognitive and visual domains; The private representation is output by the private channels in the cognitive and visual domains, capturing the private information related to the cognitive and visual domains. and , then the private representation is: ； ； In the formula, is the private encoding function of the private channel in the visual field, is the private encoding function of the private channel in the cognitive domain, This is achieved through fully connected neural layers. is the parameter of the private encoding function of the private channel in the visual field, are the parameters of the private encoding function of the private channel of the cognitive domain; S213, after obtaining the public representation and the private representation, performing a splicing operation on the obtained public representation and the private representation, and performing an emotion recognition task; Among them, the concatenated representations of the public and private representations in the visual and cognitive domains are defined as: ； ； In the formula, It is the spliced ​​representation of the visual field. It is a representation after splicing of cognitive domains; The expressions of emotion recognition tasks in the visual domain and cognitive domain are: ； ； In the formula, and is the predicted label of the image and EEG signal, using KNN as the function Model.

5. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 1, characterized in that: In S2, the specific steps of emotion recognition of learner dialogue text based on the second-order interactive attention mechanism are as follows: S221, constructing explicit information pairs and implicit information pairs based on the learners' learning emotions and learning experience reports as inputs, and sending them to the information association module; S222, encode the dialogue information, regard the dialogue speech and situation between the learner and the agent as explicit information, regard the reasoning knowledge as implicit information, and process the explicit information and implicit information to obtain the semantic representation of the dialogue speech representation, the situation representation and the reasoning knowledge; S223, using the information association module to capture important associated words in explicit and implicit information, constructing an associated matrix, first-order interactive attention, second-order interactive attention, and storing associated words in memory, to obtain an associated representation, the expression of which is: ; In the formula, For association representation, For related information, , is the number of associated words in the memory, For encoder; S224, input the dialogue speech representation, situation representation, association representation and reasoning knowledge representation into the aggregation network to obtain the emotion representation, and use the emotion representation to predict the emotion probability respectively; wherein the expression of the emotion probability is: ； ; ； ； In the formula, , represents the emotional probability for the dialogue utterances, represents the sentiment probability for the context, represents the sentiment probability for the association, Represent sentiment probability for inference knowledge, is the softmax function, is the amount of emotion, and represents an aggregate network with the same structure but different parameters, Speaking for the dialogue, To express the situation, For association representation, for reasoning knowledge representation; S225, multiply the emotion probability as the final emotion probability, and use the log-likelihood loss function to calculate the emotion probability and the true label Perform parameter optimization; its expression is: ; ,in, The loss function for training and optimizing the model is calculated by calculating the logarithmic difference between the predicted emotion probability and the true emotion label.

6. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 1, characterized in that: In S2, the specific steps of identifying the learner's emotional state by integrating brain-vision-language multimodal data are as follows: S231. Receive sentiment prediction results from the second-order attention mechanism , sentiment prediction tags based on visual images and emotion prediction labels based on EEG signals , and concatenate them into a vector ; S232, the spliced ​​input Through multiple fully connected layers MLP processing, each layer is transformed by weight matrix and activation function to generate the final sentiment prediction result, which is expressed as: ; The MLP is a multi-layer perceptron, comprising multiple fully connected layers and activation functions.

7. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 1, characterized in that: In S3, the multimodal empathy response generation framework introduces three models: perception and retrieval model PRM, generation model GM, and retrieval enhancement model RAM to process multimodal data; the multimodal empathy response generation framework uses ViT-G / 14, Qformer, and linear layers to encode images, uses DALL·E2 image decoder for image decoding, and uses GPT-4 for language modeling.

8. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 1, characterized in that: In S3, the specific steps for constructing an empathy response model based on multimodal representation are as follows: S31. Perform multimodal input perception and feature fusion: The conversation content between the learner and the agent is used as multimodal input and converted into a feature vector that can be solved by a large language model. The image is encoded through a pre-trained visual encoder and converted into a feature vector through BLIP and MLP. Fusion with text features; each text tag is embedded as a semantic information vector of the text ; S32, perform multimodal output and generation: perform operations of expanding the vocabulary, text generation, and image generation to obtain a loss function optimization model, and perform retrieval enhanced image generation operations.

9. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 8, characterized in that: In S32, the expressions of the loss functions obtained are: ； ； ； In the formula, for language modeling loss, Generate loss for the image, is the image retrieval loss, are the features of the input text and image, Represented as generated textual and visual tokens, , is the embedding representation of the image and represents the visual features, is the contrast loss from text to image, is the image-to-text contrast loss, is the hidden state in the generation task, is the query feature, is a frozen text encoder, is the description of the image, It’s emotional information.

10. The method for empathic response of an intelligent agent based on brain-computer coupling under the guidance of learner emotions according to claim 1, characterized in that: In S3, the empathy response model joint learning training adopts an end-to-end model training method, uses an adapter fine-tuning method for joint fine-tuning, synchronously updates a limited number of parameters in the LLM, and simultaneously updates the input linear projection layer and feature mapping module to obtain the final overall loss function; the final loss function includes the language modeling loss , image generation loss and image retrieval loss , then the overall loss function is expressed as: ； In the formula, is a hyperparameter. For perception and retrieval models, ; For the generative model, .

Images