Dynamic emotion interaction system and method based on multi-modal physiological signals and adaptive feedback

Abstract

The invention discloses a dynamic emotion interaction system based on multi-modal physiological signals and self-adaptive feedback, which comprises a multi-modal physiological signal recognition and alignment module, a cross-modal fusion module, a self-adaptive feedback mechanism module, an emotion classification module and a dynamic feedback mechanism module, a pre-trained large-scale multi-modal language model is used for emotion recognition, a self-adaptive gating mechanism and a graph attention network are used for dynamic fusion of features and adjustment of modal weights, and emotion reasoning and interaction are enhanced. Besides, the adaptive feedback mechanism utilizes an AGDF model to introduce feature decoupling and dynamic fusion into an Emotion-LLaMA model, so that real-time optimization of emotion recognition and interaction is realized, unique information and common information of different modes can be effectively balanced, refined feature interaction is realized, the emotion recognition accuracy in a complex scene is remarkably improved, and the emotion recognition efficiency is improved. And naturalness and individuation of emotional interaction are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and human-computer interaction, and particularly relates to a dynamic emotional interaction system and method based on multi-modal physiological signals and adaptive feedback. BACKGROUND

[0002] Artificial intelligence (AI) and human-computer interaction (HCI) include machine learning (ML), natural language processing (NLP), knowledge representation and reasoning, etc., and are composed of perception layer, decision layer, feedback layer and other layers.

[0003] Generally speaking, multi-modal physiological signal recognition and adaptive feedback cannot be parallelized, and existing algorithms (such as CNN+LSTM) use group data when training, but ignore individual physiological differences (such as some people are naturally fast in speech or expressionless), resulting in a sharp drop in recognition accuracy for specific users. At present, most researches tend to use attention mechanism-based fusion models to improve the accuracy of emotion analysis in specific scenarios through complex multi-modal feature fusion. For example, the Emotion-LLaMA model (arXiv:2406.11161v2) proposed by Cheng et al. uses HuBERT and VideoMAE encoders, but does not solve the problem of real-time processing on mobile terminals.

[0004] Generally speaking, existing emotion recognition has a fixed strategy library, the current system feedback mode is single, the real-time physiological response of the user is not considered, the feedback mechanism is rigid, there is a lack of quantitative evaluation of the feedback effect, and the strategy cannot be dynamically adjusted, resulting in low emotion recognition accuracy and poor user experience. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application provides a dynamic emotional interaction system and method based on multi-modal physiological signals and adaptive feedback, which aims to accurately identify emotions based on multi-modal audio and micro-expression signals while performing adaptive feedback of different modalities.

[0006] According to an aspect of the present application, a dynamic emotional interaction system based on multi-modal physiological signals and adaptive feedback is provided, comprising:

[0007] A multi-modal physiological signal recognition and alignment module is used to extract low-level features of visual, speech and text modalities from a multi-modal emotion data set MERR and perform preprocessing;

[0008] A cross-modal fusion module is used to realize interaction between modalities based on a cross-modal attention module, and a multi-head self-attention mechanism is used to calculate the correlation between modalities to generate a fused cross-modal feature sequence;

[0009] An adaptive feedback mechanism module is configured to decouple the fused cross-modal features to obtain common features shared by cross-modal and unique features exclusive to each modality, and to dynamically fuse the decoupled common features and unique features again;

[0010] An emotion classification module is configured to output discrete emotion category labels with intensity based on the dynamically fused multi-modal features through a classification model;

[0011] A dynamic feedback mechanism module is configured to adaptively adjust the emotion category labels according to the emotion classification results in combination with real-time environmental or user feedback data.

[0012] As a further technical solution, the multi-modal physiological signal recognition and alignment module is further configured to execute the following instructions:

[0013] The extracted low-level features are added with a position encoding sequence after a one-dimensional spatio-temporal convolution layer, and the position encoding is consistent with the position encoding in the Transformer model in terms of construction method;

[0014] The visual, speech and text information are respectively preprocessed to obtain multi-modal features of the visual, speech and text.

[0015] As a further technical solution, the adaptive feedback mechanism module is further configured to execute the following instructions:

[0016] An adaptive gating mechanism is called to calculate the offset of the speech and visual features in the text semantic space, and dynamically adjust the weight of the modality;

[0017] An inter-modal relationship graph is constructed through a graph attention network GAT, wherein the nodes represent feature vectors of each modality, and the edge weights are calculated through an attention mechanism.

[0018] As a further technical solution, the adaptive feedback mechanism module is further configured to execute the following instructions:

[0019] Before feature decoupling, the fused cross-modal features are encoded and aligned.

[0020] As a further technical solution, the dynamic feedback mechanism module is further configured to execute the following instructions:

[0021] Real-time user physiological response data is collected to generate feedback evaluation indicators;

[0022] The emotion classification model parameters are optimized through reinforcement learning, and the loss function is:

[0023]

[0024] wherein is the label adjusted according to the feedback, For the model prediction emotion label, β is the weight coefficient, and the KL divergence is used to constrain the model updating amplitude.

[0025] According to an aspect of the present application, a dynamic emotion interaction method based on multi-modal physiological signals and adaptive feedback is provided, comprising:

[0026] Low-level features of visual, speech, and text modalities are extracted from a multi-modal emotion dataset MERR and preprocessed;

[0027] Interactions between modalities are realized based on a cross-modal attention module, and a multi-head self-attention mechanism is used to calculate the correlation between modalities to generate a fused cross-modal feature sequence;

[0028] The fused cross-modal features are decoupled to obtain common features shared by cross-modalities and unique features owned by each modality, and the decoupled common features and unique features are further dynamically fused;

[0029] Based on the multi-modal features after dynamic fusion, a classification model is used to output discrete emotion category labels with intensity;

[0030] According to the emotion classification result, the emotion category labels are adaptively adjusted in combination with real-time environmental or user feedback data.

[0031] As a further technical solution, the decoupled common features and unique features are further dynamically fused, comprising:

[0032] An adaptive gating mechanism is called to calculate the offset of speech and visual features in the text semantic space, and the weights of the modalities are dynamically adjusted;

[0033] A graph attention network GAT is used to construct an inter-modal relationship graph, wherein the nodes represent the feature vectors of each modality, and the edge weights are calculated through an attention mechanism.

[0034] According to an aspect of the present application, a dynamic emotion interaction device based on multi-modal physiological signals and adaptive feedback is provided, comprising a memory and a processor, the memory storing program instructions executed by the processor, and the processor calling the program instructions to execute the dynamic emotion interaction method based on multi-modal physiological signals and adaptive feedback.

[0035] According to an aspect of the present application, a non-transitory computer readable storage medium is provided, which stores computer instructions for executing the dynamic emotion interaction method based on multi-modal physiological signals and adaptive feedback.

[0036] According to an aspect of the specification of the present application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the dynamic emotional interaction method based on multi-modal physiological signals and adaptive feedback.

[0037] Compared with the prior art, the present application has the following beneficial effects:

[0038] The present application comprises a multi-modal physiological signal recognition and alignment module, a cross-modal fusion module, an adaptive feedback mechanism module, an emotion classification module, and a dynamic feedback mechanism module. The pre-trained large multi-modal language model (Emotion-LLaMA) is used for emotion recognition. The adaptive gating mechanism and the graph attention network (GAT) are used for dynamic fusion of features and adjustment of modal weights, enhancing emotion reasoning and interaction. In addition, the adaptive feedback mechanism uses the AGDF (adaptive gating decoupling feature fusion) model to introduce feature decoupling and dynamic fusion into the Emotion-LLaMA model, realizing real-time optimization of emotion recognition and interaction, so that it can effectively balance the unique information and common information of different modalities, while realizing fine feature interaction, significantly improving the emotion recognition accuracy in complex scenarios, and effectively improving the naturalness and personalization of emotional interaction. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 A schematic diagram of the dynamic emotional interaction system based on multi-modal physiological signals and adaptive feedback provided by the embodiments of the present application.

[0041] Figure 2 A multi-modal physiological signal processing flowchart provided by the embodiments of the present application.

[0042] Figure 3 A cross-modal feature fusion flowchart provided by the embodiments of the present application.

[0043] Figure 4 An adaptive feedback mechanism flowchart provided by the embodiments of the present application. DETAILED DESCRIPTION

[0044] The terms "including", "containing", "having" and their conjugates, as used throughout the present specification and in the following claims, are meant to encompass the presence of stated features, steps or components, but do not preclude the presence or addition of one or more other features, steps, components or groups thereof.

[0045] The term "coupled" (or connected) as used throughout this specification and in the following claims, can refer to either a direct or indirect connection. For example, if a first device is coupled (or connected) to a second device, it can be directly connected to the second device or it can be indirectly connected to the second device through one or more other devices or a connection means. The terms "first", "second", and the like, as used throughout this specification, are used for naming only and do not necessarily have an implicitly limited upper or lower quantity limit, unless specifically so limited. Also, the use of the same reference numerals in different drawings illustrates the same or similar components.

[0046] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of the present application. In addition, the technical features in each of the embodiments provided by the present application or in a single embodiment can be combined with each other to form new technical solutions, and such combination is not restricted by the order of steps and / or the mode of structural components. However, the combination of technical solutions should be considered to exist and be within the scope of the present application when the combination of technical solutions is not contradictory or cannot be realized by those of ordinary skill in the art.

[0047] The embodiment of the application provides a dynamic emotion interaction system based on multi-modal physiological signals and adaptive feedback, comprising: a multi-modal physiological signal recognition and alignment module, which is used for extracting low-level features of visual, speech and text modalities from a multi-modal emotion data set MERR and performing preprocessing; a cross-modal fusion module, which is used for realizing inter-modal interaction based on a cross-modal attention module Crossmodal Transformers, and calculating inter-modal correlation by using a multi-head self-attention mechanism to generate a fused cross-modal feature sequence; an adaptive feedback mechanism module, which is used for decoupling the fused cross-modal features to obtain common features shared by cross-modal and unique features exclusively owned by each modality, and dynamically fusing the decoupled common features and unique features; an emotion classification module, which is used for outputting discrete emotion category labels with intensity based on the dynamically fused multi-modal features through a classification model; and a dynamic feedback mechanism module, which is used for adaptively adjusting the emotion category labels according to the emotion classification results, combining real-time environment or user feedback data, and finally outputting more accurate or personalized labels.

[0048] Please refer to Figure 1 and Figure 2 , the low-level features of visual, speech and text modalities extracted from the multi-modal emotion data set MERR are first subjected to a one-dimensional spatio-temporal convolution layer, that is, the sequence has "potential context awareness". Then, a position encoding sequence is added, at this time the position encoding is consistent with the position encoding construction method in the Transformer model. Then, the multi-modal physiological signals are preprocessed. For visual signals, facial action units (AUs) and key points are extracted through the OpenFace tool, and a spatio-temporal convolution network (such as a 3D CNN) is used to extract time sequence features; for speech signals, Librosa is used to extract acoustic features (such as MFCC, pitch, energy), and a Transformer encoder is used to generate time sequence features; for text information, a pre-trained BERT model is used to extract word vectors, and position encoding is added to retain sequence information, to obtain the preprocessed multi-modal physiological data.

[0049] Then enter the cross-modal fusion module, please refer to Figure 3 , the inter-modal interaction is realized through Crossmodal Transformers, each modality feature is mapped to a unified semantic space through a linear projection layer, the inter-modal correlation is calculated by using a multi-head self-attention mechanism, so as to generate a fused cross-modal feature sequence. At the same time, instruction fine-tuning is performed, pre-training is performed based on the MERR data set, and the model parameters are optimized through the emotion classification task to enhance the inference ability of the multi-modal physiological signal and adaptive feedback dynamic emotion interaction model.

[0050] After the fused cross-modal features are encoded and aligned, the adaptive feature decoupling step first performs contrastive learning to construct positive and negative sample pairs, decomposes each modal feature into shared features that share emotional semantics and unique features that are specific to modal-specific information, and maximizes the similarity of shared features and minimizes the similarity of unique features using a contrastive loss.

[0051] In the adaptive dynamic fusion step, an adaptive gating mechanism is first invoked to calculate the offset of speech and visual features in the text semantic space, and the modal weight is dynamically adjusted. The formula is:

[0052] (1)

[0053] where ga and gb are gating vectors, , are learnable parameters, is the speech modal feature, is the visual modal feature, is the reference vector in the text semantic space, is the dynamically adjusted modal weight coefficient.

[0054] Semantic offset calculation:

[0055] ,

[0056] ;

[0057] where , is the modal feature encoder, Tproj is the real-time projection feature vector of the current input text, is the offset of the speech modal and the text semantic space, is the offset of the visual modal and the text semantic space, is the original speech feature vector at the ith moment, is the original visual feature vector at the jth moment, and Tcenter is the pre-trained universal emotional reference vector.

[0058] where the text semantic center vector and the text real-time projection vector are:

[0059]

[0060]

[0061] N is the number of text samples in the training set, is the kth training text. Wp is the projection weight for the user's real-time input sentence, p is the projection bias.

[0062] Dynamic gating weight generation adopts EMA filtering for feature smoothing:

[0063]

[0064]

[0065] Wp is the original weight of the speech modality, Wv is the original weight of the visual modality, and σ is the Sigmoid activation function, Wp is the speech weight learning parameter, Wv is the visual weight learning parameter, and EMA is the exponential moving average.

[0066] Normalized fusion output:

[0067]

[0068] wherein is the fused multi-modal feature, is the learnable fusion weight of the speech modality, is the learnable fusion weight of the visual modality, is the bias term, is a scaling factor to prevent Softmax gradient vanishing, and the Softmax function ensures weight normalization.

[0069] At the same time, the graph attention network (GAT) is used to aggregate neighborhood information. The graph attention network is used to model the nonlinear dependence between modalities, and the contribution of each modality feature is dynamically allocated through attention weights. Specifically, the visual, speech, and text features are taken as graph nodes, and the multi-head attention mechanism is used to calculate the cross-modal correlation. The finally fused feature not only retains modality specificity, but also enhances the representation of shared emotional semantics, and its formula is:

[0070] (2)

[0071] wherein α uv is the attention weight, f is the LeakyReLU activation function, Wv is the node feature vector, is the parameter matrix of the attention mechanism, is the edge weight of the adjacency matrix, and v is the normalized attention score.

[0072] Attention weight formula in graph attention network (GAT):

[0073] ​

[0074] a is a learnable attention parameter vector, Wu, Wv are the modal-specific transformation matrices of nodes u and v, 、 is the input feature vector of node u and v, is the set of neighbor nodes of node u.

[0075] Please refer to Figure 4 In the emotion classification step, the dynamically fused features are input into the fully connected layer, and the cross-entropy loss is used for emotion classification based on Ekman six categories or VAD dimensions, and the mean absolute error (MAE) is used to predict the emotion intensity.

[0076] The emotion classification module outputs discrete emotion categories (such as "happy", "angry", "sad") based on dynamically fused multi-modal features (visual, speech, text) through a classification model (such as a fully connected layer + Softmax). In emotion classification, the label with intensity is directly output, for example: Ekman six categories expansion: happy (strong), angry (medium), sad (weak); or VAD dimensions: valence = 0.8, arousal = 0.6, dominance = 0.4.

[0077] The output emotion label is then adjusted in real time through a dynamic feedback mechanism, including adjusting the output emotion label in real time through environmental adjustment such as automatically reducing the brightness of the light or playing soothing music, through interactive response such as switching the robot to a soothing dialogue mode, and collecting multi-modal physiological feedback data of the user to iteratively optimize the model. This implementation significantly improves the accuracy of emotion recognition and system response capability in complex scenarios by combining cutting-edge multi-modal alignment and dynamic fusion technology, and combines a feedback mechanism for real-time physiological response of the user, adds a dynamic adjustment strategy, and improves the user experience.

[0078] Specifically, the model optimization of the dynamic feedback mechanism module includes:

[0079] Real-time acquisition of user physiological response data (such as heart rate, facial action units), generation of feedback evaluation indicators;

[0080] Optimization of emotion classification model parameters through reinforcement learning, with the loss function being:

[0081]

[0082] wherein is the label adjusted according to the feedback, is the emotion label predicted by the model, β is the weight coefficient, and KL divergence is used to constrain the model update amplitude.

[0083] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a dynamic emotion interaction method based on multi-modal physiological signals and adaptive feedback. First, low-level features of visual, speech, and text modalities are extracted from a multi-modal emotion data set MERR and preprocessed. Then, inter-modal interaction is realized based on a cross-modal attention module, and a multi-head self-attention mechanism is used to calculate inter-modal correlation to generate a fused cross-modal feature sequence. Subsequently, the fused cross-modal features are decoupled to obtain common features shared by cross-modal and unique features of each modality, and the decoupled common features and unique features are dynamically fused again. Then, based on the multi-modal features after dynamic fusion, a classification model is used to output discrete emotion category labels with intensity. Finally, according to the emotion classification result, real-time environmental or user feedback data are combined to adaptively adjust the emotion category labels.

[0084] As a preferred, the decoupled common features and unique features are dynamically fused again, including: calling an adaptive gating mechanism to calculate the offset of speech and visual features in the text semantic space, and dynamically adjusting the weight of the modalities.

[0085] An inter-modal relationship graph is constructed by a graph attention network GAT, where nodes represent feature vectors of each modality, and edge weights are calculated by an attention mechanism.

[0086] As a preferred, the method further includes: after the extracted low-level features pass through a one-dimensional spatiotemporal convolution layer, a position encoding sequence is added, and the position encoding is consistent with the position encoding in the Transformer model in terms of construction method; visual, speech, and text information are preprocessed respectively to obtain multi-modal features of the visual, speech, and text.

[0087] As a preferred, the method further includes: before feature decoupling, the fused cross-modal features are encoded and aligned.

[0088] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a dynamic emotion interaction device based on multi-modal physiological signals and adaptive feedback. The device includes a memory and a processor. The memory stores program instructions executed by the processor. The processor calls the program instructions to execute the dynamic emotion interaction method based on multi-modal physiological signals and adaptive feedback.

[0089] In the embodiments of the present application, the memory can be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), and can also be a volatile memory, such as a random-access memory (RAM). The memory can be any other medium capable of carrying or storing desired program codes in the form of instructions or data structures and capable of being accessed by a computer, but is not limited to this. The memory in the embodiments of the present application can also be a circuit or any other device capable of realizing a storage function, for storing program instructions and / or data.

[0090] In the embodiments of the present application, the processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware processor execution or executed by a combination of hardware and software modules in the processor.

[0091] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a non-transitory computer-readable storage medium storing computer instructions, which cause the computer to perform the dynamic emotional interaction method based on multi-modal physiological signals and adaptive feedback.

[0092] Based on the same inventive concept as the foregoing embodiments, the embodiments of the present application also provide a computer program product comprising a computer program, which, when executed by a processor, implements the dynamic emotional interaction method based on multi-modal physiological signals and adaptive feedback.

[0093] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) containing computer-usable program code.

[0094] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0095] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0096] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0097] In summary, the present invention provides a multimodal emotion recognition and dynamic interaction system and method based on physiological signals. The multimodal physiological signal recognition and alignment module of the system is based on the pre-trained multimodal large language model Emotion-LLaMA, including MERR dataset construction, multimodal Emotion-LLaMA model architecture and training program. The MERR dataset is constructed through a comprehensive process of emotion annotations in video data to expand the range of emotion categories and annotations of existing datasets; the multimodal emotion LLaMA model realizes multimodal encoding and alignment, and maps multimodal features to a unified semantic space through a linear projection layer for LLaMA model fusion analysis; finally, the basic emotion feature learning of the model is realized through pre-training and instruction fine-tuning, thereby enhancing the model reasoning ability.

[0098] The adaptive feedback dynamic emotion interaction system disclosed by the application is based on an AGDF model, features are decomposed through contrast learning, and an adaptive gating mechanism and a graph attention network are fused, feature decoupling and dynamic fusion are introduced into the Emotion-LLaMA, so that the model can effectively balance unique information and common information of different modalities, and fine feature interaction is realized.

[0099] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A dynamic emotional interaction system based on multimodal physiological signals and adaptive feedback, characterized by: include: Multimodal physiological signal recognition and alignment module, which is used to extract low-level features of visual, speech, and text modalities from the multimodal emotion dataset MERR and perform preprocessing; The cross-modal fusion module is used to realize inter-modal interaction based on the cross-modal attention module, and uses a multi-head self-attention mechanism to calculate the correlation between modalities and generate a fused cross-modal feature sequence; The adaptive feedback mechanism module is used to decouple the fused cross-modal features to obtain common features shared across modalities and unique features exclusive to each modality, and then dynamically fuse the decoupled common features and unique features; The sentiment classification module is used to output discrete sentiment category labels with intensity through a classification model based on the dynamically fused multimodal features; The dynamic feedback mechanism module is used to adaptively adjust the emotion category labels based on the emotion classification results and combined with real-time environment or user feedback data.

2. The dynamic emotion interaction system based on multimodal physiological signals and adaptive feedback according to claim 1, characterized in that: The multimodal physiological signal recognition and alignment module is further configured to execute the following instructions: The extracted low-level features are passed through a one-dimensional spatiotemporal convolutional layer and then a positional encoding sequence is added. The positional encoding is constructed in the same way as in the Transformer model. The visual, speech and text information are preprocessed separately to obtain multimodal features of vision, speech and text.

3. The dynamic emotion interaction system based on multimodal physiological signals and adaptive feedback according to claim 1, characterized in that: The adaptive feedback mechanism module is further configured to execute the following instructions: Invoke the adaptive gating mechanism to calculate the offset of speech and visual features in the text semantic space and dynamically adjust the weight of the modality; The inter-modality relationship graph is constructed through the graph attention network GAT, where the nodes represent the feature vectors of each modality and the edge weights are calculated through the attention mechanism.

4. The dynamic emotion interaction system based on multimodal physiological signals and adaptive feedback according to claim 1, characterized in that: The adaptive feedback mechanism module is further configured to execute the following instructions: Before feature decoupling, the fused cross-modal features are encoded and aligned.

5. The dynamic emotion interaction system based on multimodal physiological signals and adaptive feedback according to claim 1, characterized in that: The dynamic feedback mechanism module is further used to execute the following instructions: Collect user physiological response data in real time and generate feedback evaluation indicators; The sentiment classification model parameters are optimized through reinforcement learning, and the loss function is: ， in For labels adjusted based on feedback, is the sentiment label predicted by the model, β is the weight coefficient, and KL divergence is used to constrain the model update amplitude.

6. A dynamic emotional interaction method based on multimodal physiological signals and adaptive feedback, characterized in that: include: Extract low-level features of visual, speech, and text modalities from the multimodal sentiment dataset MERR and perform preprocessing; It implements inter-modal interaction based on the cross-modal attention module and uses a multi-head self-attention mechanism to calculate inter-modal correlations and generate a fused cross-modal feature sequence. Decouple the fused cross-modal features to obtain common features shared across modalities and unique features unique to each modality, and then dynamically fuse the decoupled common features and unique features. Based on the dynamically fused multimodal features, the classification model outputs discrete emotion category labels with intensity. Based on the sentiment classification results, combined with real-time environment or user feedback data, the sentiment category labels are adaptively adjusted.

7. The dynamic emotion interaction method based on multimodal physiological signals and adaptive feedback according to claim 6, characterized in that: Dynamically integrate the decoupled common features and unique features, including: Invoke the adaptive gating mechanism to calculate the offset of speech and visual features in the text semantic space and dynamically adjust the weight of the modality; The inter-modality relationship graph is constructed through the graph attention network GAT, where the nodes represent the feature vectors of each modality and the edge weights are calculated through the attention mechanism.

8. A dynamic emotional interaction device based on multimodal physiological signals and adaptive feedback, characterized in that: It includes a memory and a processor, the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the dynamic emotion interaction method based on multimodal physiological signals and adaptive feedback as described in any one of claims 6 to 7.

9. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions enable the computer to execute the dynamic emotion interaction method based on multimodal physiological signals and adaptive feedback according to any one of claims 6 to 7.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the dynamic emotion interaction method based on multimodal physiological signals and adaptive feedback according to any one of claims 6 to 7 is implemented.

Images