AI interaction method and device based on emotion recognition, electronic device, and storage medium

Abstract

The application relates to an AI interaction method and device based on emotion recognition, an electronic device and a storage medium. The method comprises the following steps: acquiring multi-modal data of a target user; performing feature fusion on the multi-modal data by using a pre-trained emotion interaction engine to obtain feature fusion data; performing emotion recognition and context analysis on the feature fusion data respectively to obtain an emotion recognition result and a comprehensive representation of a current state of the user; generating an AI response strategy corresponding to the target user based on the emotion recognition result and the comprehensive representation of the current state of the user, and determining target emotion relief content from a pre-constructed emotion relief database; and generating response content based on the AI response strategy, the feature fusion data, the emotion recognition result and the target emotion relief content, and outputting the response content. In this way, the accuracy of the emotion recognition result can be improved, the generated response content can meet the personal preferences of the target user, and the purpose of improving the AI interaction effect is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an AI interaction method and device based on emotion recognition, an electronic device, and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, AI interaction technology based on emotion recognition can recognize user emotions and provide positive emotional value for users, and thus has been widely used in many fields, such as mental health support, customer service, and education guidance.

[0003] However, in the prior art, a traditional large language model is usually used to perform emotion recognition on single-modal data, such as using a GTP model to perform emotion recognition on user input text or voice, and then implementing a simple question and answer dialogue, so there is a problem of poor AI interaction effect. Therefore, how to improve the AI interaction effect has become a technical problem to be solved. SUMMARY

[0004] The present application provides an AI interaction method and device based on emotion recognition, an electronic device, and a storage medium to solve the problem of poor AI interaction effect in the prior art.

[0005] In a first aspect, the present application provides an AI interaction method based on emotion recognition, which comprises:

[0006] obtaining multi-modal data of a target user;

[0007] performing feature fusion on the multi-modal data by using a pre-trained emotion interaction engine to obtain feature fusion data;

[0008] performing emotion recognition and context analysis on the feature fusion data respectively to obtain an emotion recognition result and a comprehensive representation of a current state of the user;

[0009] generating an AI response strategy corresponding to the target user based on the emotion recognition result and the comprehensive representation of the current state of the user, and determining target emotion relief content from a pre-constructed emotion relief database, wherein the emotion relief database stores a plurality of emotion relief contents, and each emotion relief content corresponds to one attribute label;

[0010] generating response content based on the AI response strategy, the feature fusion data, the emotion recognition result, and the target emotion relief content, and outputting the response content.

[0011] Optionally, the feature fusion of the multi-modal data is performed by using the pre-trained emotional interaction engine to obtain feature fusion data, including:

[0012] The multi-modal data is preprocessed by using the pre-trained emotional interaction engine to obtain feature data corresponding to each modality data, wherein the multi-modal data includes at least one of image data, voice data, touch data, heart rate data, and motion data;

[0013] The feature data corresponding to each modality data is first feature fused by using a graph attention network to obtain an intermediate feature vector corresponding to each modality data, wherein the intermediate feature vector is used to reflect the association between the self-modality data and other modality data;

[0014] The intermediate feature vector is second feature fused by using a deep learning model to obtain the feature fusion data, wherein the feature fusion data is used to represent the time sequence features of each modality data.

[0015] Optionally, the feature fusion data is subjected to emotional recognition and context analysis respectively to obtain an emotional recognition result and a comprehensive representation of a current state of a user, including:

[0016] The feature fusion data is subjected to emotional recognition by using an emotional recognition model to obtain the emotional recognition result, wherein the emotional recognition result is used to represent the emotional category and emotional intensity of the target user;

[0017] The historical interaction data of the target user is obtained, and context analysis is performed based on the historical interaction data and the feature fusion data to generate a comprehensive representation of the current state of the user, wherein the comprehensive representation of the current state of the user is used to represent the emotional category, emotional intensity, physiological state, and behavior state of the target user.

[0018] Optionally, the AI response strategy corresponding to the target user is generated based on the emotional recognition result and the comprehensive representation of the current state of the user, and target emotional relief content is determined from a pre-constructed emotional relief database, including:

[0019] Based on the emotional recognition result and the comprehensive representation of the current state of the user, a causal diagram is constructed, wherein the causal diagram is used to represent the causal relationship between the user state and the response action, and the user state includes the emotional recognition result and the comprehensive representation of the current state of the user;

[0020] Based on the causal diagram and a preset reward function, the AI response strategy corresponding to the target user is determined, wherein the AI response strategy is used to represent one or more response actions.

[0021] obtaining historical interaction data of the target user, and determining a historical preference vector of the target user based on the historical interaction data of the target user, and determining the target emotion relief content from the emotion relief database based on the historical preference vector of the target user and the comprehensive representation of the current state of the user.

[0022] Optionally, the generating response content based on the AI response strategy, the feature fusion data, the emotion recognition result and the target emotion relief content, and outputting the response content, comprises:

[0023] in a case where the AI response strategy comprises text response, generating target response text based on the feature fusion data and the emotion recognition result;

[0024] in a case where the AI response strategy comprises voice response, generating target response text based on the feature fusion data and the emotion recognition result, and converting the target response text into voice to obtain target response voice;

[0025] in a case where the AI response strategy comprises multimedia content response, obtaining target multimedia content based on the target emotion relief content, wherein the type of the target multimedia content comprises at least one of music, video and picture;

[0026] in a case where the AI response strategy comprises physical feedback response, generating target physical feedback based on the emotion recognition result, wherein the target physical feedback comprises at least one of vibration of a haptic vibration motor and mechanical action performed by a flexible mechanical component.

[0027] Optionally, after the generating response content based on the AI response strategy, the feature fusion data, the emotion recognition result and the target emotion relief content, and outputting the response content, the method further comprises:

[0028] collecting feedback of the target user based on the response content to obtain multi-modal feedback data;

[0029] using the feedback data to adaptively optimize model parameters of the emotion interaction engine.

[0030] Optionally, before the using a pre-trained emotion interaction engine to perform feature fusion on the multi-modal data to obtain feature fusion data, the method further comprises:

[0031] training initial model parameters of the emotion interaction engine using meta-learning, wherein the initial model parameters are learned based on multiple learning tasks of a single user.

[0032] training the global model parameters of the sentiment interaction engine by federated learning, wherein the global model parameters are aggregated based on the initial model parameters of multiple users.

[0033] In a second aspect, the embodiments of the present application further provide an AI interaction device based on sentiment recognition, the device comprising:

[0034] an acquisition module configured to acquire multi-modal data of a target user;

[0035] a feature fusion module configured to perform feature fusion on the multi-modal data by using a pre-trained sentiment interaction engine to obtain feature fusion data;

[0036] an analysis module configured to perform sentiment recognition and context analysis on the feature fusion data respectively to obtain a sentiment recognition result and a comprehensive representation of a current state of the user;

[0037] a determination module configured to generate an AI response strategy corresponding to the target user based on the sentiment recognition result and the comprehensive representation of the current state of the user, and determine target emotion relief content from a pre-constructed emotion relief database, wherein the emotion relief database stores a plurality of emotion relief contents, and each of the emotion relief contents corresponds to an attribute label one by one;

[0038] an output module configured to generate response content based on the AI response strategy, the feature fusion data, the sentiment recognition result and the target emotion relief content, and output the response content.

[0039] In a third aspect, the embodiments of the present application further provide an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus.

[0040] The memory is configured to store a computer program.

[0041] The processor is configured to execute the program stored on the memory to implement the AI interaction method based on sentiment recognition of any one of the first aspect.

[0042] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the AI interaction method based on sentiment recognition of any one of the first aspect.

[0043] The above technical solutions provided by the embodiments of the present application have the following advantages compared with the prior art:

[0044] The method provided by the embodiment of the application comprises the following steps: obtaining multi-modal data of a target user; performing feature fusion on the multi-modal data by using a pre-trained emotion interaction engine to obtain feature fusion data; performing emotion recognition and context analysis on the feature fusion data respectively to obtain an emotion recognition result and a comprehensive representation of a current state of the user; generating an AI response strategy corresponding to the target user based on the emotion recognition result and the comprehensive representation of the current state of the user, and determining target emotion relief content from a pre-constructed emotion relief database, wherein the emotion relief database stores a plurality of emotion relief contents, and each emotion relief content corresponds to an attribute label in a one-to-one manner; and generating response content based on the AI response strategy, the feature fusion data, the emotion recognition result and the target emotion relief content, and outputting the response content. In this way, the multi-modal data of the target user can be fused and emotion-recognized by using the pre-trained emotion interaction engine, and an AI response strategy that meets the individuality of the target user can be generated based on the emotion recognition result and the comprehensive representation of the current state of the user, thereby improving the accuracy of the emotion recognition result and enabling the generated response content to meet the personal preferences of the target user, and thus achieving the purpose of improving the AI interaction effect. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings, which are incorporated herein and constitute part of the specification, illustrate embodiments consistent with the application and, together with the description, serve to explain the principles of the application.

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0047] Figure 1 A flowchart of an AI interaction method based on emotion recognition provided by the embodiment of the application;

[0048] Figure 2 A flowchart of another AI interaction method based on emotion recognition provided by the embodiment of the application;

[0049] Figure 3 A structural diagram of an AI interaction device based on emotion recognition provided by the embodiment of the application;

[0050] Figure 4 A structural diagram of an electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0051] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0052] Referring to Figure 1 , Figure 1 A flowchart of an AI interaction method based on emotion recognition provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the AI interaction method based on emotion recognition can include the following steps: Figure 1

[0053] Step S101, acquiring multi-modal data of a target user.

[0054] It should be noted that the AI interaction method based on emotion recognition provided by the embodiments of the present application can be applied on a user terminal (such as a smart toy, an education terminal, a medical device, etc.) alone, and can also be applied on a user terminal and a cloud platform together, and the embodiments of the present application do not make specific limitations. When applied on a user terminal alone, the multi-modal data of the target user can be collected by the user terminal, and then the multi-modal data of the target user is subjected to feature fusion and emotion recognition to obtain AI response content for the target user; when applied on a user terminal and a cloud platform together, the multi-modal data of the target user can be collected by the user terminal and sent to the cloud platform, and then the multi-modal data of the target user is subjected to feature fusion and emotion recognition by the cloud platform to obtain AI response content returned to the user terminal.

[0055] Specifically, the target user can be any user who needs to perform AI interaction. The multi-modal data can be user data collected by a multi-source sensor, which can include but is not limited to image data, voice data, touch data, heart rate data and motion data, etc. The image data can be collected by a camera, the voice data can be collected by a microphone, the touch data can be collected by a flexible electronic skin, the heart rate data can be collected by a photoplethysmogram (PPG) sensor, and the motion data can be collected by an inertial measurement unit (IMU).

[0056] Step S102, performing feature fusion on the multi-modal data by using a pre-trained emotion interaction engine to obtain feature fusion data.

[0057] ​Specifically, the above-mentioned emotional interaction engine is pre-trained, and the emotional interaction engine can include a multi-modal feature fusion module, an emotion recognition module, a context analysis module, an AI response strategy generation module, a response content generation module, etc. Among them, the multi-modal feature fusion module is mainly used for feature fusion of multi-modal data to obtain feature fusion data. The emotion recognition module is mainly used for emotion recognition of the feature fusion data to obtain an emotion recognition result. The context analysis module is mainly used for context analysis of the feature fusion data to obtain a comprehensive representation of the current state of the user. The AI response strategy generation module is mainly used for generating an AI response strategy corresponding to the target user based on the emotion recognition result and the comprehensive representation of the current state of the user. The response content generation module is mainly used for generating response content based on the AI response strategy, the feature fusion data, the emotion recognition result and the target emotional relief content. The above-mentioned multi-modal feature fusion module can include a graph attention network (Graph Attention Network, GAT for short) and a deep learning model (such as a Transformer model, etc.). That is, the graph attention network and the deep learning model can be used to fuse the features of the multi-modal data to obtain the feature fusion data. Among them, the graph attention network generates a unified feature representation by analyzing the correlation between each modal data (for example, the cooperative change of facial expressions and heart rate). The deep learning model is used to process the dynamic change of data (such as the change of voice tone over time), to ensure that the timing features of the emotion are captured.

[0058] Step S103, respectively, emotion recognition and context analysis are performed on the feature fusion data to obtain an emotion recognition result and a comprehensive representation of the current state of the user.

[0059] Specifically, the above-mentioned emotion recognition module can be implemented by a bidirectional long short-term memory network (Bidirectional Long Short-Term Memory, BiLSTM for short) or a convolutional neural network-long short-term memory network (Convolutional Neural Networks-Long Short-Term Memory networks, CNN-LSTM for short) or other networks. The above-mentioned emotion recognition result is used to represent the probability distribution of the emotion of the target user, such as , wherein, represents the probability that the user is currently in the i-th emotional category, and satisfies .

[0060] The above context analysis is used to generate a comprehensive representation of the current state of the user in combination with the historical interaction data of the user and the real-time physiological state and behavior state of the user. The comprehensive representation of the current state of the user can include representations of dimensions such as an emotional category (such as “anxiety” and the like), an emotional intensity (1-5 levels and the like), a physiological state (such as a heart rate variability and the like), a behavior state (such as an activity level), and the like.

[0061] In step S104, an AI response strategy corresponding to the target user is generated based on the emotional recognition result and the comprehensive representation of the current state of the user, and a target emotional relief content is determined from a pre-constructed emotional relief database. The emotional relief database stores a plurality of emotional relief contents, and each emotional relief content corresponds to one attribute label.

[0062] Specifically, the AI response strategy generation module can be implemented by using causal reinforcement learning. The AI response strategy is used to represent one or more response actions in line with the user's preferences, such as playing music, nodding, and the like. The response content generation module can include a Bidirectional Encoder Representations from Transformers (BERT) model, an end-to-end text-to-speech synthesis model Tacotron, and the like. The BERT model is used to generate a target response text, and the Tacotron model is used to generate a target response voice. In the construction phase, the emotional relief database needs to collect multimedia data from the Internet , including soothing music, inspirational pictures, healing videos, physiological feedback samples (such as heart rate variability), and the like. Each emotional relief content is labeled with an attribute label , wherein, is the emotional category (such as “anxiety”) of the i-th emotional relief content, is the emotional intensity (1-5 levels) of the i-th emotional relief content, is the physiological effect (such as heart rate variability) of the i-th emotional relief content, is the behavior feature (such as activity level) of the i-th emotional relief content. The emotional relief database can be stored in a NoSQL database (such as MongoDB and the like) of a cloud platform, which supports fast retrieval. The target emotional relief content can refer to the emotional relief content with the highest matching degree with the emotional recognition result of the target user and the comprehensive representation of the current state of the user in the emotional relief database.

[0063] In step S105, a response content is generated based on the AI response strategy, the feature fusion data, the emotional recognition result, and the target emotional relief content, and the response content is output.

[0064] Specifically, the response content can be in the form of text, voice, multimedia, physical feedback, etc. When outputting the response content, the response can be performed according to the form of the response content, for example, for multimedia or text in the form of pictures, videos, etc., output through a display interface; for multimedia or voice in the form of music, etc., output through a loudspeaker; for physical feedback, output through vibration of a tactile vibration motor, action of a flexible mechanical component, etc.

[0065] In the above manner, the pre-trained emotional interaction engine can be used to fuse and perform emotional recognition on the multi-modal data of the target user, and an AI response strategy conforming to the individuality of the target user can be generated through the comprehensive representation of the emotional recognition result and the current state of the user, thereby improving the accuracy of the emotional recognition result, and enabling the generated response content to meet the personal preferences of the target user, and thus achieving the purpose of improving the AI interaction effect.

[0066] In an optional embodiment, the step S102 of using the pre-trained emotional interaction engine to fuse the features of the multi-modal data to obtain the feature fusion data includes:

[0067] The pre-trained emotional interaction engine is used to pre-process the multi-modal data to obtain feature data corresponding to each modality data, wherein the multi-modal data includes at least one of image data, voice data, touch data, heart rate data, and motion data;

[0068] The graph attention network is used to perform first feature fusion on the feature data corresponding to each modality data to obtain an intermediate feature vector corresponding to each modality data, wherein the intermediate feature vector is used to reflect the association between the own modality data and other modality data;

[0069] The deep learning model is used to perform second feature fusion on the intermediate feature vector to obtain the feature fusion data, wherein the feature fusion data is used to represent the time sequence features of each modality data.

[0070] Specifically, when the multi-modal feature fusion module in the pre-trained emotional interaction engine is used to fuse the features of the multi-modal data, the multi-modal data can be pre-processed first to obtain feature data corresponding to each modality data, then the graph attention network is used to perform first feature fusion on the feature data corresponding to each modality data to obtain an intermediate feature vector corresponding to each modality data, and then the deep learning model is used to perform second feature fusion on the intermediate feature vector to obtain the feature fusion data. When pre-processing the image data, the original image is adjusted to a fixed resolution (such as 224x224 pixels, etc.), and then normalized using the mean and standard deviation of the image pixel values (usually calculated based on a training data set). Specifically, the following formula can be used to represent:

[0071] ;

[0072] wherein, represents the original image data at time t (the original image data is usually the image of the user's face captured by the camera, containing expression information such as frowning and smiling), represents the original image data the image feature vector after preprocessing, represents the original image is adjusted to a fixed resolution of 224x224 pixels, and of course can also be other resolutions, so as to unify the image size, facilitate model processing, and ensure input consistency. represents the mean value of the image pixel value, represents the standard deviation of the adjusted image pixel value, and the mean value is subtracted in the formula can make the pixel value centered on 0, reducing the data distribution deviation. In the formula, the standard deviation is divided by can make the pixel value scale consistent, improving the stability of model training.

[0073] When preprocessing the speech data, the original speech signal can be denoised, and then the Mel-frequency cepstral coefficients of the denoised speech signal can be extracted to obtain a speech feature matrix. Specifically, the following formula can be used to represent:

[0074] ;

[0075] wherein, represents the original speech signal at time t (the original speech signal can be collected by a microphone, and contains speech features such as tone and speed, which can reflect the user's emotions). represents the original speech signal the speech feature matrix after preprocessing, the dimension of the speech feature matrix is , and the speech feature matrix represents that each frame of speech contains 13 cepstral coefficients, represents the number of frames of the speech segment. represents that the original speech signal is denoised to remove background noise and improve the quality of the speech signal, ensuring the accuracy of the extracted features. represents the extraction process of the Mel-frequency cepstral coefficients (Mel-Frequency Cepstral Coefficients), which is used to convert the denoised speech signal into a set of features, capture the frequency and time domain characteristics of the speech, and is suitable for emotion analysis.

[0076] In preprocessing the touch data, the original touch data can be normalized to obtain a touch feature vector. Specifically, the following formula can be used to represent:

[0077]

[0078] wherein, represents the original touch data at time t (which can be the touch force collected by the flexible electronic skin sensor, and can reflect user interaction behaviors such as caressing and pressing hard, and provide emotional cues). represents the original touch data The touch feature vector after preprocessing. Norm( ) represents the normalization of the original touch data to scale the touch force to the range of 0 to 1, eliminate the dimensional difference, and facilitate model processing. represents the maximum value in the original touch data , which serves as the denominator of the normalization, ensuring that the touch data divided by the maximum value is in the range of 0 to 1.

[0079] In preprocessing the heart rate data, the original heart rate signal can be filtered, and then the heart rate variability can be calculated based on the filtered heart rate signal to obtain a heart rate feature vector. Specifically, the following formula can be used to represent:

[0080]

[0081] wherein, represents the original heart rate signal at time t (which can be collected by a PPG sensor, and can reflect user physiological states such as heart rate, for emotional analysis). represents the filtered heart rate signal after filtering the original heart rate signal , which is used to calculate the heart rate variability (Heart Rate Variability, abbreviated as HRV). represents the application of a band-pass filter (Band-Pass Filter, abbreviated as ) to filter the original heart rate signal . The frequency range of the band-pass filter can be 0.5Hz to 5Hz. In this way, low-frequency drift and high-frequency noise can be removed, and the main signal related to heart rate can be preserved. represents the heart rate variability, which measures the fluctuation of the heart rate signal, and can reflect the emotional state of the user. represents the calculation formula of HRV, and the calculation result is the standard deviation based on the filtered heart rate signal, which quantifies the heart rate fluctuation. The larger the standard deviation, the higher the HRV, reflecting the emotional change. ​​This indicates the number of sampling points for the heart rate signal. Represents the filtered heart rate signal The mean.

[0082] When preprocessing motion data, the raw IMU data can be subjected to a Fast Fourier Transform (FFT) to obtain the IMU feature vector. This can be represented by the following formula:

[0083] ;

[0084] in, The raw IMU data at time t includes acceleration and angular velocity. It reflects the user's motion state, such as being stationary or jittering, and provides behavioral characteristics. This indicates that a Fast Fourier Transform (FFT) is applied to the raw IMU data. This allows for the conversion of time-domain signals into frequency-domain signals, enabling the extraction of frequency features of motion, such as jitter frequency. Represents raw IMU data The preprocessed IMU feature vector contains information such as jitter frequency. This represents the sum of squares of acceleration along the x, y, and z axes in the raw IMU data. This allows for the synthesis of triaxial accelerations to calculate the total motion intensity, which is used to extract jitter frequencies. This represents the absolute value of the spectral amplitude of the total acceleration, which reflects the frequency distribution of the motion. The larger the amplitude, the more significant the motion at that frequency. argmax represents the frequency index for taking the maximum value, thus identifying the frequency with the largest amplitude in the spectrum as the main jitter frequency. This represents the main jitter frequency extracted, which serves as an IMU feature vector and reflects the user's motion pattern. For example, high-frequency jitter indicates tension.

[0085] When using a graph attention network to perform the first feature fusion on the feature data corresponding to each modality to obtain the intermediate feature vectors corresponding to each modality, the preprocessed data can be used... The data is input into an attention network, which allows the network to capture the correlations between different modalities. Specifically, GAT updates node features through an attention mechanism, using the following formula:

[0086] ;

[0087] in, Let i be the i-th modal feature (such as facial expression). Let j be the modal feature (e.g., heart rate). for and attention weight between two modalities, whose value ranges from 0 to 1. The determination of the contribution size of the update, the greater the value, the greater the influence of on . W represents a learnable weight matrix, and σ is a ReLU activation function, which is used for nonlinear transformation of the fused features, enhances the model expression ability, and highlights important features. represents the i-th modal feature in the updated feature vector of the l+1-th layer (next layer) of GAT. It is the new feature representation output by GAT, which fuses the information of the modal feature itself and other related modalities, and is used for subsequent sentiment recognition. represents the feature vector of the j-th modal feature in the l-th layer (current layer) of GAT. represents the i-th modal feature of the neighbor modal. represents a scaling factor, which is used to prevent the similarity value from being too large or too small.

[0088] In the above manner, the node features of each layer of GAT can be updated, and finally the intermediate feature vectors corresponding to the modal data , i.e., the node features of the last layer of GAT, can be obtained.

[0089] Then, the intermediate feature vectors corresponding to the modal data are input into a deep learning model (such as a Transformer model) for second feature fusion to obtain feature fusion data. Specifically, it can be represented by the following formula:

[0090] ;

[0091] wherein, represents the feature fusion data output by the model, which is the feature representation after Transformer processing and contains the time series information of multi-modal data (such as the change of emotion over time), which is used for subsequent tasks. represents the intermediate feature vector output to the model, which contains the comprehensive information of the expression, voice, heart rate, touch, motion and other modalities, reflecting the current state of the user. Q, K and V are query (Query), key (key) and value (Value), , , , , and These are the learnable weight matrices. This represents the normalization factor.

[0092] In the above manner, graph attention networks can be used to perform first feature fusion on the feature data corresponding to each modality. Through the attention mechanism, each modality is dynamically integrated to generate a unified feature representation. Then, a deep learning model is used to perform second feature fusion on the intermediate feature vectors to extract the temporal features of each modality data and obtain feature fusion data. This facilitates subsequent sentiment classification and response content generation based on the feature fusion data.

[0093] In an optional embodiment, step S103 above, which involves performing sentiment recognition and context analysis on the feature fusion data to obtain a comprehensive representation of the sentiment recognition result and the user's current state, includes:

[0094] An emotion recognition model is used to perform emotion recognition on feature fusion data to obtain emotion recognition results, which are used to characterize the emotion category and intensity of the target user.

[0095] The system acquires historical interaction data of the target user and performs contextual analysis based on the historical interaction data and feature fusion data to generate a comprehensive representation of the user's current state. This comprehensive representation of the user's current state is used to characterize the target user's emotional category, emotional intensity, physiological state, and behavioral state.

[0096] Specifically, a Bidirectional Long Short-Term Memory (BiLSTM) network can be used as the emotion recognition model to perform emotion recognition on feature fusion data, obtaining the emotion recognition result, namely, the emotion category and intensity of the target user. The processing flow of the BiLSTM network is as follows:

[0097] ;

[0098] ;

[0099] ;

[0100] in, This represents feature fusion data, which can be generated by processing multi-source data (such as facial expressions, speech, heart rate, touch, etc.) using a Graph Attention Network (GAT) and a Transformer. It serves as the input for emotion recognition, containing comprehensive cross-modal information, and its dimensionality depends on the model design. and These represent the hidden states of the Bidirectional Long Short-Term Memory (BiLSTM) network at time step t (forward and backward), respectively. BiLSTM is bidirectional. For the forward LSTM, the context from the beginning of the sequence to time step t-1 is captured; For the backward LSTM, the context from the end of the sequence to time step t+1 is captured. These hidden states can help the model understand the contextual dependencies in the time series. and denote the output hidden states of the forward and backward LSTMs at time step t, respectively. is calculated based on and reflecting the information from the past to the present; is calculated based on and reflecting the information from the future to the present. and provide the complete temporal context in combination. denotes the concatenation vector of the forward and backward hidden states, with dimension and twice the dimension. The concatenated vector integrates the bidirectional contextual information for subsequent sentiment classification. denotes the weight matrix of the output layer, which maps the concatenated hidden state to the dimension of sentiment categories (i.e., k dimensions, where k is the number of sentiment categories). denotes the bias vector of the output layer, with dimension k, which is used to adjust the results of linear transformation and enhance the expressive ability of the model. denotes the sentiment classification output at time step t, which is a k-dimensional vector, and after normalization by the softmax function, it represents the probability distribution, i.e., where, is the probability of the i-th sentiment category, satisfying . denotes the sentiment probability distribution, which is a k-dimensional vector .

[0101] In addition, the sentiment interaction engine can generate a comprehensive representation of the user's current state based on the user's historical interaction data and real-time physiological and behavioral characteristics (such as heart rate variability, activity level, etc.). This comprehensive representation of the user's current state contains information corresponding to the database annotation attributes (such as sentiment categories, sentiment intensity, physiological state, and behavioral state, etc.). In this way, it is convenient for subsequent determination of the target emotion relief content from the emotion relief database based on the comprehensive representation of the user's current state.

[0102] In an optional embodiment, the above step S104, based on the sentiment recognition result and the comprehensive representation of the user's current state, generates an AI response strategy corresponding to the target user, and determines the target emotion relief content from the pre-constructed emotion relief database, including:

[0103] Based on the comprehensive representation of the emotion recognition result and the current state of the user, a causal graph is constructed, wherein the causal graph is used to represent the causal relationship between the user state and the response action, and the user state includes the comprehensive representation of the emotion recognition result and the current state of the user;

[0104] Based on the causal graph and the preset reward function, an AI response strategy corresponding to the target user is determined, wherein the AI response strategy is used to represent one or more response actions;

[0105] The historical interaction data of the target user is obtained, and based on the historical interaction data of the target user, a historical preference vector of the target user is determined, and based on the historical preference vector of the target user and the comprehensive representation of the current state of the user, target emotion relief content is determined from an emotion relief database.

[0106] Specifically, when generating the AI response strategy corresponding to the target user based on the comprehensive representation of the emotion recognition result and the current state of the user, a causal graph can be constructed based on the comprehensive representation of the emotion recognition result and the current state of the user, and an AI response strategy corresponding to the target user can be determined based on the causal graph and a preset reward function. The causal graph can be represented as G=(V,E), wherein G represents the causal graph, V represents the node set of the causal graph, and each node can include emotion state , fusion feature , action and the like. E represents the edge set of the causal graph, which is used to describe the causal relationship between variables, such as emotion state influencing action or action influencing next time emotion . The causal graph is used to model the causal association between the emotion state and the response action, and to improve the explainability of the strategy. The reward function can be represented as , wherein represents the state vector at time step t, which is composed of the current emotion state and the feature fusion data , i.e. . is the current emotion category (usually the category with the highest probability or some representation of the probability distribution) extracted from . provides multi-modal context information to enhance the richness of the state. is the input data of reinforcement learning, which is used to determine the action. represents the action at time step t, such as “playing music”, “displaying healing video”, “executing haptic feedback”, etc. represents the reward function of executing action in state . represents the state perform an action , the expected value of the affective state at the next time instant. is similar to is the affective probability distribution at the next time instant. This part measures the immediate impact of the action on the user's affect. represents the difference in long-term effects based on the Q value, is the weight coefficient for balancing immediate and long-term rewards. is the reward Q value of the current action, is the reward Q value of other alternative actions, which part optimizes the strategy by comparing the long-term value of the action. represents the reward Q value of the state-action pair measures the long-term cumulative reward of performing action a in state s. , is the concatenation vector of state and action, is the learnable weight matrix. The Q value is used to evaluate the pros and cons of the action, and to optimize the long-term interaction effect. In obtaining the target emotional relief content, the historical interaction data of the target user can be obtained, and based on the historical interaction data of the target user, a historical preference vector of the target user is determined, and based on the historical preference vector of the target user and the comprehensive representation of the current state of the user, the target emotional relief content is determined from the emotional relief database. Wherein, the target emotional relief content determined from the emotional relief database can be realized through the following formula:

[0107]

[0108] ;

[0109] Wherein, C represents the extracted target emotional relief content (such as soothing music, inspirational pictures, healing videos, etc.). D represents the emotional relief database, which stores multimedia content annotated with attributes such as emotional category, emotional intensity, physiological state and behavior state. S represents the comprehensive representation of the current state of the user. represents the similarity between the comprehensive representation S of the current state of the user and the emotional relief content c in the emotional relief database. represents the historical preference vector of the target user, reflecting the user's past interaction preferences (such as preference for a certain type of music, etc.). represents the matching degree of the historical preference vector and the emotional relief content c, which is usually calculated by vector inner product. represents the weight parameter, which is used to balance the similarity score and the matching degree score.

[0110] ​In this way, by constructing a causal graph and combining a reward function to optimize the AI response strategy, the interaction strategy can be dynamically optimized, and the optimal response can be selected according to user feedback (such as heart rate changes, touch feedback), rather than relying on static mapping. And the causal reinforcement learning evaluates the potential effects of different actions through counterfactual reasoning, for example, "what would happen if another response was chosen". This ensures that the system prioritizes actions that contribute to long-term improvement of user emotions (such as sustained anxiety relief). The optimized reward is based on Q-value, combined with the historical preference vector , ensuring that the strategy takes into account the user's historical interaction data, thereby improving long-term user satisfaction. In addition, since the causal graph G =(V,E) explicitly models the causal relationship between state S and action A, such as "playing a healing video directly leading to a decrease in heart rate", this makes the interaction strategy more interpretable, making it easier to analyze which responses are effective for emotional improvement, which is better than the "black box" method of direct mapping.

[0111] In an optional embodiment, the above step S105, based on the AI response strategy, the feature fusion data, the emotion recognition result and the target emotional relief content, generates the response content, and outputs the response content, including:

[0112] In the case where the AI response strategy includes a text response, a target response text is generated based on the feature fusion data and the emotion recognition result;

[0113] In the case where the AI response strategy includes a voice response, a target response text is generated based on the feature fusion data and the emotion recognition result, and the target response text is converted into voice to obtain a target response voice;

[0114] In the case where the AI response strategy includes a multimedia content response, a target multimedia content is obtained based on the target emotional relief content, wherein the type of the target multimedia content includes at least one of music, video and picture;

[0115] In the case where the AI response strategy includes a physical feedback response, a target physical feedback is generated based on the emotion recognition result, wherein the target physical feedback includes at least one of vibration of a haptic vibration motor and mechanical action performed by a flexible mechanical component.

[0116] Specifically, in the case where the AI response strategy includes a text response, a target response text can be generated based on the feature fusion data and the emotion recognition result. The implementation process is as follows:

[0117] ;

[0118] wherein, represents the generated target response text, which is the output of the BERT model. is a piece of natural language text used for language interaction with the user (such as comforting words or guiding sentences). represents the current emotional state at time step t, usually the highest probability emotion category (such as "happy", "anxious") extracted from the emotion probability distribution or the probability distribution itself. is used to guide the BERT model to generate text content that matches the user's emotions. represents feature fusion data, which is processed by a graph attention network and a Transformer to generate multi-source data (such as expressions, speech, heart rate, touch, motion, etc.). provides rich contextual information to help the BERT model generate more accurate and personalized text. represents the t-th word in the generated target response text. In the generation process of the BERT model, is the next word predicted based on the previous text , emotional state and feature fusion data . represents the probability of generating word under the condition of given previous text , emotional state and feature fusion data . The BERT model selects the most suitable word by maximizing this probability to generate the complete text T.

[0119] In the case of AI response strategy including voice response, the target response text can be generated based on feature fusion data and emotion recognition results, and the target response text can be converted into voice to obtain the target response voice. The implementation process is as follows:

[0120] ;

[0121] where, represents the target response voice synthesized by the Tacotron2 model, which is the output of converting the target response text T into audio. is a two-dimensional matrix with dimensions , where represents the time step of the target response voice, i.e. the total number of frames of the target response voice, which depends on the length of the target response text T and the sampling rate of voice synthesis. represents the feature dimension of each frame of the target response voice, usually the dimension of the mel spectrum output by Tacotron2. describes the spectral characteristics of the voice, which is used for subsequent vocoder to generate waveforms.

[0122] In the case that the AI response strategy includes multimedia content response, the target multimedia content can be obtained based on the target emotion relief content. The implementation process has been described in detail in the above embodiments, which will not be repeated here.

[0123] In the case that the AI response strategy includes physical feedback response, the target physical feedback can be generated based on the emotion recognition result, and the implementation process is as follows:

[0124] ;

[0125] ;

[0126] wherein, represents the vibration frequency (unit: Hz) of the haptic vibration motor, used for physical feedback, reflecting the intensity of the user's emotional state. represents the scaling factor of the vibration frequency, the size of k can be 10 Hz / unit. is used to convert the quantized value of the emotional state to the actual vibration frequency. represents the quantized value of the emotional state , which is usually a scalar calculated based on the emotion category or probability distribution. reflects the intensity of the emotion or the characteristics of a specific emotion, and determines the size of the vibration frequency. represents the action angle (unit: degree) of the flexible mechanical component, used for physical feedback (such as nodding or arm movement). represents the sign (positive or negative) of the emotional state , used to determine the direction of action. For example, positive emotions (such as "happy") may correspond to positive actions (such as nodding), and negative emotions (such as "sad") may correspond to negative actions (such as bowing). represents the intensity of the emotional state , defined as the maximum value of the emotion probability distribution P( ), i.e. Intensity( )=max(P( )), which is a scalar reflecting the confidence or intensity of the current emotion category, used to adjust the action amplitude.

[0127] In the above manner, various forms of response content can be obtained, making the AI response content more diverse. In particular, the haptic vibration motor simulates the heartbeat rhythm, and the flexible mechanical component realizes the action feedback, significantly enhancing the immersion and emotional expression ability of physical interaction compared to traditional hardware design.

[0128] In an optional embodiment, after the step S105 described above, based on the AI response strategy, the feature fusion data, the sentiment recognition result and the target emotional relief content, the response content is generated and output, the method further comprises:

[0129] Collecting feedback of the target user based on the response content to obtain multi-modal feedback data;

[0130] Adaptive optimization of the model parameters of the emotional interaction engine using the feedback data.

[0131] Specifically, after the response content is output, feedback of the target user based on the response content can also be collected to obtain multi-modal feedback data, such as new voice data, new heart rate data, etc. of the target user. Then the model parameters of the emotional interaction engine are adaptively optimized using the feedback data, and the specific process is as follows:

[0132] ;

[0133] Among them, represents an optimization objective function, which is used to evaluate and optimize the performance of the model parameters . The contributions of the sentiment recognition accuracy, voice feedback and heart rate feedback are integrated and calculated by weighted summation. represents the model parameters of the emotional interaction engine, including the weights of GAT, Transformer, BiLSTM and other networks. The optimization objective is to improve the model performance by adjusting . represents the accuracy of sentiment recognition, which measures the consistency of the model's predicted emotional output y (i.e. the class with the highest probability in P(e)) and the true emotional label . The calculation formula is the number of correctly predicted samples divided by the total number of samples, and the weight is 0.4, indicating that the accuracy is the main factor of optimization. y represents the model's predicted emotional output, which is usually the highest probability emotional category selected from , generated by the BiLSTM model, for comparison with the true emotional label . represents the user's true emotional label, which is usually obtained through labeling or external reference (such as user self-reporting). represents the average value of all users' voice feedback , reflecting the overall satisfaction of users with AI response, with a weight of 0.3 and N being the total number of users. represents the normalized value of heart rate change feedback , which is usually the result of standardization or absolute value processing of . Quantify the strength of physiological feedback, weight 0.3, reflect the contribution of heart rate change to model optimization.

[0134] In the above manner, the emotional interaction engine can be continuously optimized according to user feedback, so that the emotional interaction engine is more and more in line with user preferences.

[0135] In an optional embodiment, before the step S102 described above, the method further comprises:

[0136] The initial model parameters of the emotional interaction engine are trained by using meta-learning, wherein the initial model parameters are learned based on multiple learning tasks of a single user;

[0137] The global model parameters of the emotional interaction engine are trained by using federated learning to obtain the emotional interaction engine, wherein the global model parameters are aggregated based on the initial model parameters of multiple users.

[0138] Specifically, when the emotional interaction engine is trained, the initial model parameters of the emotional interaction engine can be trained by using meta-learning, and then the global model parameters of the emotional interaction engine are trained by using federated learning to obtain the emotional interaction engine. The meta-learning here can be a model-agnostic meta-learning (MAML) algorithm, which can be expressed by the following formula:

[0139] ;

[0140] Wherein, represents the original model parameters of the kth user, represents the learned model parameters of the kth user, represents the learning rate, represents the gradient calculation of represents the oth task, such as user emotion recognition. represents the loss function of task The federated learning (FL) here can be expressed by the following formula:

[0141] ;

[0142] Wherein, is the user weight, represents the global model parameters, represents the learned model parameters of the kth user, and N represents the total number of users.

[0143] ​In this way, meta-learning can be used to achieve fast adaptation across users, and federated learning can be used to optimize the global model through local training and cloud aggregation, avoiding the uploading of data sets, thereby improving cross-scenario adaptability and ensuring user privacy compared to traditional solutions.

[0144] In an optional embodiment, the AI interaction method based on emotion recognition provided by the embodiments of the present application can be applied to an emotion interaction system, which can include five core modules: a sensor module, a processing module, an output module, a cloud connection module, and a cloud platform. The sensor module integrates a high-definition camera (for expression collection), a microphone (for voice collection), a touch sensor (for interaction action collection), and adds a PPG sensor (for heart rate monitoring), a flexible electronic skin (for touch force perception), and an IMU (for motion data collection), realizing multi-dimensional data input. The processing module uses an embedded AI processor to run the emotion AI interaction engine, fuses multi-modal features through graph attention networks and Transformers, and improves model adaptability and privacy protection capability through meta-learning and federated learning. The output module includes a speaker (for voice response) and a display screen (for emotion relief content display), and adds a tactile vibration motor and a flexible mechanical component to provide physical feedback and action expression, enhancing the interactive experience. The cloud connection module communicates with the cloud platform through the network, accesses the emotion relief database, and aggregates model parameters to realize dynamic optimization. The hardware connection relationship is that the data collected by the sensor module is transmitted to the processing module, the processing module generates real-time responses, and then presents them through the output module, while interacting with the optimization engine of the cloud platform. This architecture significantly improves emotion recognition accuracy, interaction intelligence, and cross-scenario adaptability through multi-source data fusion and causal reinforcement learning, and is suitable for smart toys, education, and medical fields, with technical leading nature and application expansion potential. The AI interaction process based on emotion recognition thereof is shown in Figure 2 as follows:

[0145] Step S201, constructing an emotion relief database and an emotion interaction engine.

[0146] Step S202, collecting and fusing multi-source multi-modal data.

[0147] Step S203, generating AI response strategies and target emotion relief content based on causal reasoning.

[0148] Step S204, multi-sensory real-time AI response output.

[0149] Step S205, adaptive model optimization based on multi-source feedback.

[0150] The application significantly improves the emotion recognition and AI interaction performance through multi-source data fusion, fusion of multiple algorithms and hardware design. The emotion recognition accuracy and robustness are improved, and the model can work stably in complex environments. The model quickly adapts to new users, and the cross-scene adaptability is enhanced. The interaction strategy is more intelligent, the user satisfaction is improved, and the long-term effect is better. The addition of flexible skin and tactile feedback improves the sense of immersion and provides a richer user experience. Privacy is protected. The application realizes high precision, intelligence and excellent experience in the fields of intelligent toys, education and the like, and has significant practical value.

[0151] Referring to Figure 3 , Figure 3 A structural schematic diagram of an AI interaction device based on emotion recognition provided by an embodiment of the application is shown in FIG. 1. As shown in FIG. 1, the AI interaction device 300 based on emotion recognition includes: Figure 3

[0152] The acquisition module 301 is configured to acquire multi-modal data of a target user.

[0153] The feature fusion module 302 is configured to perform feature fusion on the multi-modal data by using a pre-trained emotion interaction engine to obtain feature fusion data.

[0154] The analysis module 303 is configured to perform emotion recognition and context analysis on the feature fusion data respectively to obtain an emotion recognition result and a comprehensive representation of a current state of the user.

[0155] The determination module 304 is configured to generate an AI response strategy corresponding to the target user based on the emotion recognition result and the comprehensive representation of the current state of the user, and determine target emotion relief content from a pre-constructed emotion relief database, wherein the emotion relief database stores a plurality of emotion relief contents, and each emotion relief content corresponds to one attribute tag.

[0156] The output module 305 is configured to generate response content based on the AI response strategy, the feature fusion data, the emotion recognition result and the target emotion relief content, and output the response content.

[0157] Further, the feature fusion module 302 includes:

[0158] The preprocessing sub-module is configured to perform preprocessing on the multi-modal data by using the pre-trained emotion interaction engine to obtain feature data corresponding to each modality data, wherein the multi-modal data includes at least one of image data, voice data, touch data, heart rate data and motion data.

[0159] ​The first fusion sub-module is configured to perform first feature fusion on the feature data corresponding to each modality data by using a graph attention network to obtain an intermediate feature vector corresponding to each modality data, wherein the intermediate feature vector is used to reflect the correlation between the modality data itself and other modality data.

[0160] The second fusion sub-module is configured to perform second feature fusion on the intermediate feature vector by using a deep learning model to obtain feature fusion data, wherein the feature fusion data is used to represent the time sequence features of each modality data.

[0161] Further, the analysis module 303 comprises:

[0162] The sentiment recognition sub-module is configured to perform sentiment recognition on the feature fusion data by using a sentiment recognition model to obtain a sentiment recognition result, wherein the sentiment recognition result is used to represent the sentiment category and sentiment intensity of the target user.

[0163] The analysis sub-module is configured to obtain historical interaction data of the target user, and perform context analysis based on the historical interaction data and the feature fusion data to generate a comprehensive representation of the current state of the user, wherein the comprehensive representation of the current state of the user is used to represent the sentiment category, sentiment intensity, physiological state and behavior state of the target user.

[0164] Further, the determination module 304 comprises:

[0165] The construction sub-module is configured to construct a causal graph based on the sentiment recognition result and the comprehensive representation of the current state of the user, wherein the causal graph is used to represent the causal relationship between the user state and the response action, and the user state includes the sentiment recognition result and the comprehensive representation of the current state of the user.

[0166] The first determination sub-module is configured to determine an AI response strategy corresponding to the target user based on the causal graph and a preset reward function, wherein the AI response strategy is used to represent one or more response actions.

[0167] The second determination sub-module is configured to obtain historical interaction data of the target user, and determine a historical preference vector of the target user based on the historical interaction data of the target user, and determine target emotion relief content from an emotion relief database based on the historical preference vector of the target user and the comprehensive representation of the current state of the user.

[0168] Further, the output module 305 comprises:

[0169] The first generation sub-module is configured to generate a target response text based on the feature fusion data and the sentiment recognition result in the case that the AI response strategy includes a text response.

[0170] The second generation sub-module is configured to, in a case where the AI response strategy comprises a voice response, generate a target response text based on the feature fusion data and the emotion recognition result, and convert the target response text into a voice to obtain a target response voice.

[0171] The third generation sub-module is configured to, in a case where the AI response strategy comprises a multimedia content response, obtain a target multimedia content based on the target emotion relief content, wherein the target multimedia content comprises at least one of music, a video, and a picture.

[0172] The fourth generation sub-module is configured to, in a case where the AI response strategy comprises a physical feedback response, generate a target physical feedback based on the emotion recognition result, wherein the target physical feedback comprises at least one of vibration of a haptic vibration motor and mechanical action performed by a flexible mechanical component.

[0173] Further, the AI interaction device 300 based on emotion recognition further comprises:

[0174] The collection module is configured to collect feedback of the target user based on the response content to obtain multi-modal feedback data.

[0175] The optimization module is configured to perform adaptive optimization on model parameters of the emotion interaction engine by using the feedback data.

[0176] Further, the AI interaction device 300 based on emotion recognition further comprises:

[0177] The first training module is configured to train initial model parameters of the emotion interaction engine by using meta-learning, wherein the initial model parameters are learned based on multiple learning tasks of a single user.

[0178] The second training module is configured to train global model parameters of the emotion interaction engine by using federated learning to obtain the emotion interaction engine, wherein the global model parameters are aggregated based on the initial model parameters of multiple users.

[0179] It should be noted that the AI interaction device 300 based on emotion recognition can implement the steps of the AI interaction method based on emotion recognition provided in any one of the foregoing method embodiments, and achieve the same technical effects, which will not be repeated here.

[0180] As shown in Figure 4 The present application also provides an electronic device, which comprises a processor 411, a communication interface 412, a memory 413, and a communication bus 414, wherein the processor 411, the communication interface 412, and the memory 413 communicate with each other through the communication bus 414,

[0181] The memory 413 is configured to store a computer program.

[0182] In an embodiment of the present application, the processor 411, when executing the program stored in the memory 413, implements the emotion recognition based AI interaction method provided by any one of the foregoing method embodiments.

[0183] The embodiments of the present application further provide a computer readable storage medium, which has a computer program stored thereon, and the computer program, when executed by a processor, implements the emotion recognition based AI interaction method provided by any one of the foregoing method embodiments.

[0184] It should be noted that, in this document, the terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0185] The above description is merely specific embodiments of the present application, enabling those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. An AI interaction method based on emotion recognition, characterized in that, The method comprises: acquiring multi-modal data of a target user; performing feature fusion on the multi-modal data by using a pre-trained emotional interaction engine to obtain feature fusion data; performing emotional recognition and context analysis on the feature fusion data respectively to obtain an emotional recognition result and a comprehensive representation of a current state of the user, wherein the feature fusion data is used to represent time sequence features of each modality data, the comprehensive representation of the current state of the user is obtained based on historical interaction data of the target user and the feature fusion data, and the comprehensive representation of the current state of the user is used to represent a sentiment category, a sentiment intensity, a physiological state and a behavior state of the target user; based on the emotional recognition result and the comprehensive representation of the current state of the user, generating an AI response strategy corresponding to the target user, and determining target emotional relief content from a pre-constructed emotional relief database, wherein the emotional relief database stores a plurality of emotional relief contents, and each emotional relief content corresponds to one attribute label; based on the AI response strategy, the feature fusion data, the emotional recognition result and the target emotional relief content, generating response content, and outputting the response content; wherein the generation of the AI response strategy corresponding to the target user based on the emotional recognition result and the comprehensive representation of the current state of the user, and the determination of the target emotional relief content from the pre-constructed emotional relief database, comprises: based on the emotional recognition result and the comprehensive representation of the current state of the user, constructing a causal graph, wherein the causal graph is used to represent the causal relationship between the user state and the response action, and the user state includes the emotional recognition result and the comprehensive representation of the current state of the user; based on the causal graph and a preset reward function, determining the AI response strategy corresponding to the target user, wherein the AI response strategy is used to represent one or more response actions; acquiring historical interaction data of the target user, and based on the historical interaction data of the target user, determining a historical preference vector of the target user, and based on the historical preference vector of the target user and the comprehensive representation of the current state of the user, determining the target emotional relief content from the emotional relief database. 2.The AI interaction method based on emotion recognition according to claim 1, characterized in that, The feature fusion of the multi-modal data by using the pre-trained emotional interaction engine to obtain the feature fusion data comprises: preprocessing the multi-modal data by using the pre-trained emotional interaction engine to obtain feature data corresponding to each modality data, wherein the multi-modal data includes at least one of image data, voice data, touch data, heart rate data and motion data; performing first feature fusion on the feature data corresponding to each modality data by using a graph attention network to obtain an intermediate feature vector corresponding to each modality data, wherein the intermediate feature vector is used to reflect the association between the self-modality data and other modality data; performing second feature fusion on the intermediate feature vector by using a deep learning model to obtain the feature fusion data. 3.The AI interaction method based on emotion recognition according to claim 1, characterized in that, The emotion recognition result and the comprehensive representation of the current state of the user are obtained by performing emotion recognition on the feature fusion data using an emotion recognition model, wherein the emotion recognition result is used to represent the emotion category and emotion intensity of the target user. The emotion recognition result and the comprehensive representation of the current state of the user are obtained by performing emotion recognition on the feature fusion data using an emotion recognition model, wherein the emotion recognition result is used to represent the emotion category and emotion intensity of the target user. The emotion recognition result and the comprehensive representation of the current state of the user are obtained by performing emotion recognition on the feature fusion data using an emotion recognition model, wherein the emotion recognition result is used to represent the emotion category and emotion intensity of the target user. 4.The AI interaction method based on emotion recognition according to claim 1, characterized in that, The response content is generated based on the AI response strategy, the feature fusion data, the emotion recognition result, and the target emotion relief content, and the response content is output, including: In the case that the AI response strategy includes text response, target response text is generated based on the feature fusion data and the emotion recognition result; In the case that the AI response strategy includes voice response, target response text is generated based on the feature fusion data and the emotion recognition result, and the target response text is converted into voice to obtain target response voice; In the case that the AI response strategy includes multimedia content response, target multimedia content is obtained based on the target emotion relief content, wherein the type of the target multimedia content includes at least one of music, video, and picture; In the case that the AI response strategy includes physical feedback response, target physical feedback is generated based on the emotion recognition result, wherein the target physical feedback includes at least one of vibration of a haptic vibration motor and mechanical action performed by a flexible mechanical component. 5.The AI interaction method based on emotion recognition according to claim 1, characterized in that, After the response content is generated based on the AI response strategy, the feature fusion data, the emotion recognition result, and the target emotion relief content, and the response content is output, the method further includes: Feedback data of the target user based on the response content is collected to obtain multi-modal feedback data; The model parameters of the emotion interaction engine are adaptively optimized using the feedback data. 6.The AI interaction method based on emotion recognition according to claim 1, characterized in that, Before the feature fusion data is obtained by performing feature fusion on the multi-modal data using the pre-trained emotion interaction engine, the method further includes: The initial model parameters of the emotion interaction engine are trained using meta-learning, wherein the initial model parameters are learned based on multiple learning tasks of a single user; The global model parameters of the emotion interaction engine are trained using federated learning to obtain the emotion interaction engine, wherein the global model parameters are aggregated based on the initial model parameters of multiple users.

7. An AI interaction device based on emotion recognition, characterized by, The device includes: An acquisition module for acquiring multi-modal data of a target user; A feature fusion module for performing feature fusion on the multi-modal data using a pre-trained emotion interaction engine to obtain feature fusion data; The analysis module is configured to perform sentiment recognition and context analysis on the feature fusion data respectively to obtain a sentiment recognition result and a comprehensive representation of a current state of the user, wherein the feature fusion data is used to represent time sequence features of the various modal data, the comprehensive representation of the current state of the user is obtained based on historical interaction data of the target user and the feature fusion data, and the comprehensive representation of the current state of the user is used to represent a sentiment category, a sentiment intensity, a physiological state and a behavior state of the target user. The determination module is configured to generate an AI response strategy corresponding to the target user based on the sentiment recognition result and the comprehensive representation of the current state of the user, and determine target emotion relief content from a pre-constructed emotion relief database, wherein the emotion relief database stores a plurality of emotion relief contents, and each emotion relief content corresponds to one attribute label. The output module is configured to generate response content based on the AI response strategy, the feature fusion data, the sentiment recognition result and the target emotion relief content, and output the response content. The determination module includes: The construction submodule is configured to construct a causal graph based on the sentiment recognition result and the comprehensive representation of the current state of the user, wherein the causal graph is used to represent a causal relationship between a user state and a response action, and the user state includes the sentiment recognition result and the comprehensive representation of the current state of the user. The first determination submodule is configured to determine the AI response strategy corresponding to the target user based on the causal graph and a preset reward function, wherein the AI response strategy is used to represent one or more response actions. The second determination submodule is configured to obtain historical interaction data of the target user, determine a historical preference vector of the target user based on the historical interaction data of the target user, and determine the target emotion relief content from the emotion relief database based on the historical preference vector of the target user and the comprehensive representation of the current state of the user.

8. An electronic device, comprising: The computer program is executed by the processor to implement the AI interaction method based on sentiment recognition as claimed in any one of claims 1-6. The computer program is executed by the processor to implement the AI interaction method based on sentiment recognition as claimed in any one of claims 1-6. ​ 9. A computer readable storage medium having stored thereon a computer program, characterized in that, ​

Images