Emotional state recognition method and device for multi-modal data, equipment and medium

Abstract

The invention relates to an emotional state recognition method and device for multi-modal data, equipment and a medium, and relates to the field of intelligent perception. According to the technical scheme, the emotion list is constructed; based on the emotion list, constructing a deep learning model of a fusion graph attention network and a long and short term memory network; training the deep learning model according to the training data set, and marking a training result through a loss function; on the basis of the weight matrix updating model, determining a corresponding optimal weight matrix when the loss function is at a minimum value; substituting the optimal weight matrix into a deep learning model, and determining an emotion recognition model; and recognizing the input multi-modal data through the emotion recognition model to obtain an emotion state recognition result. In this case, the utilization efficiency of the multi-modal data is improved, and the continuous capture of the emotional state and the real-time prediction of the change trend are ensured.

Description

Technical Field

[0001] The present invention relates to the field of intelligent perception technology, and in particular to a method, device, equipment and medium for multimodal data-oriented emotional state recognition. Background Art

[0002] Emotion recognition, a core supporting technology in fields such as human-computer interaction, mental health assessment, and intelligent customer service systems, aims to accurately perceive human emotional states through multi-dimensional modal information such as voice signals, facial expressions, body movements, and physiological indicators. In particular, in scenarios such as autonomous vehicle intelligent driving systems and mobile robot environmental adaptation, accurate and real-time recognition of the emotional state of surrounding people can provide key decision-making support for navigation and obstacle avoidance strategies and dynamic motion planning, significantly improving system safety and user-friendliness in complex human-computer interaction scenarios.

[0003] In recent years, with the rapid development of deep learning technology, emotion recognition methods based on multimodal data fusion have become a research focus in academia and industry. By integrating heterogeneous modal data from different sensors, these technologies aim to construct a more expressive emotion feature space. Recurrent neural networks (RNNs) and their variants have demonstrated unique advantages in modeling the temporal dynamics of speech emotion features. Graph neural networks (GNNs) and graph attention networks (GATs) have played an important role in analyzing the structured features of facial expressions and modeling the joint relationships of body movements, driving significant progress in multimodal emotion recognition technology.

[0004] However, current mainstream deep learning algorithms still face significant technical bottlenecks when handling multimodal emotion recognition tasks: First, feature fusion mechanisms for heterogeneous modal data such as speech, vision, and physiological signals are still imperfect. Existing methods often use simple splicing or weighted summation to integrate cross-modal features, failing to fully exploit the complementarity and correlation between different modalities. This leads to inefficient information utilization of multimodal data, which directly affects the efficiency and accuracy of emotion recognition. Second, key technologies are missing in capturing the continuous dynamics of emotional states. Existing models generally lack the ability to model the long-range dependencies of emotional state time series, making it difficult to achieve real-time prediction of emotional evolution trends. This makes it impossible to meet the application requirements of real-time, continuous, and accurate emotional state recognition in highly dynamic and highly interactive scenarios in complex urban traffic environments. These technical shortcomings make it difficult for existing solutions to provide effective emotional state support in practical applications such as dynamic path planning for unmanned vehicles and environmental behavior prediction for mobile robots. Innovative technical solutions are urgently needed to address these shortcomings. Summary of the Invention

[0005] The purpose of the present invention is to provide a method, device, equipment and medium for emotional state recognition based on multimodal data to solve the problems existing in the above-mentioned prior art.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention provides a method for identifying emotional states based on multimodal data, the method being applied to a computer device and comprising:

[0008] Constructing an emotion list; the emotion list includes at least like, dislike, fear, anger, joy, trust, optimism, pessimism, sadness, surprise, and expectation emotional states;

[0009] Based on the emotion list, a deep learning model integrating graph attention network and long short-term memory network is constructed;

[0010] Training the deep learning model based on the training data set and annotating the training results using a loss function;

[0011] Based on the weight matrix update model, determining the optimal weight matrix corresponding to the minimum value of the loss function;

[0012] Substituting the optimal weight matrix into the deep learning model to determine an emotion recognition model;

[0013] The input multimodal data is recognized by the emotion recognition model to obtain an emotion state recognition result.

[0014] In one possible implementation, constructing a deep learning model that integrates a graph attention network and a long short-term memory network based on the emotion list includes:

[0015] Based on the emotion list, construct a fusion graph neural network model; the fusion graph neural network model includes nodes, edges, and weights;

[0016] Assigning a learnable weight to the edges in the fusion graph neural network model through an attention mechanism;

[0017] Applying a σ activation function to the aggregated features to generate output features for each node in the fusion graph neural network model;

[0018] The output features of each node in the fusion graph neural network model are serialized according to time, and a long short-term memory network is constructed.

[0019] In one possible implementation, each node in the fusion graph neural network model represents an emotional state. When each node is assigned a feature vector, the feature vector can characterize the attributes of the emotional state represented by the corresponding node, and the feature vector is represented by a four-dimensional column vector.

[0020] The edges in the fusion graph neural network model represent the relationships between different emotional states, which are obtained by calculating the Euclidean distance between different emotional states. Emotional states with mutual relationships are connected by edges;

[0021] The weights in the fusion graph neural network model indicate the importance of each edge to the node.

[0022] In one possible implementation, assigning a learnable weight to the edges in the fusion graph neural network model through an attention mechanism includes:

[0023]

[0024] Where, is the attention coefficient of the kth attention mechanism between nodes i and j; a k is a learnable attention parameter; W k is the weight matrix of the k-th attention mechanism; h j is the eigenvector of node j; N(i) is the set of adjacent nodes of node i; LeakyReLU is the linear rectification function; T is the matrix transpose, which is a conventional operation on the matrix. Its core definition is to interchange the rows and columns of the matrix; h m It is the aggregation of the feature vectors of all nodes in the neighboring node set N(j) of node j.

[0025] In one possible implementation, applying a σ activation function to the aggregated features to generate output features for each node in the fusion graph neural network model includes:

[0026]

[0027] Where h' i is the output feature of node i; the value of K is 7; is the attention coefficient of the kth attention mechanism between nodes i and j; N(i) is the set of adjacent nodes of node i; W k is the weight matrix of the k-th attention mechanism; h j is the feature vector of node j.

[0028] In one possible implementation, serializing the output features of each node in the fusion graph neural network model according to time includes:

[0029] h t =tanh(W xh x t +W hh x t +b h );

[0030] Where h t is the hidden state at time step t; x t is the input at time step t; W xh is the weight input to the hidden layer; W hh is the weight from the historical hidden state to the current hidden state; b h is the bias term.

[0031] In one possible implementation, the step of constructing a long short-term memory network includes:

[0032] y t =W hy h t +b y ;

[0033] Where y t is the output feature; W hy is the weight from the hidden state to the output layer; h t is the hidden state at time step t; b y is the bias term.

[0034] In a possible implementation, the training data set is a public data set that integrates text, speech, and facial expressions;

[0035] The loss function is a cross entropy model, including:

[0036]

[0037] Where L is the loss function; y i is the correct input label; p i is the predicted probability of emotion classification i.

[0038] In a possible implementation, the weight matrix updating model includes:

[0039]

[0040] Where, ω + is the weight matrix after update; ω is the weight matrix before update; η is the learning speed; is the gradient of the weight matrix, which is the partial derivative of each vector of the weight matrix.

[0041] In a second aspect, the present invention provides an emotional state recognition device for multimodal data, the device being applied to the emotional state recognition method for multimodal data as described above, the device comprising:

[0042] A construction module for constructing an emotion list; the emotion list includes at least like, dislike, fear, anger, joy, trust, optimism, pessimism, sadness, surprise, and expectation;

[0043] The building module is further used to build a deep learning model that integrates a graph attention network and a long short-term memory network based on the emotion list;

[0044] A training and annotation module is used to train the deep learning model according to the training data set and annotate the training results using a loss function;

[0045] A determination module is used to update the model based on the weight matrix and determine the optimal weight matrix corresponding to the minimum value of the loss function;

[0046] The determination module is further configured to substitute the optimal weight matrix into the deep learning model to determine an emotion recognition model;

[0047] The recognition module is used to recognize the input multimodal data through the emotion recognition model to obtain an emotion state recognition result.

[0048] In a third aspect, the present invention provides a computer device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the processor can load and execute at least one instruction, at least one program, code set or instruction set to implement the emotional state recognition method for multimodal data provided above.

[0049] In a fourth aspect, the present invention provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The processor can load and execute at least one instruction, at least one program, code set or instruction set to implement the emotional state recognition method for multimodal data provided above.

[0050] In a fifth aspect, the present invention provides a computer program product or computer program, comprising computer program instructions stored in a computer-readable storage medium. A processor reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing a computer device to perform the method for multimodal data-based emotional state recognition as provided above.

[0051] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0052] This technical solution constructs an emotion list; based on the emotion list, builds a deep learning model that integrates a graph attention network and a long-short-term memory network; trains the deep learning model based on a training dataset and annotates the training results using a loss function; updates the model based on a weight matrix to determine the optimal weight matrix corresponding to the minimum loss function; substitutes the optimal weight matrix into the deep learning model to determine the emotion recognition model; and uses the emotion recognition model to identify the input multimodal data to obtain the emotion state recognition result. This improves the efficiency of multimodal data utilization, ensures the continuous capture of emotional states, and ensures the real-time prediction of changing trends. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0054] Figure 1 A flow chart of an emotional state recognition method for multimodal data provided by an exemplary embodiment of the present invention is shown.

[0055] Figure 2 A structural block diagram of an emotional state recognition device for multimodal data provided by an exemplary embodiment of the present invention is shown.

[0056] Figure 3 A schematic structural diagram of a computer device for executing an emotional state recognition method for multimodal data provided by an exemplary embodiment of the present invention is shown. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0059] Figure 1 A flowchart of a method for multimodal data-based emotional state recognition provided by an exemplary embodiment of the present invention is shown. The method for multimodal data-based emotional state recognition is applied to a computer device and includes:

[0060] Step 101: construct an emotion list; the emotion list at least includes like, dislike, fear, anger, joy, trust, optimism, pessimism, sadness, surprise, and expectation emotional states.

[0061] In the embodiment of the present application, the purpose of constructing an emotion list is to clearly define the target range of emotion recognition and provide a unified labeling system for subsequent model training and classification. By presetting standardized emotion categories, the consistency of training data annotation can be ensured, so that the model output results have clear semantic directionality. At the same time, each state in the emotion list corresponds to a node in the subsequent fusion graph neural network model, laying the foundation for building an emotion relationship network and helping the model learn the relationship between different emotions. This step is the logical starting point of the entire recognition method, and the emotion boundaries it defines directly affect the classification granularity and application scenario adaptability of the model.

[0062] Step 102: Based on the emotion list, a deep learning model integrating a graph attention network and a long short-term memory network is constructed.

[0063] In detail, the above-mentioned deep learning model for constructing a fusion graph attention network and a long short-term memory network based on the emotion list includes: constructing a fusion graph neural network model based on the emotion list; the fusion graph neural network model includes nodes, edges, and weights; assigning a learnable weight to the edges in the fusion graph neural network model through the attention mechanism; applying the σ activation function to the aggregated features to generate the output features of each node in the fusion graph neural network model; serializing the output features of each node in the fusion graph neural network model according to time, and constructing a long short-term memory network.

[0064] Furthermore, each node in the above-mentioned fusion graph neural network model represents an emotional state. When each node is assigned a feature vector, the feature vector can characterize the attributes of the emotional state represented by the corresponding node, and the feature vector is represented by a four-dimensional column vector; the edges in the fusion graph neural network model represent the relationship between different emotional states, which is obtained by calculating the Euclidean distance between different emotional states, and emotional states with mutual relationships are connected by edges; the weights in the fusion graph neural network model indicate the importance of each edge to the node. In this case, the fusion graph neural network achieves structured representation and dynamic association learning of emotional states through three-dimensional modeling of nodes, edges, and weights, abstracting each emotion into a four-dimensional feature vector (such as dimensions such as valence / positivity and arousal / arousal), making emotions quantifiable and supporting mathematical operations; constructing edge connections by calculating the Euclidean distance between emotions (such as the close distance between "anger" and "disgust"), automatically capturing implicit relationships; introducing an attention weight mechanism to dynamically adjust the importance of edges (such as the higher transmission weight of "surprise" over "fear" in a specific context); this design transforms emotion recognition into a feature propagation process on a graph structure, providing LSTM with a spatiotemporal feature sequence, which is suitable for analyzing the evolution path of emotions in multimodal data (such as the synergistic relationship between speech and expression), and improving recognition accuracy and interpretability.

[0065] In an embodiment of the present application, a deep learning model that integrates a graph attention network (GAT) and a long short-term memory network (LSTM) is constructed, the core function of which is to provide a core architecture for multimodal emotion recognition that combines structural association modeling and time series analysis. First, the graph neural network model constructed based on the emotion list defines each emotion as a node, quantifies the relationship between emotions with Euclidean distance and connects them through edges to form a spatial association structure of emotional states; the edges are given learnable weights through the attention mechanism, so that the model dynamically focuses on key emotional interactions and enhances the pertinence of feature aggregation. Secondly, the σ activation function is applied to the aggregated features, and node output features are generated through nonlinear transformation to enhance the representation ability of emotional states. Finally, the node features are serialized in time and input into the LSTM, and its memory unit is used to capture the continuous change pattern of emotional states over time, effectively processing the implicit temporal dependencies in multimodal data (such as the dynamic evolution of voice intonation and facial expressions). This fusion architecture not only models the static associations between emotions through the graph network, but also captures dynamic temporal features with the help of LSTM, providing an efficient framework for the spatiotemporal feature fusion of multimodal data, and is particularly suitable for scenarios that require real-time response to emotional changes.

[0066] Furthermore, the attention mechanism is used to assign a learnable weight to the edges in the fusion graph neural network model, including:

[0067]

[0068] Where, is the attention coefficient of the kth attention mechanism between nodes i and j; a k is a learnable attention parameter; W k is the weight matrix of the kth attention mechanism; h j is the eigenvector of node j; N(i) is the set of adjacent nodes of node i; LeakyReLU is the linear rectification function; T is the matrix transpose, which is a conventional operation on the matrix. Its core definition is to interchange the rows and columns of the matrix; h m It is the aggregation of the feature vectors of all nodes in the neighboring node set N(j) of node j.

[0069] In the embodiment of the present application, the attention coefficient between nodes is calculated by the attention mechanism Dynamically determining edge weights enables the model to adaptively learn the importance of connections between emotional nodes. The numerator utilizes the LeakyReLU activation function (to prevent vanishing gradients on negative inputs) to process node features, highlighting key connections through exponential operations. The denominator normalizes adjacent nodes to ensure a balanced distribution of weights. This allows the model to focus on important emotional relationships (such as the strong correlation between "anger" and "disgust"), optimize feature aggregation, and enhance the fusion graph neural network's ability to model complex interactions between emotional states.

[0070] Specifically, the σ activation function is applied to the aggregated features to generate the output features of each node in the fusion graph neural network model, including:

[0071]

[0072] Where h' i is the output feature of node i; the value of K is 7; is the attention coefficient of the kth attention mechanism between nodes i and j; N(i) is the set of adjacent nodes of node i; W k is the weight matrix of the k-th attention mechanism; h j is the feature vector of node j.

[0073] In the embodiment of the present application, the node output feature h' is generated by applying the σ activation function (Sigmoid) to the aggregated feature i Its core function is to introduce nonlinear transformation and map the fused features to the (0, 1) interval, so that the output has a probability-like interpretability and can represent the intensity or possibility of the emotional state. By weighted summing the results of 7 attention mechanisms (K=7), multi-dimensional information is integrated to enhance feature richness. With the help of attention coefficient The feature h of the adjacent node N(i) j Weighted, combined with the weight matrix W k , making full use of the associations between emotion nodes in the graph structure so that the output features of each node can effectively capture the emotional interaction relationship.

[0074] More specifically, the output features of each node in the fusion graph neural network model are serialized according to time, including:

[0075] h t =tanh(W xh x t +W hh x t +b h );

[0076] Where h t is the hidden state at time step t; x t is the input at time step t; Wxh is the weight input to the hidden layer; W hh is the weight from the historical hidden state to the current hidden state; b h is the bias term.

[0077] In the embodiment of the present application, the output features of the fusion graph neural network are serialized in time, and a time series model is constructed to capture the dynamic changes of emotions. The input x is processed by the tanh activation function. t , using W xh 、W hh Learning input and historical information on the current hidden state h t The impact of b h Adjusting model offsets. This process enables the model to understand how emotions evolve over time. For example, in speech or video, it captures how multimodal information like tone and expression change over time, effectively integrating spatiotemporal features.

[0078] It is worth mentioning that building a long short-term memory network includes:

[0079] y t =W hy h t +b y ;

[0080] Where y t is the output feature; W hy is the weight from the hidden state to the output layer; h t is the hidden state at time step t; b y The LSTM network can accurately identify continuous changes in emotional states, playing an important role in supporting unmanned vehicles and mobile robots that need to respond differently based on a person's real-time emotional state.

[0081] In the embodiment of the present application, W hy For the hidden state h t Weighted, combined with b y Adjustment, converting the temporal emotional information captured by LSTM into output features y t This process effectively extracts key features of continuously changing emotions, enabling the model to accurately identify emotional evolution. For autonomous vehicles and mobile robots, real-time emotion recognition is fundamental to interactive decision-making, such as adjusting responses to angry passengers.

[0082] Step 103: Train the deep learning model based on the training data set and annotate the training results using a loss function.

[0083] Specifically, the above training dataset is a public dataset that integrates text, speech, and facial expressions.

[0084] In the examples of this application, a public dataset with a large number of labeled samples ensures the breadth and depth of training, covering a wide range of emotional scenarios. A loss function quantifies the training results, measuring the difference between the predicted and true emotion labels. This provides clear guidance for model optimization, prompting the model to continuously adjust its parameters and reduce errors.

[0085] Furthermore, the above loss function is a cross entropy model, including:

[0086]

[0087] Where L is the loss function; y i is the correct input label; p i is the predicted probability of emotion classification i.

[0088] In the embodiment of the present application, y i represents the real emotion label (such as "anger"), p i is the probability of the model predicting the emotion. When the prediction deviates greatly from the true value, the loss function value L increases, and vice versa. By minimizing L, the model adjusts the parameters to make the predicted probability p i Approaching the true label y i In multimodal emotion recognition, it effectively measures the deviation between the comprehensive prediction after integrating text, speech, and expressions and the actual emotion, guiding the model to learn more accurate feature representations, strengthening the ability to distinguish emotion categories, ensuring that the output is consistent with the actual emotional state, and improving the classification accuracy and reliability under complex multimodal data.

[0089] Step 104 : updating the model based on the weight matrix to determine the optimal weight matrix corresponding to the minimum loss function.

[0090] Specifically, the weight matrix update model includes:

[0091]

[0092] Where, ω + is the weight matrix after update; ω is the weight matrix before update; η is the learning speed; is the gradient of the weight matrix, which is the partial derivative of each vector of the weight matrix.

[0093] In the embodiment of the present application, the weight update step size is controlled by the learning speed η, and the gradient This guides the update direction and continuously iterates the weight matrix ω to reduce the loss function value. When the loss function is at its minimum, the corresponding optimal weight matrix enables the model to achieve optimal performance in multimodal emotion recognition, accurately capturing the mapping relationship between fused features such as text, speech, and expressions and emotion labels, and reducing prediction bias.

[0094] Step 105: Substitute the optimal weight matrix into the deep learning model to determine the emotion recognition model.

[0095] In the embodiments of the present application, the optimal weight matrix is a set of parameters that minimizes the loss function after training and optimization, representing the model's best fit for the mapping relationship between multimodal data (text, speech, facial expressions) and emotion labels. After substitution, the model parameters are fixed, forming an emotion recognition model with stable predictive capabilities. This ensures that in practical applications, the model can accurately process newly input multimodal data based on previously learned features and patterns, achieving accurate discrimination of emotional states.

[0096] Step 106: Identify the input multimodal data using an emotion recognition model to obtain an emotion state recognition result.

[0097] In the examples of this application, the model accurately identifies the emotional states (such as like, anger, sadness, etc.) contained in the data based on the feature representation and emotion mapping relationship learned in early training. This step realizes the leap from model training to practical application, enabling intelligent systems (such as unmanned vehicles and mobile robots) to obtain real-time human emotional feedback, adjust interactive behaviors accordingly, and improve the naturalness and effectiveness of human-computer interaction.

[0098] Figure 2 A structural block diagram of an emotional state recognition device for multimodal data provided by an exemplary embodiment of the present invention is shown. The emotional state recognition device for multimodal data is applied to the emotional state recognition method for multimodal data described above, and the device includes:

[0099] A construction module 201 is used to construct an emotion list; the emotion list includes at least like, dislike, fear, anger, joy, trust, optimism, pessimism, sadness, surprise, and expectation;

[0100] The construction module 201 is further used to construct a deep learning model that integrates the graph attention network and the long short-term memory network based on the emotion list;

[0101] A training and annotation module 202 is used to train the deep learning model based on the training data set and annotate the training results using a loss function;

[0102] A determination module 203 is configured to update the model based on the weight matrix and determine the optimal weight matrix corresponding to the minimum value of the loss function;

[0103] The determination module 203 is further configured to substitute the optimal weight matrix into the deep learning model to determine the emotion recognition model;

[0104] The recognition module 204 is configured to recognize the input multimodal data using an emotion recognition model to obtain an emotion state recognition result.

[0105] In one possible implementation, based on the emotion list, a deep learning model integrating graph attention network and long short-term memory network is constructed, including:

[0106] Based on the emotion list, a fusion graph neural network model is constructed; the fusion graph neural network model includes nodes, edges, and weights;

[0107] Assign a learnable weight to the edges in the fusion graph neural network model through the attention mechanism;

[0108] Apply the σ activation function to the aggregated features to generate the output features of each node in the fusion graph neural network model;

[0109] The output features of each node in the fusion graph neural network model are serialized according to time, and a long short-term memory network is constructed.

[0110] In one possible implementation, each node in the fusion graph neural network model represents an emotional state. When each node is assigned a feature vector, the feature vector can characterize the attributes of the emotional state represented by the corresponding node. The feature vector is represented by a four-dimensional column vector.

[0111] The edges in the fusion graph neural network model represent the relationships between different emotional states, which are obtained by calculating the Euclidean distance between different emotional states. Emotional states with mutual relationships are connected by edges;

[0112] The weights in the fusion graph neural network model indicate the importance of each edge to the node.

[0113] In one possible implementation, an attention mechanism is used to assign a learnable weight to the edges in the fusion graph neural network model, including:

[0114]

[0115] Where, is the attention coefficient of the kth attention mechanism between nodes i and j; a k is a learnable attention parameter; W k is the weight matrix of the k-th attention mechanism; h j is the eigenvector of node j; N(i) is the set of adjacent nodes of node i; LeakyReLU is the linear rectification function; T is the matrix transpose, which is a conventional operation on the matrix. Its core definition is to interchange the rows and columns of the matrix; h mIt is the aggregation of the feature vectors of all nodes in the neighboring node set N(j) of node j.

[0116] In one possible implementation, a σ activation function is applied to the aggregated features to generate output features for each node in the fusion graph neural network model, including:

[0117]

[0118] Where h' i is the output feature of node i; the value of K is 7; is the attention coefficient of the kth attention mechanism between nodes i and j; N(i) is the set of adjacent nodes of node i; W k is the weight matrix of the kth attention mechanism; h j is the feature vector of node j.

[0119] In one possible implementation, the output features of each node in the fusion graph neural network model are serialized according to time, including:

[0120] h t =tanh(W xh x t +W hh x t +b h );

[0121] Where h t is the hidden state at time step t; x t is the input at time step t; W xh is the weight input to the hidden layer; W hh is the weight from the historical hidden state to the current hidden state; b h is the bias term.

[0122] In one possible implementation, building a long short-term memory network includes:

[0123] y t =W hy h t +b y ;

[0124] Where y t is the output feature; W hy is the weight from the hidden state to the output layer; h t is the hidden state at time step t; b y is the bias term.

[0125] In one possible implementation, the training dataset is a public dataset that integrates text, speech, and facial expressions;

[0126] The loss function is a cross entropy model, including:

[0127]

[0128] Where L is the loss function; y i is the correct input label; p i is the predicted probability of emotion classification i.

[0129] In one possible implementation, the weight matrix update model includes:

[0130]

[0131] Where, ω + is the weight matrix after update; ω is the weight matrix before update; η is the learning speed; is the gradient of the weight matrix, which is the partial derivative of each vector of the weight matrix.

[0132] It should be noted that the emotional state recognition device for multimodal data provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0133] Figure 3 A schematic diagram of the structure of a computer device for executing a method for multimodal data-oriented emotional state recognition provided by an exemplary embodiment of the present invention is shown. The computer device includes:

[0134] The processor 301 includes one or more processing cores. The processor 301 executes various functional applications and data processing by running software programs and modules.

[0135] Receiver 302 and transmitter 303 can be implemented as a communication component, which can be a communication chip. Optionally, the communication component can include signal transmission functionality. That is, transmitter 303 can be used to transmit control signals to the image acquisition device and scanning device, and receiver 302 can be used to receive corresponding feedback instructions.

[0136] The memory 304 is connected to the processor 301 via a bus 305 .

[0137] The memory 304 may be used to store at least one instruction, and the processor 301 may be used to execute the at least one instruction to implement each step in the above method embodiment.

[0138] An embodiment of the present invention also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, which is loaded and executed by a processor to implement the above-mentioned emotional state recognition method for multimodal data.

[0139] The present invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for multimodal data-based emotional state recognition described in any of the above embodiments.

[0140] Optionally, the computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disk. Among them, the random access memory may include a resistive random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0141] It should be understood that the specific examples herein are only intended to help those skilled in the art to better understand the present disclosure, rather than to limit the scope of the present invention.

[0142] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of this disclosure.

[0143] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the present disclosure is not limited thereto.

[0144] Unless otherwise indicated, all technical and scientific terms used in this disclosure have the same meaning as commonly understood by those skilled in the art in the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms "a", "above", and "the" used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0145] It is understood that the processor of the present disclosure can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-mentioned method implementation can be completed by hardware integrated logic circuits in the processor or software instructions. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in this disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this disclosure can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above-mentioned method in combination with its hardware.

[0146] It will be understood that the memory in the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be read-only memory (ROM), programmable read-only memory (programmable ROM, PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory may be random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0147] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this specification.

[0148] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices and units can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0149] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0150] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of this embodiment.

[0151] In addition, each functional unit in each embodiment of this specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0152] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art, or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0153] The above description is merely a specific embodiment of this specification, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this specification should be included in the scope of protection of this specification. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for emotional state recognition based on multimodal data, characterized in that: The method is applied to a computer device, and the method includes: Constructing an emotion list; the emotion list includes at least like, dislike, fear, anger, joy, trust, optimism, pessimism, sadness, surprise, and expectation emotional states; Based on the emotion list, a deep learning model integrating graph attention network and long short-term memory network is constructed; Training the deep learning model based on the training data set and annotating the training results using a loss function; Based on the weight matrix update model, determining the optimal weight matrix corresponding to the minimum value of the loss function; Substituting the optimal weight matrix into the deep learning model to determine an emotion recognition model; The input multimodal data is recognized by the emotion recognition model to obtain an emotion state recognition result.

2. The method for emotional state recognition based on multimodal data according to claim 1, characterized in that: The method of constructing a deep learning model based on the emotion list that integrates a graph attention network and a long short-term memory network includes: Based on the emotion list, construct a fusion graph neural network model; the fusion graph neural network model includes nodes, edges, and weights; Assigning a learnable weight to the edges in the fusion graph neural network model through an attention mechanism; Applying a σ activation function to the aggregated features to generate output features for each node in the fusion graph neural network model; The output features of each node in the fusion graph neural network model are serialized according to time, and a long short-term memory network is constructed.

3. The method for multimodal data-based emotional state recognition according to claim 2, wherein: Each node in the fusion graph neural network model represents an emotional state. When each node is assigned a feature vector, the feature vector can characterize the attributes of the emotional state represented by the corresponding node. The feature vector is represented by a four-dimensional column vector. The edges in the fusion graph neural network model represent the relationships between different emotional states, which are obtained by calculating the Euclidean distance between different emotional states. Emotional states with mutual relationships are connected by edges; The weights in the fusion graph neural network model indicate the importance of each edge to the node.

4. The method for multimodal data-based emotional state recognition according to claim 2, wherein: The attention mechanism is used to assign a learnable weight to the edges in the fusion graph neural network model, including: Where, is the attention coefficient of the kth attention mechanism between nodes i and j; a k is a learnable attention parameter; W k is the weight matrix of the k-th attention mechanism; h j is the eigenvector of node j; N(i) is the set of adjacent nodes of node i; LeakyReLU is the linear rectification function; T is the matrix transpose, which is a conventional operation on the matrix. Its core definition is to interchange the rows and columns of the matrix; h m It is the aggregation of the feature vectors of all nodes in the neighboring node set N(j) of node j.

5. The method for emotional state recognition based on multimodal data according to claim 2, characterized in that: Applying a σ activation function to the aggregated features to generate output features for each node in the fusion graph neural network model includes: Where h i ' is the output feature of node i; the value of K is 7; is the attention coefficient of the kth attention mechanism between nodes i and j; N(i) is the set of adjacent nodes of node i; W k is the weight matrix of the k-th attention mechanism; h j is the feature vector of node j.

6. The method for multimodal data-oriented emotional state recognition according to claim 2, characterized in that: The step of serializing the output features of each node in the fusion graph neural network model according to time includes: h t =tanh(W xh x t +W hh x t +b h ); Where h t is the hidden state at time step t; x t is the input at time step t; W xh is the weight input to the hidden layer; W hh is the weight from the historical hidden state to the current hidden state; b h is the bias term.

7. The method for multimodal data-oriented emotional state recognition according to claim 2, characterized in that: The constructing of the long short-term memory network includes: y t =W hy h t +b y ; Where y t is the output feature; W hy is the weight from the hidden state to the output layer; h t is the hidden state at time step t; b y is the bias term.

8. The method for emotional state recognition based on multimodal data according to claim 1, characterized in that: The training data set is a public data set that integrates text, speech, and facial expressions; The loss function is a cross entropy model, including: Where L is the loss function; y i is the correct input label; p i is the predicted probability of emotion classification i.

9. The method for emotional state recognition based on multimodal data according to claim 1, characterized in that: The weight matrix updating model includes: Where, ω + is the weight matrix after update; ω is the weight matrix before update; η is the learning speed; is the gradient of the weight matrix, which is the partial derivative of each vector of the weight matrix.

10. An emotional state recognition device for multimodal data, characterized in that: The device is applied to the method for multimodal data-oriented emotional state recognition according to any one of claims 1 to 9, and the device comprises: A construction module for constructing an emotion list; the emotion list includes at least like, dislike, fear, anger, joy, trust, optimism, pessimism, sadness, surprise, and expectation; The building module is further used to build a deep learning model that integrates a graph attention network and a long short-term memory network based on the emotion list; A training and annotation module is used to train the deep learning model according to the training data set and annotate the training results using a loss function; A determination module is used to update the model based on the weight matrix and determine the optimal weight matrix corresponding to the minimum value of the loss function; The determination module is further configured to substitute the optimal weight matrix into the deep learning model to determine an emotion recognition model; The recognition module is used to recognize the input multimodal data through the emotion recognition model to obtain an emotion state recognition result.

Images