A micro-expression recognition method based on multimodal fusion

Abstract

The present application discloses a micro-expression recognition method based on multimodal fusion, which relates to the field of micro-expression recognition technology, including: collecting multimodal data, the multimodal data includes user data and environmental data, the user data includes micro-expressions and physiological parameters; the environmental data includes light and temperature and humidity, and micro-expressions refer to fine facial expression features; extracting representation vectors of micro-expressions, the representation vectors represent the semantic features of micro-expression images or video sequences; constructing a micro-expression knowledge graph according to the correspondence between the representation vectors of micro-expressions and different emotion categories; using an entity alignment algorithm, in the same feature space, using semantic constraints to align the micro-expression knowledge graph and physiological parameters to construct a multimodal knowledge graph; based on the multimodal knowledge graph and environmental data, constructing a micro-expression recognition model, and outputting the micro-expression recognition results. In view of the problem of low micro-expression recognition accuracy in the prior art, the present application improves the accuracy of micro-expression recognition.

Description

Technical Field

[0001] The present application relates to the technical field of micro-expression recognition, and in particular to a micro-expression recognition method based on multimodal fusion. Background Art

[0002] Micro-expression recognition is an important research topic in facial expression analysis. Micro-expressions are subtle expressions that appear for a short time and can reflect people's true inner emotional state. Compared with obvious macro-expressions, micro-expressions are more difficult to detect, so how to improve the accuracy of micro-expression recognition is a key technical problem.

[0003] Traditional micro-expression recognition methods are mainly based on manually designed feature extraction and classification models, and the recognition accuracy is low. With the development of deep learning, various convolutional neural networks and recurrent neural networks have been introduced into micro-expression recognition tasks, improving the ability of feature learning. However, these methods are too dependent on single visual data and do not consider the intrinsic correlation between micro-expressions and other physiological signals, so there is still much room for improvement in recognition accuracy.

[0004] In the related art, for example, a multimodal emotion recognition method and device is provided in Chinese patent document CN116955601A. The method includes: determining the target user's micro-expression knowledge graph, micro-action knowledge graph and text emotion analysis vector; fusing the micro-expression knowledge graph with the micro-action knowledge graph, and determining a preset emotion vector according to the fusion result; determining the emotion analysis result according to the preset emotion vector and the text emotion analysis vector; determining the emotion of the target user according to the emotion analysis result. The multimodal emotion recognition method provided in the embodiment of the present application constructs a knowledge graph based on multiple modal entities such as vision, speech and text. However, this scheme mainly relies on feature extraction. If the feature extraction algorithm cannot effectively capture the key features of micro-expressions, it will affect the accuracy of subsequent emotion recognition. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In response to the problem of low accuracy in micro-expression recognition in the prior art, the present application provides a micro-expression recognition method based on multimodal fusion, which improves the accuracy of micro-expression recognition by collecting multimodal data, constructing a micro-expression knowledge graph, and using an entity alignment algorithm to achieve feature alignment of micro-expressions and physiological parameters.

[0007] 2. Technical solution

[0008] The purpose of this application is achieved through the following technical solutions.

[0009] The embodiments of this specification provide a micro-expression recognition method based on multimodal fusion, including: collecting multimodal data, the multimodal data includes user data and environmental data, the user data includes micro-expressions and physiological parameters; the environmental data includes light and temperature and humidity, and micro-expressions refer to fine facial expression features; extracting representation vectors of micro-expressions, the representation vectors represent the semantic features of micro-expression images or video sequences; constructing a micro-expression knowledge graph according to the correspondence between the representation vectors of micro-expressions and different emotion categories; using an entity alignment algorithm, in the same feature space, using semantic constraints to align the micro-expression knowledge graph and physiological parameters to construct a multimodal knowledge graph; based on the multimodal knowledge graph and environmental data, constructing a micro-expression recognition model, and outputting an emotion analysis vector representing the micro-expression recognition result.

[0010] Specifically, the acquisition of micro-expression image sequences: Use a high-speed camera to capture facial area videos, ensuring that the shooting frame rate is not less than 200fps to capture rapid micro-expression changes. Use image preprocessing technology to improve the quality of the acquired images. Physiological parameter acquisition: Use wearable devices to collect user physiological data, including heart rate, blood pressure, skin galvanic response and other information. The sampling frequency is not less than 10Hz. Light intensity acquisition: Use ambient light sensors to collect light intensity information within the visible spectrum, with a sampling frequency of 10Hz. Temperature and humidity acquisition: Use temperature and humidity sensors to collect temperature and humidity information of the user's environment, with a sampling frequency of 1Hz. Cut and segment micro-expression image sequences to extract facial areas. Eliminate noise interference in physiological signals. Label micro-expression video images and mark the timestamps of light, temperature and humidity. Unify data with different sampling rates onto the same timeline. Use NoSQL databases to store the collected multimodal time series data. Use dense vector representation to store micro-expression and physiological sequence features.

[0011] Specifically, the collected micro-expression image sequence is preprocessed by cropping, scaling, etc. and input into the feature extraction module. The spatial features of the image are extracted frame by frame using a convolutional neural network (such as ResNet) to obtain a feature map that describes the local texture of the image. The image sequence and its spatial feature map are input into a recurrent neural network (such as LSTM) to learn the temporal variation features and output a hidden state vector representing the temporal correlation. The hidden state vector is input into the attention module to learn the importance of different temporal sequences for micro-expression recognition and output the weight score of the time step. The spatial feature map, the temporal hidden state vector and the temporal weight are multiplied and summed to obtain a micro-expression representation vector that integrates spatial and temporal information. The micro-expression representation vector is fine-tuned by the fully connected layer to strengthen the feature learning of the micro-expression key frames. Each dimension in the final output representation vector represents a facial micro-expression action, reflecting the semantic information of the micro-expression image sequence.

[0012] Specifically, create nodes representing three categories of positive emotions, negative emotions, and neutral emotions, and use word vectors to represent the semantic information of the nodes. The micro-expression representation vector and the vectorized representation of the emotion category node are used as input and input into the LSTM network based on the attention mechanism. The LSTM network learns the semantic association between the representation vector and the nodes of different emotion categories. The attention mechanism outputs the connection weight between the representation vector and each category node. A knowledge graph is constructed based on the representation vector, emotion category nodes, and connection weights. The nodes represent the emotion categories, and the edges represent the correspondence between micro-expressions and emotions. The knowledge graph representation is regularized, and the model's learning ability for implicit features is enhanced through negative sampling. The knowledge graph finally obtained can represent the emotional connotation reflected by the micro-expression image or video sequence.

[0013] Among them, the entity alignment algorithm, a representation learning algorithm that can model entities and their relationships, realizes the alignment of different entities by mapping to a unified vector space. A knowledge graph translation model (such as TransR) is used to realize the alignment of micro-expression knowledge graphs and physiological parameters in the same feature space. Semantic constraints refer to the use of semantic information of entities and relationships, adding prior knowledge to the representation learning process, and constraining the model fitting results. According to the semantic consistency of the two modalities of micro-expressions and physiological parameters, constraints are added to optimize the fusion process of the two modal features, so that the learned multimodal knowledge graph is more semantically coherent. In this application, two types of feature vectors are extracted; mapped to a shared semantic space; semantic constraints are added in the shared space; the objective function is optimized to achieve feature alignment; and the aligned multimodal knowledge graph is obtained.

[0014] Among them, the micro-expression recognition model refers to a machine learning model specifically used to recognize facial micro-expressions. Its input is multimodal data containing a sequence of facial images, and its output is a sentiment analysis vector reflecting the micro-expression recognition results. In this application, the micro-expression feature extractor: uses a convolutional neural network to extract features from facial images to obtain a spatial and temporal representation of facial movements. Multimodal feature fusion: fuses the extracted micro-expression features with other modal features such as speech and text on a knowledge graph. Model output layer: based on the fused multimodal features, outputs a sentiment analysis vector representing the final recognition result. The knowledge graph provides structured information and enhances feature expression. By constructing and training such a micro-expression recognition model, facial micro-expressions can be analyzed in a complex environment, and a sentiment analysis vector reflecting the recognition results can be output to provide support for subsequent emotional judgments.

[0015] Specifically, the aligned multimodal knowledge graph, micro-expression sequence, physiological data, and environmental data (light, temperature and humidity) are fused at the feature level. Using a multi-layer perceptron or GRU network, the fused multimodal features are used as input to learn complex nonlinear relationships. Dropout technology is used to prevent overfitting. The model output represents the probability distribution of three emotion categories (positive, negative, and neutral), namely the sentiment analysis vector. The category with the highest probability in the sentiment analysis vector is used as the recognition result of the input micro-expression sequence. Using a multimodal micro-expression dataset, the model's recognition accuracy, recall rate, F1 value and other indicators are evaluated. The model structure and hyperparameters are adjusted according to the evaluation results to continuously optimize the model's micro-expression recognition performance.

[0016] Furthermore, a representation vector of micro-expressions is extracted, where the representation vector represents the semantic features of a micro-expression image or video sequence, including: taking the collected micro-expression image sequence as input, and using a convolutional neural network to obtain a feature map A representing the spatial features of the image sequence; taking the micro-expression image sequence and the feature map A as input, and using a recurrent neural network to obtain a latent state vector B representing the temporal features; taking the latent state vector B as input, and using a neural network with an attention mechanism to calculate weight scores C at different time steps; combining the obtained feature map A, latent state vector B, and weight score C to obtain a representation vector D of micro-expressions representing the semantic information of the image sequence, wherein each dimension in the representation vector D corresponds to a different facial muscle movement.

[0017] Among them, feature mapping refers to the feature representation of the input image extracted by a convolutional neural network, which retains the spatial information of the input image, and each pixel in each feature map represents the feature response value at a different position. In this application, the collected micro-expression image sequence is used as input and input into the convolutional neural network. The network gradually extracts the spatial features of the image through operations such as convolution and pooling, and outputs multiple two-dimensional feature maps. Each feature map represents the feature response of the input image sequence under a specific filter. This set of feature maps as a whole reflects the spatial information of the micro-expression image sequence, such as local texture and edges. Compared with directly using image pixels, feature mapping can learn high-level semantic features and retain spatial correlation, providing a spatial feature basis for the subsequent fusion of temporal information.

[0018] Among them, the hidden state vector is a vector representation used to describe the temporal information of the sequence in the recurrent neural network, which reflects the correlation between the input at the current moment and the input at the previous and next moments. In this application, the micro-expression image sequence and its spatial feature map are used as input and input into a recurrent neural network (such as LSTM), and the network will learn the temporal correlation features between different moments in the sequence. The hidden state vector of the hidden layer of the network reflects the trend information of the spatial feature map changing on the time axis, describes the process of the micro-expression action evolving from one state to another, and represents the temporal characteristics of the micro-expression. The hidden state vector integrates the temporal information of the sequence, so that the model can perceive the temporal correlation of micro-expression actions, rather than relying solely on static spatial information, which is very important for expressing subtle changes in micro-expressions. The hidden state vector is used in this technical solution to represent the temporal characteristics of the micro-expression sequence and provide timing information for constructing the representation vector.

[0019] Among them, the weight score is a set of numbers output by the attention mechanism, which indicates the importance of different parts of the input to the model output, and is usually used to weight the input. In this application, the hidden state vector B is used as input and input into the neural network of the attention mechanism. The attention mechanism analyzes the features of each moment in the hidden state vector and calculates the importance of each moment to the final micro-expression recognition, that is, the weight score C. The weight score can treat different moments of the micro-expression sequence differently, so that the model pays more attention to the most critical micro-expression representation frames. This is more conducive to extracting decisive information than simply averaging the features at different times. The weight score is used in this technical solution to indicate the importance of different time steps to micro-expression recognition, and to provide Sidenote for constructing semantic representation vectors.

[0020] Among them, the representation vector is a set of numbers used to represent the semantic features in the input data, and each dimension represents an attribute or feature. In this application, the extracted feature map A, the hidden state vector B and the weight score C are combined, and the weighted sum is calculated to obtain the representation vector D of the micro-expression image sequence. The representation vector D combines spatial features, temporal features and attention weights, and can represent the semantic information contained in the micro-expression sequence, that is, different facial micro-expression movements. Compared with individual features, the representation vector integrates the features of different modalities and is a semantic feature expression of information convergence. At the same time, the weighted effect strengthens the key frames, which is conducive to expressing key action features. Each dimension of the representation vector D corresponds to a facial micro-expression movement using an orthogonal transformation, such as raising eyebrows, drooping corners of the mouth, etc., thus reflecting the semantic information of the entire micro-expression sequence.

[0021] Furthermore, the feature map A represents the texture features of the micro-expression image sequence; the hidden state vector B represents the change trend between the feature maps A of adjacent time steps in the micro-expression image sequence, reflecting the change process of the texture features of micro-expressions between adjacent time series; the weight score C is arranged in descending order according to the importance contribution of different time steps to micro-expression recognition, and the importance contribution is calculated by the neural network of the attention mechanism based on the input hidden state vector B; the representation vector D uses L2 regularization to prevent overfitting.

[0022] Among them, texture features are visual features that represent the texture structure and texture of local areas of images. They describe the texture characteristics of images such as roughness and smoothness by extracting pixel get attributes, variances, edges and other information. In this application, feature map A is obtained by processing a sequence of micro-expression images using a convolutional neural network, which retains the spatial information of the original image. Each pixel in feature map A reflects the texture features of that position, such as wrinkles and small deformations in eyebrows, mouth and other parts that can reflect micro-expression movements, so feature map A mainly represents the texture features of the micro-expression sequence. Texture features can reflect tiny texture changes caused by changes in facial muscles, and are very important visual clues for analyzing micro-expressions. Therefore, this technical solution uses feature map A to represent this information and provide spatial feature support for the representation vector.

[0023] Specifically, the feature map A is input to the recurrent neural network, and the feature map of each time series in A reflects the micro-expression texture features at that moment. The hidden state vector B of the hidden layer of the recurrent network will integrate the feature map A of the current moment and the previous and next moments to learn the change relationship between adjacent time series. Since the feature map A reflects the texture features of the micro-expression image, the hidden state vector B will reflect how the micro-expression facial texture changes between two adjacent moments, that is, the change trend of the texture features on the time axis. For example, the micro-expression gradually changes from the neutral state to the eye brow raising, and the hidden state vector B will capture this texture deformation process. Compared with the single static feature map A, the temporal change features provided by the hidden state vector B can better represent the subtle movement process of the micro-expression. Finally, the hidden state vector B and the feature map A are fused together to form the representation vector D, providing static and dynamic texture features.

[0024] Specifically, the hidden state vector B is input into the neural network of the attention mechanism. The attention mechanism analyzes the hidden state vector at each moment and evaluates the importance of different moments to the final micro-expression recognition. Specifically, the correlation between each hidden state vector and the final output is calculated as the importance score, and the correlation is calculated using an inner product or a weighted function. The calculated importance score is the weight score C, which indicates the importance of different time steps to the output result. Then, the different moments are sorted in descending order according to the value of the weight score C. The moments at the front of the order have higher weight scores, which means that their corresponding features are more important for micro-expression recognition. Such weighting on the time axis can make the model pay more attention to the moments that have a great impact on the results, so as to achieve the purpose of extracting decisive information. The weight score C provides temporal differentiation for feature maps and hidden state vectors, so that key frames are strengthened.

[0025] Among them, L2 regularization is a regularization technique that penalizes model complexity by introducing the L2 norm of the weight parameter in the model loss function, so that the model remains in a lower complexity space, thereby preventing overfitting. In this application, the micro-expression representation vector D will be input as a feature into the subsequent classification model. In order to prevent the classification model from being too complex and overfitting the training data, the representation vector D is L2 regularized. The L2 norm of the representation vector D is added as a regularization term in the loss function of the representation vector to limit the representation vector parameters from being too large. The regularization term coefficient controls the regularization strength. L2 regularization keeps the model simple by limiting the feature parameters, avoids learning the noise and randomness of specific training data, and enhances the generalization ability of the model.

[0026] Furthermore, the convolutional neural network uses the residual network ResNet; the recurrent neural network uses the long short-term memory network LSTM; and the neural network of the attention mechanism uses the spatial attention mechanism. Among them, the residual network directly adds the input features to the deeper layer through the residual connection to avoid the gradient disappearance and can train a deeper network. ResNet is used as a convolutional neural network to extract richer micro-expression spatial features. LSTM is a RNN that uses a gate mechanism to record long-range dependencies and better learn time series data. LSTM is used as a recurrent neural network to better learn the temporal features of micro-expressions. Spatial attention learns the importance of different input positions and weights the spatial features. The spatial attention mechanism is used to automatically focus on the key areas of micro-expressions.

[0027] Specifically, the residual network structure enables the convolutional neural network to learn the spatial features of facial images more deeply, such as wrinkles, expression details, etc., providing rich spatial information for the follow-up. The gate mechanism and memory cells of LSTM can better perceive the temporal dynamic process of facial expression changes and provide key timing information. Spatial attention can automatically focus on the areas where facial expression changes most obviously, improving the ability to represent key frames. ResNet provides a spatial feature foundation, LSTM provides dynamic feature supplements, and the attention mechanism is optimized and adjusted. The three complement each other to form a multi-dimensional representation vector, which more comprehensively and accurately represents the semantic information of micro-expressions.

[0028] Furthermore, according to the correspondence between the representation vectors of micro-expressions and different emotion categories, a micro-expression knowledge graph is constructed, including: obtaining the representation vectors of micro-expressions; constructing three types of nodes representing positive emotions, negative emotions and neutral emotions, and using word vectors to represent different types of nodes; taking the nodes represented by word vectors and the representation vectors as input, constructing an LSTM network based on the attention mechanism, and outputting the representation vectors and the connection weights between the nodes; constructing a micro-expression knowledge graph according to the representation vectors, each type of node and the connection weights.

[0029] Specifically, three nodes are created to represent the three categories of positive emotion, negative emotion and neutral emotion respectively. A dense vector is learned for each node as its representation. The word vector model is used to learn the node representation. Models such as word2vector or GloVe are used. According to the emotional vocabulary training, the word vector representing the emotional concept is obtained. A large-scale corpus containing three categories of emotional words is constructed as training data. On the basis of the corpus training, the micro-expression dataset is used to further fine-tune the expressiveness of the node vector. Finally, each node vector represents the semantic information of its corresponding emotional type, and the dimension represents the implicit features related to the emotion. The node vector is used as the input of the subsequent model to learn the relationship between micro-expressions and different emotions.

[0030] More specifically, according to the correspondence between micro-expressions and typical emotions, such as raised eyebrows for positive emotions and drooped mouth corners for negative emotions, the correspondence between micro-expressions and emotion categories is compiled. A dataset of micro-expression images with emotion categories annotated is constructed, and the image recognition model is used to train the micro-expression images to learn the correspondence between facial expressions and emotion categories. The trained model can input micro-expression images, and the output is the probability of positive emotion and the probability of negative emotion. The micro-expression image is divided into emotion categories according to the emotion type with the largest probability value. The corresponding emotion type can be annotated in other modalities such as text, and jointly trained with facial micro-expression images to enhance the model's ability to learn the correspondence between micro-expressions and emotion categories. A model for identifying facial muscle movements can be introduced to assist in linking micro-expression muscle changes and emotion types. Emotions can be identified by recording the trend of micro-expression sequence changes, such as gradually changing from smile to sadness can be determined as negative emotions.

[0031] More specifically, word vector is a technology that uses a real number vector to represent the meaning of a word. Through word vector technology, a dense vector representation of each word can be obtained. In this application, the distributed semantic representation capability is strong, and the word vector is trained based on large-scale corpus, which can represent the semantic information of the word. The relationship between words is represented by vector space, and the similarity between word meanings can be calculated. The word vector is a dense vector with a fixed dimension, which is convenient for subsequent network processing. The word vector can be integrated into the neural network for end-to-end training, and fine-tuning and optimization of the knowledge graph node representation can be performed. The word vector technology is mature, and pre-trained word vectors representing emotional vocabulary can be easily obtained.

[0032] Specifically, input the word vector representing the node and the representation vector representing the micro-expression. Concatenate the word vector and the representation vector and input them into the LSTM of the attention mechanism. LSTM captures the temporal correlation characteristics between the word vector and the representation vector. The attention mechanism analyzes the relative importance of different nodes to the current micro-expression. Output the connection weights between the representation vectors at different time steps and the word vectors of each node. The weight indicates the degree of relevance between the node and the current micro-expression at this moment. Finally, the connection weights at different moments are combined, and a stable connection relationship is output to construct a knowledge graph. Repeat this process to gradually build the connection between micro-expressions and multiple emotional nodes. End-to-end training of the attention LSTM model optimizes feature extraction and connection weight prediction.

[0033] Specifically, word vectors are used to represent positive emotion nodes, negative emotion nodes, and neutral emotion nodes. The connection weights between nodes output by the LSTM network using the attention mechanism are used as the weights of the edges. The nodes and weighted edges are input into the graph database to form the network structure of the knowledge graph. According to the connection weights between the representation vector and the nodes, the representation vector is mapped into the graph as a micro-expression instance and associated with the corresponding emotion node. Input the representation vector of the micro-expression. Search for nodes in the graph and calculate the similarity between the representation vector and each emotion node vector. The similarity calculation can be performed using vector cosine similarity, inner product, etc. Find the top-K emotion nodes with the highest similarity to the representation vector. Use the connection weights between the representation vector and the emotion node output by the attention LSTM model. Among the top-K similar nodes, select the emotion node with the largest connection weight value. The largest connection weight means that this emotion node has the strongest correlation with the current micro-expression. Therefore, we determine that the emotion category represented by this most correlated emotion node is the emotion type expressed by the current micro-expression. Finally, the emotion type label corresponding to the micro-expression is output. The graph structure and connection weight optimization are continuously improved through training samples. Based on the constructed knowledge graph, graph parsing and reasoning tasks such as micro-expression understanding and sentiment analysis are carried out.

[0034] Furthermore, in the LSTM network based on the attention mechanism, the attention mechanism uses scaled dot product. A micro-expression knowledge graph is constructed, and the knowledge graph is constructed using Neo4j. Among them, scaled dot product (Scaled Dot-Product Attention) is an attention mechanism that calculates the dot product of the query vector and the key vector and the product of the scaling factor, and then performs softmax normalization to obtain the value used as the attention weight. In this application, the output of the LSTM network is used as the query vector q. The word vector of the emotional node is used as the key vector k. The scaled dot product of q and k is calculated to obtain the weight value, which indicates the importance of the node at the current moment. The weight value is normalized by softmax to obtain the attention weight of each node at the current moment. The weighted summation obtains the final connection weights of different nodes.

[0035] Among them, Neo4j is a graph database that uses a network structure to store data and uses nodes and relationships to represent and store data. It is a popular tool for building knowledge graphs. In this application, nodes are created in Neo4j to represent three types of emotions: positive, negative, and neutral. Micro-expression representation vector nodes are created to connect with emotion nodes. Weighted relationship edges are created using the connection weights output by the LSTM network. A weighted knowledge graph network is formed in the Neo4j graph database. Graph parsing, emotional reasoning, etc. are performed based on Neo4j's query language Cypher.

[0036] Furthermore, an entity alignment algorithm is used to align the micro-expression knowledge graph and physiological parameters in the same feature space using semantic constraints to construct a multimodal knowledge graph, including: obtaining the micro-expression knowledge graph and physiological parameters, the physiological parameters include heart rate and blood pressure; extracting the semantic feature vectors of the micro-expression knowledge graph nodes through a node embedding model; extracting the time series features of the physiological parameters through an LSTM network; using a t-distributed random nearest neighbor embedding algorithm to map the extracted semantic feature vectors and time series features to the same low-dimensional space; in the mapped low-dimensional space, using a knowledge graph translation model to align the semantic feature vectors of the micro-expression knowledge graph and the time series features of the physiological parameters to obtain an aligned multimodal knowledge graph.

[0037] Specifically, a knowledge graph has been constructed to represent the connection between micro-expressions and different emotional concepts. Use a heart rate monitoring device to obtain the real-time heart rate data of the subjects. Use an electronic blood pressure monitor to obtain the blood pressure data of the subjects. When the subjects watch videos that can trigger emotional responses, their micro-expressions and physiological data are recorded synchronously. The representation vectors of the collected micro-expression images are extracted. The physiological data is pre-processed by filtering and denoising. The pre-processed physiological parameter data is time-aligned and associated with the micro-expression knowledge graph. The associated graph data and physiological parameters are input together into the subsequent multimodal sentiment analysis model.

[0038] Among them, the node embedding model is a graph neural network that learns the Embedding operation that maps nodes to low-dimensional vector space by aggregating neighbor node information, and obtains a feature vector that can express the semantic information of the node. The semantic feature vector of the node is the vector representation learned by the node embedding model. It integrates the semantic information of the node in the knowledge graph network structure and can be used for downstream tasks. In this application, a node embedding model is used to take the emotional nodes and micro-expression nodes in the micro-expression knowledge graph as input, and learn to obtain a feature vector that combines the semantic information of the node as the graph input for subsequent multimodal sentiment analysis. Using node embedding, a more semantically expressive node representation can be obtained, and the topological structure information of the knowledge graph is retained, which is conducive to graph-based analysis and reasoning.

[0039] Among them, time series features refer to features extracted from time series data that represent time dependencies, reflecting the pattern of data changes over time. In this application, the collected time series data of physiological parameters are encoded and formatted so that they can be input into LSTM. Construct an LSTM network, input the physiological parameter data of the time step into the input layer, and output the feature vector at that moment into the output layer. The temporal dependencies of physiological parameters are captured through the gating structure of LSTM. The network is trained to learn and extract the intrinsic features of the physiological parameter time series. For a physiological parameter time series sample, LSTM outputs a sequence of feature vectors by time step. A feature vector of fixed length is obtained by using operations such as pooling on the feature sequence. The feature vector represents the time series features of the sample's physiological parameters. LSTM can effectively learn the intrinsic laws of time series data and extract important features of the time series.

[0040] Among them, t-SNE (t-Distributed Stochastic Neighbor Embedding) is a nonlinear dimensionality reduction algorithm that can map high-dimensional data to low-dimensional space while retaining the distance relationship between data. Mapping is the process of using an algorithm to calculate the coordinate transformation of high-dimensional features to low-dimensional space. Low-dimensional space is a representation space with lower dimensions, usually 2D or 3D, which is convenient for visualization. In this application, the semantic feature vector of the node and the time series features of the physiological signal are used as high-dimensional inputs. The t-SNE algorithm is used to map the two to a 2D or 3D low-dimensional space. The mapped features can be visualized and used as model input. t-SNE can retain the relative relationship between data, and the distance correlation of different modal features in low-dimensional space.

[0041] Among them, the knowledge graph translation model is a cross-modal graph neural network model that can learn the semantic mapping relationship between different modalities and realize the alignment of different modal graphs. In this application, the input is: node semantic features of micro-expression knowledge graph and time series features of physiological signals. Goal: Learn the alignment relationship between the two and obtain an aligned multimodal knowledge graph. Use the knowledge graph translation model to learn the mapping between node semantic features and time series features. Perform cross-modal alignment on the two to obtain an aligned multimodal knowledge graph that combines the two features.

[0042] Specifically, the node semantic feature vector and the physiological parameter time series feature vector of the micro-expression knowledge graph are input. The t-SNE algorithm is used to map the two feature vectors to the same low-dimensional space. Two subgraphs are constructed in the low-dimensional space, corresponding to the two feature vectors respectively. The knowledge graph translation model is trained, with the node semantic feature subgraph as the source graph and the time series feature subgraph as the target graph. The model learns the alignment mapping relationship between the source graph and the target graph nodes. According to the learned mapping relationship, the corresponding nodes in the two subgraphs are fused to form a new aligned node. A new knowledge graph is constructed using the fused node to form an aligned multimodal knowledge graph. Each node vector in the new graph contains semantic features and time series features. The edge connection relationship fuses the topological structure information of the two features. Finally, an aligned graph that fuses the two modal knowledge of micro-expressions and physiological parameters is obtained.

[0043] Furthermore, the node embedding model adopts the Translational Embedding model. Among them, the Translational Embedding model is a graph neural network model that can learn the vector representation of nodes so that the difference between the embedding vectors of the connected node pairs is close to the vector of the relationship between them. In this application, the network structure of the micro-expression knowledge graph is input. The Translational Embedding model is used to learn the semantic embedding representation of the nodes in the graph. The model goal is that the difference between the embedding vectors of the connected nodes should be close to the vector of the relationship type of the edge. In this way, the semantic expression of the node combined with the topological structure can be learned. Finally, the node embedding vector is output as a semantic feature for subsequent graph alignment. The graph structure can be modeled directly without the need to independently train word vectors, and semantic structure knowledge can be learned.

[0044] Furthermore, the knowledge graph translation model adopts the TransR model. Among them, TransR is a knowledge graph embedding model, which introduces relationship vectors to project the embedding vectors of entities into the space corresponding to the relationship, so as to learn the semantic information of entities and relationships. In this application, the knowledge graph network structure of two modalities is input. The TransR model is used for embedding learning of entities and relationships. The model maps the correspondence between entities of different modalities through projection. Cross-modal entity and relationship expressions are learned. Finally, the aligned multimodal knowledge graph network is output. TransR can model complex relational structures and learn cross-modal semantic mappings, so it is very suitable for knowledge graph alignment.

[0045] Furthermore, the dimension after t-SNE mapping is 2 to 64. Specifically, the t-SNE algorithm can map high-dimensional data to a low-dimensional space for easy visualization. The dimension of the low-dimensional space after mapping is usually set to 2 or 3 dimensions for visualization. However, only 2-3 dimensions cannot obtain sufficient expressive power. Taking into account the balance between visualization and expressive power, this scheme sets the target dimension of t-SNE mapping to 2-64 dimensions. 2 dimensions are used for the final visualization presentation. 64 dimensions are used to retain more sufficient feature expression capabilities. The upper and lower limits of the Setting target dimension are 2 to 64. Specific target dimensions can be selected in the middle, such as 16 or 32 dimensions. Specific settings are made according to the data scale and feature complexity.

[0046] 3. Beneficial effects

[0047] Compared with the prior art, the advantages of this application are:

[0048] The use of multimodal data, including user data and environmental data, fully considers the diverse expressions of users in different emotional states, as well as the impact of environmental factors on micro-expressions, thereby improving the comprehensiveness and robustness of micro-expression recognition;

[0049] The spatial features of micro-expression image sequences are extracted through convolutional neural networks, combined with recurrent neural networks to obtain temporal features, and the use of attention mechanisms effectively captures the semantic information of micro-expressions and improves the richness and expressiveness of representation vectors.

[0050] By constructing a micro-expression knowledge graph, the semantic information of micro-expressions is associated with emotion categories, providing effective references and constraints for micro-expression recognition, and improving recognition accuracy and interpretability;

[0051] The scaled dot product is used as part of the attention mechanism, which effectively increases the model's attention to important time steps in the micro-expression sequence, thereby improving the recognition performance;

[0052] Through the entity alignment algorithm, the micro-expression knowledge graph is feature-aligned with physiological parameters, making full use of multimodal information and improving the comprehensive performance and applicability of micro-expression recognition;

[0053] The Translational Embedding model was used for node embedding, which effectively mapped the node information in the micro-expression knowledge graph to a low-dimensional space, improving the information processing efficiency and the interpretability of the model.

[0054] The TransR model was used for knowledge graph translation, which achieved the association and alignment between the micro-expression knowledge graph and physiological parameters, providing reliable technical support for multimodal information fusion;

[0055] t-SNE mapping is used to map high-dimensional data to low-dimensional space, preserving the local structure and characteristics of the data, providing a powerful tool for data visualization and analysis; BRIEF DESCRIPTION OF THE DRAWINGS

[0056] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0057] Figure 1 is an exemplary flow chart of a micro-expression recognition method based on multimodal fusion according to some embodiments of this specification;

[0058] Figure 2 is an exemplary flow chart of extracting representation vectors of micro-expressions according to some embodiments of this specification;

[0059] Figure 3 is an exemplary flow chart for constructing a micro-expression knowledge graph according to some embodiments of this specification;

[0060] Figure 4 It is an exemplary flowchart for constructing a multimodal knowledge graph according to some embodiments of this specification. DETAILED DESCRIPTION

[0061] The method and system provided in the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0062] Figure 1 This is an exemplary flow chart of a micro-expression recognition method based on multimodal fusion shown in some embodiments of this specification, collecting multimodal data, the multimodal data includes user data and environmental data, the user data includes micro-expressions and physiological parameters; the environmental data includes light and temperature and humidity, and micro-expressions refer to fine facial expression features; extracting representation vectors of micro-expressions, the representation vectors represent the semantic features of micro-expression images or video sequences; constructing a micro-expression knowledge graph based on the correspondence between the representation vectors of micro-expressions and different emotion categories; using an entity alignment algorithm, in the same feature space, using semantic constraints to align the micro-expression knowledge graph and physiological parameters to construct a multimodal knowledge graph; based on the multimodal knowledge graph and environmental data, constructing a micro-expression recognition model, and outputting an emotion analysis vector representing the micro-expression recognition result;

[0063] Multimodal data acquisition: Collect multimodal input data including micro-expression image sequences, physiological parameters, and environmental data. Use a high-speed camera from ableView, model FastCamSA3120K, with a maximum acquisition rate of 120,000fps. Ask the subject to sit in a well-lit room, and fix the camera about 0.5 meters in front of the subject to ensure that the entire facial area is captured. Ask the subject to watch videos or image content that can trigger emotional responses, such as film and television clips of emotions such as joy and fear. Set the camera to continuously capture facial images at a speed of 120fps. After the subject finishes watching the emotional stimulation content, continue to collect for 1 to 2 minutes. High-speed cameras can generate a large amount of image data, so ensure that the storage device has a large enough capacity. Filter out key frames containing micro-expressions from the acquired high-frame rate image sequence and store them as a data set containing rich micro-expressions. Use the EmpaticaE4 wearable device to simultaneously collect multiple physiological signals such as heart rate, skin conductivity, and skin temperature. Wear the E4 device on the subject's wrist and adjust the strap to ensure good contact between the sensor and the skin. The subjects were asked to perform emotion-eliciting tasks, such as watching videos, images, and interacting. The E4 device was turned on, and the sampling frequency was set to 64 Hz. Physiological data was continuously collected as the subjects performed the tasks. The E4 data was transferred to the storage device in real time and recorded synchronously with the facial image sequence. The collected data was preprocessed by filtering and denoising to generate clean physiological parameter time series data. Finally, a data set containing physiological response characteristics such as heart rate and skin conductivity was obtained. The Intel Realsense D435i RGB-D camera was used to obtain color and depth images at the same time. In the room where the subjects performed the task, an RGB-D camera was set up for panoramic shooting. The RGB image resolution was set to 1920x1080 and the frame rate was 30 fps; the Depth image resolution was set to 640x480 and the frame rate was 30 fps. When the subjects performed the emotion-eliciting task, the RGB-D camera continuously captured the entire scene environment. The RGB image and the Depth image were stored in the form of video streams, and they were synchronously recorded and timestamped with other data. Key frames were extracted from the stored video streams to obtain data containing environmental scenes and personnel information. Use a network keyer to connect the high-speed camera, RGB-D camera and physiological sensor to achieve system synchronous triggering. All data sources are marked with the same timestamp before the acquisition begins. When the subject performs the emotion elicitation task, facial images, physiological data and environmental data are synchronously acquired. The data from different sources are stored as raw files at their respective acquisition frame rates, and the file name contains the synchronized timestamp. According to the task time period, different data sources are segmented and data fragments of the corresponding time period are extracted. According to the timestamp, the data fragments from different sources are aligned to construct multimodal samples. Finally, multimodal input data consisting of facial image sequences, physiological data sequences and environmental data aligned with synchronized timestamps is formed.

[0064] Figure 2 This is an exemplary flow chart for extracting representation vectors of micro-expressions according to some embodiments of this specification, and a data set of image sequences containing rich micro-expressions is collected, with a length of about 10-20 frames. The image is pre-processed by graying, scaling, etc., and adjusted to meet the network input specifications. ResNet-50 is selected as the convolutional neural network model for feature extraction. The pre-trained ResNet-50 model parameters are loaded. The image sequence is input into the ResNet-50 model in sequence. The feature map of the last layer is extracted from the convolutional network as the spatial feature representation of the entire sequence, denoted as A. Obtain the feature map matrix A, whose shape is [height, width, channels]. Flatten A into a vector with a length of height*width*channels. Calculate the L2 norm (Euclidean norm) of the vector: L2_norm=sqrt(sum(v**2)), where v is the flattened A vector. Regularize the vector v: v=v / (L2_norm+eps), where eps is a small smoothing term to avoid zero division errors. Reshape the regularized vector v to its original shape [height, width, channels]. The resulting matrix is ​​the feature map A after L2 regularization. This regularization process can reduce the range of eigenvalues ​​and play a certain regularization role.

[0065] Micro-expression image sequence, length is T frames. Feature map A extracted by ResNet. The input layer inputs the image sequence and feature A. The number of LSTM layers is set to 2. The output layer outputs the temporal hidden state vector B. Set the sequence length to T, and input the image and A in sequence. LSTM captures the temporal characteristics of the image sequence. For time step t, input the current image feature xt and the hidden state ht-1 at the previous moment. Calculation inside the LSTM unit: input gate: it=σ(Wi·[ht-1,xt]+bi), forget gate: ft=σ(Wf·[ht-1,xt]+bf), output gate: ot=σ(Wo·[ht-1,xt]+bo), candidate hidden state: c~t=tanh(Wc·[ht-1,xt]+bc), cell state: ct=ft*ct-1+it*c~t, current hidden state: ht=ot*tanh(ct). Output the hidden state vector ht at the current moment as ht. Calculate the hidden state vectors of T time steps in sequence. Preprocess the micro-expression image sequence to be predicted and adjust it to the model input specification with a length of T frames. Use ResNet to extract the spatial feature map A of the entire sequence. Load the pre-trained LSTM model. Initialize the hidden state h0 and cell state c0 of LSTM. Input the first frame image x1 and feature A according to the time step, and calculate h1. Input the second frame image x2 and feature A, and calculate h2. Input subsequent frame images and feature A in sequence. LSTM calculates and outputs ht frame by frame based on the current input and the hidden state at the previous moment. Record the T hidden state vectors output by LSTM as sequence-level temporal features B. B is passed to the subsequent attention network to fuse the final micro-expression features. The dimension of B is 64 or 128 dimensions, representing the characteristics of the time series. B serves as the input of the subsequent attention network.

[0066] Attention network structure: Input layer: accepts the hidden state vector B output by LSTM, and the shape of B is [time steps, vector dimension]. Dense layer: fully connected layer, the output dimension is T (time steps). Softmax layer: outputs the attention weight score C, the shape is [T]. Input batches of hidden states B to learn the importance of different time sequences for classification. Get the LSTM output hidden state vector B of the test image sequence, the shape is [T, D]. Input layer: input B. Fully connected layer: output dimension is T. Activation function: Tanh. Softmax layer: output weight C, the shape is [T]. Load the pre-trained attention network parameters. Input the test sequence B into the attention network. The fully connected layer learns the features of different time sequences. Softmax is converted to a weight score in the range of [0, 1]. Output is a weight vector C of length T. Each value in C represents the importance weight of the time sequence to the classification. C is used for subsequent weighted fusion with other features. The calculation method of attention weight C is: C=Softmax(FC(B)), where B is the hidden state vector output by LSTM, and FC is the fully connected layer. Properties of Softmax: The Softmax function converts the input value to the range of [0, 1]. And ensure that the sum of the output values ​​is 1. After Softmax, the output is also larger for inputs that are relatively larger. Explanation of weight C: Each value in C represents the importance contribution of the corresponding time step to the final classification. The time series with high importance has a larger value after FC and Softmax mapping. The weights are arranged in order of size. The maximum weight corresponds to the time series that contributes most to the classification. Finally, C is fused with other features according to the weight.

[0067] Spatial feature A, shape is [h, w, c], temporal feature B, shape is [T, d], weight vector C, shape is [T]. Temporal feature B, shape = [T, D], weight vector C, shape = [T]. Transpose B to shape = [D, T], expand C to shape = [D, T], B' = B*C# element-by-element multiplication, B' is the weighted temporal feature, shape = [D, T]. Transpose B' to shape = [T, D], A is the image spatial feature, shape = [H, W, C], E = concat ([B', A], axis = 3), E is the concatenated fusion feature map. E is used as the input for the subsequent extraction of the representation vector D. Input feature E, shape is [H, W, C]. Global maximum pooling calculation: Initialize the output vector D with a length of C. For the i-th channel: calculate the maximum value ei of the i-th channel in E. Take ei as the value of the i-th element in D. Through this global pooling method, a fixed-length vector D can be obtained, with a length equal to the number of channels C. Each element in the vector D represents the maximum response of the corresponding feature channel. The length of D is no longer related to the length of the input sequence, achieving the effect of fixed length. D is used as the overall feature representation of the micro-expression sequence and can be input into subsequent classification / regression models. The input is a vector D with a shape of [C], where C is the vector length. Calculate the L2 norm of D: L2_norm=sqrt(sum(D^2)), and calculate the square root of the sum of the squares of each value in D element by element. Perform L2 regularization: D=D / (L2_norm+eps), where eps is a very small number to prevent division by zero errors. Regularization is achieved by dividing by the L2 norm: each element in D is reduced by the same proportion. The influence of elements with larger values ​​in D on the results is reduced. Overfitting is prevented to a certain extent. Finally, the regularized representation vector D is output. D is used as the input of models such as classifiers.

[0068] Figure 3This is an exemplary flow chart for constructing a micro-expression knowledge graph according to some embodiments of this specification, collecting words representing positive, negative and neutral emotions, with 100-200 words for each category. Use the word vector model Word2Vec to train the vector representation of the vocabulary. Take the average of the positive word set to obtain the vector p of the positive emotional node. Take the average of the negative word set to obtain the vector n of the negative emotional node. Take the average of the neutral word set to obtain the vector m of the neutral emotional node. Perform L2 normalization on p, n, and m respectively. Finally, the word vectors representing the three types of emotional nodes, positive, negative and neutral, are obtained, denoted as p, n, and m. The word vector and the micro-expression vector D are used as input and input into LSTM to model the relationship between the two. LSTM input: micro-expression representation vector D, shape =

[128] , 3 emotional node word vectors, respectively p, n, m, and the shape is

[100] . LSTM structure: input layer: input D and p, n, m, LSTM layer: learn the associated features of the two, attention layer: learn the weights of different inputs. Output layer: Output 3 weights, indicating the relevance of D to each node. Model training: Input labeled D and word vectors, learn weights to indicate the relationship between D and sentiment. Weight output: Input test D and word vectors, LSTM outputs 3 weights. Weights indicate the degree of association between D and each sentiment node. Construct connections of knowledge graphs based on weights.

[0069] The hidden states of LSTM at different times are scored, and Softmax is converted into weights in the range of [0, 1]. For positive node word vectors, the attention layer outputs weight a1, for negative node word vectors, the attention layer outputs weight a2, and for neutral node word vectors, the attention layer outputs weight a3. a1=Softmax(Score(h1)), a2=Softmax(Score(h2)), a3=Softmax(Score(h3)). a1 represents the degree of association between D and positive sentiment nodes, a2 represents the degree of association between D and negative sentiment nodes, and a3 represents the degree of association between D and neutral sentiment nodes. The connection of the knowledge graph is established according to the weights.

[0070] Create node D to represent the micro-expression representation vector, and create nodes P, N, M to represent the positive, negative and neutral emotion word vectors. D-[:RELATED{weight:0.8}]->P, D-[:RELATED{weight:0.3}]->N, D-[:RELATED{weight:0.5}]->M. Import the micro-expression vector attribute for the D node, and import the word vector attribute for the P, N, M nodes. Set the weight attribute of the RELATED relationship according to the weight output by the attention network. Micro-expression feature node D: Node name: Micro expressionEmbedding, attribute: vector=[0.1, 0.3, ...] #128-dimensional representation vector. Emotion nodes P, N, M: Node name: Positive Emotion, Negative Emotion, Neutral Emotion, attribute: vector=[0.2, 0.5, ...] #100-dimensional word vector. Related connections: Relationship type: RELATED, D-[:RELATED{weight:0.8}]->PositiveEmotion, D-[:RELATED{weight:0.3}]->NegativeEmotion, D-[:RELATED{weight:0.5}]->NeutralEmotion. The weight attribute indicates the strength of the relationship between D and each emotion node. The above nodes and connections constitute a knowledge graph connecting micro-expressions and emotions.

[0071] Figure 4 This is an exemplary flow chart for constructing a multimodal knowledge graph according to some embodiments of this specification. The input knowledge graph contains three types of nodes: micro-expression node A, emotional node B (positive, negative, neutral), and relationship node R. TransE model structure: Input graph triples (A, R, B), Embedding layer: map nodes A, B, R to vectors, and the scoring function calculates the distance between (A+R) and B. Node embedding learning: The training goal is to minimize the distance between the correct triples, and the vector representation of the nodes is learned through batch triple training. The vector of the micro-expression node A represents its semantic information, the vector of the positive emotional node B represents the positive semantics, and the vector of the relationship R represents the connection between the two nodes. Finally, the semantic vector representation of all nodes in the knowledge graph is obtained.

[0072] Sequence modeling: Input data: heart rate time series: (x1, x2, ..., xn), blood pressure time series: (y1, y2, ..., ym). LSTM model: Input layer: accept sequence values, LSTM layer: learn temporal dependencies, output layer: output feature vector of current time step. Heart rate features: input heart rate sequence to LSTM, LSTM learns sequence pattern, and outputs feature vector h1 of heart rate at each time. Blood pressure features: input blood pressure sequence to LSTM, LSTM learns sequence pattern, and outputs feature vector h2 of blood pressure at each time. Finally, LSTM is used to learn temporal dynamic features of heart rate and blood pressure respectively.

[0073] Spatial mapping: Input features: node vector X1, dimension d1, heart rate feature X2, dimension d2. Blood pressure feature X3, dimension d3, t-SNE model: Calculate the probabilistic similarity between high-dimensional inputs and learn the coordinate representation of low-dimensional space so that the similarity remains unchanged. Spatial mapping: Take X1, X2, X3 as input, learn 2-64 dimensional coordinates y1, y2, y3, so that the distance between the three in the low-dimensional space reflects the similarity in high dimensions. Output: low-dimensional representation y1 of the node vector, low-dimensional representation y2 of the heart rate feature, low-dimensional representation y3 of the blood pressure feature, the three are mapped to the same low-dimensional space for subsequent feature alignment.

[0074] Feature alignment: Input vectors: node semantic vector: y1, heart rate feature vector: y2, blood pressure feature vector: y3. TransR model: Input the vectors and relationship vectors of two nodes, map the two nodes to the same subspace through relationship transformation, and minimize the distance between the two nodes. Node alignment: Take y1 as the head entity and y2 as the tail entity, learn the transformation matrix M, and also LEARNING the M matrix of y1 and y3. The matrix M minimizes the distance between the two nodes. Align vectors: Transform y1 using the learned M to obtain the aligned semantic vector y1'. y2 and y3 remain unchanged, and the three are mapped to a common alignment space. Finally, the aligned multimodal vector is obtained to construct an aligned knowledge graph.

[0075] Build a knowledge graph and align vectors: semantic vector y1', heart rate feature vector y2, blood pressure feature vector y3. Create nodes: node A, attribute y1', node B, attribute y2, node C, attribute y3. Create connections: A-[:ALIGNED_WITH]-B, B-[:ALIGNED_WITH]-C, A-[:ALIGNED_WITH]-C. Set node properties: import the aligned semantic vector for node A, import the heart rate feature for B, and import the blood pressure feature for C. Build a multimodal graph, including: aligned semantic nodes, aligned physiological nodes, and aligned connections. Complete the multimodal knowledge graph with aligned semantic and physiological features.

[0076] Import the aligned vectors as node attributes: Node A: Name: aligned_embedding, Attributes: vector=[0.3, 0.5, 0.8...]# Aligned semantic vector. Node B: Name: aligned_heart rate, Attributes: vector=[0.2, 0.6, 0.7...]# Aligned heart rate feature vector. Node C: Name: aligned_blood pressure, Attributes: vector=[0.4, 0.1, 0.2...]# Aligned blood pressure feature vector. Create aligned connections: A-[:ALIGNED_WITH]->B, B-[:ALIGNED_WITH]->C, C-[:ALIGNED_WITH]->A, and finally construct a multimodal graph containing nodes and alignment relationships of aligned vectors.

[0077] Sentiment analysis: Based on the constructed multimodal knowledge graph, the information of different modalities is integrated to output the sentiment analysis vector representing the micro-expression recognition result. Input the multimodal graph, which contains nodes: micro-expression semantic vector A, heart rate sequence feature vector B, blood pressure sequence feature vector C. Graph convolutional network structure: Input: multimodal graph, graph convolution layer: learn node neighbor features, aggregation layer: merge features of different nodes, output layer: output sentiment analysis vector y. Model training: Use the labeled multimodal graph to train the model and learn to extract emotion features from heterogeneous nodes. Sentiment prediction: Input the tested multimodal graph, and the model outputs the sentiment analysis result vector y. 5. Vector y combines the information of micro-expressions, heart rate, and blood pressure to represent the final sentiment analysis result.

[0078] Model optimization: Regularization and adversarial learning are used to improve the generalization ability of the model and enhance the learning of implicit semantics. Add L2 regularization terms to the loss function to penalize model parameters and enhance generalization ability. L2Loss=λ*sum(θ^2). The regularization strength is controlled by adjusting λ. Adversarial training: Introduce perturbed data as adversarial samples, train original samples and adversarial samples at the same time, and improve the robustness of the model to small perturbations. Virtual adversarial training: Introduce perturbations through gradient calculation, without the need for additional adversarial samples, and enhance the generalization ability of the model online. Through regularization and adversarial learning, the model's learning of semantic information is improved, the generalization ability is enhanced, and implicit emotions are detected.

[0079] The invention of the present application and its implementation methods are described schematically above. The description is not restrictive. Without departing from the spirit or basic features of the present application, the present application can be implemented in other specific forms. What is shown in the accompanying drawings is only one of the implementation methods of the invention of the present application. The actual structure is not limited to this, and any figure mark in the claims should not limit the claims involved. Therefore, if ordinary technicians in this field are inspired by it, without departing from the purpose of the present invention, they can creatively design structural methods and embodiments similar to the technical solution, which should all belong to the scope of protection of the present application. In addition, the word "including" does not exclude other elements or steps, and the word "one" before the element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any specific order.

Claims

1. A micro-expression recognition method based on multimodal fusion, comprising: Collect multimodal data, which includes user data and environmental data. User data includes micro-expressions and physiological parameters. Environmental data includes light, temperature and humidity, and micro-expressions refer to the subtle facial expression features; Extracting representation vectors of micro-expressions, where the representation vectors represent semantic features of micro-expression images or video sequences; According to the correspondence between the representation vectors of micro-expressions and different emotion categories, a micro-expression knowledge graph is constructed; Using entity alignment algorithm, in the same feature space, semantic constraints are used to align the micro-expression knowledge graph and physiological parameters to construct a multimodal knowledge graph. Based on the multimodal knowledge graph and environmental data, a micro-expression recognition model is constructed to output a sentiment analysis vector representing the micro-expression recognition result; Extract the representation vector of micro-expression, which represents the semantic features of micro-expression image or video sequence, including: The collected micro-expression image sequence is used as input, and a convolutional neural network is used to obtain a feature map A representing the spatial features of the image sequence; Obtaining feature map A includes the following sub-steps: Input the collected micro-expression image sequence into the ResNet-50 convolutional neural network model to obtain a feature mapping matrix, the shape of which is [height, width, channels]; Flatten the feature map matrix into a vector v; Calculate the L2 norm of vector v, L2_norm = sqrt(sum(v 2 )); For L2 _ n o rm is regularized, and the regularized vector v'=v / (L2_norm+eps) ， Among them, eps is a small smoothing term; Reshape the regularized vector v into shape [height, width, channels] to obtain the feature map A; Take the micro-expression image sequence and feature map A as input, and use the recurrent neural network to obtain the hidden state vector B representing the temporal features; Obtaining the hidden state vector B includes the following sub-steps: Input the micro-expression image sequence and feature map A into the pre-trained LSTM model to obtain the hidden state vector B; Take the hidden state vector B as input and use the attention mechanism neural network to calculate the weight score C at different time steps; Calculate the weight score C at different time steps using the following formula: C = Softmax(FC(B)) Among them, FC represents the fully connected layer, and Softmax represents converting the input value to the range of 0 to 1; The acquired feature map A, hidden state vector B and weight score C are combined to obtain a representation vector D of the micro-expression representing the semantic information of the image sequence, where each dimension in the representation vector D corresponds to a different facial muscle movement; The feature map A represents the texture features of the micro-expression image sequence; The hidden state vector B represents the change trend between the feature maps A of adjacent time steps in the micro-expression image sequence, reflecting the change process of the texture features of micro-expressions between adjacent time sequences; The weight scores C are arranged in descending order according to their importance contribution to micro-expression recognition at different time steps. The importance contribution is calculated by the attention mechanism neural network based on the input hidden state vector B. The representation vector D uses L2 regularization to prevent overfitting; The convolutional neural network uses the residual network ResNet; The recurrent neural network uses the long short-term memory network LSTM; The neural network with attention mechanism adopts spatial attention mechanism; According to the correspondence between the representation vectors of micro-expressions and different emotion categories, a micro-expression knowledge graph is constructed, including: Obtain the representation vector of micro-expression; Construct three types of nodes representing positive emotions, negative emotions, and neutral emotions, and use word vectors to represent different types of nodes; The nodes and representation vectors represented by word vectors are taken as input, and an LSTM network based on the attention mechanism is constructed to output the representation vectors and the connection weights between each node. Construct a micro-expression knowledge graph based on representation vectors, various types of nodes, and connection weights; In the LSTM network based on the attention mechanism, the attention mechanism uses scaled dot product; Construct a micro-expression knowledge graph, which is built using Neo4j; Construct a multimodal knowledge graph, including: Obtain micro-expression knowledge graph and physiological parameters, including heart rate time series (x1, x2, ....., xn) and blood pressure time series (y1, y2, ....., ym) ； Extract the semantic feature vectors of micro-expression knowledge graph nodes through node embedding model; Extracting the time series features of physiological parameters through the LSTM network includes the following sub-steps: Input the heart rate time series into LSTM and output the feature vector h1 of the heart rate at each time; Input the blood pressure time series into LSTM and output the feature vector h2 of blood pressure at each time; The t-distributed random neighbor embedding algorithm is used to map the extracted semantic feature vectors and time series features to the same low-dimensional space, including the following sub-steps: The node vector, heart rate feature and blood pressure feature are used as input, and the t-SNE model is used to obtain vectors y1, y2 and y3; wherein, vector y1 represents the node semantic vector, y2 represents the heart rate feature vector, and y3 represents the blood pressure feature vector; the dimensions of y1, y2 and y3 range from 2 to 64; In the mapped low-dimensional space, the knowledge graph translation model is used to align the semantic feature vectors of the micro-expression knowledge graph and the time series features of the physiological parameters to obtain the aligned multimodal knowledge graph, which includes the following sub-steps: Take y1 as the head entity and y2 as the tail entity, and get the matrix M by minimizing the distance between the nodes; Use matrix M to transform the coordinates of y1 to obtain y1'; Map y1', y2 and y3 to the same space to obtain the aligned multimodal vectors for constructing the multimodal knowledge graph.

2. The micro-expression recognition method based on multimodal fusion according to claim 1, characterized in that: The node embedding model adopts the Translational Embedding model.

3. The micro-expression recognition method based on multimodal fusion according to claim 2, characterized in that: The knowledge graph translation model adopts the TransR model.

4. The micro-expression recognition method based on multimodal fusion according to claim 3, characterized in that: The dimensions after t-SNE mapping range from 2 to 64.

Images