An emotion state recognition method, system, electronic device, and storage medium

By employing a multimodal pre-training and fine-tuning emotional state recognition method, which combines geometric, texture, and appearance modal information and utilizes graph convolutional networks and transformer networks for feature extraction and fusion, the method addresses the issue of poor performance in single-modal recognition and improves the accuracy and stability of emotional state recognition.

CN118152982BActive Publication Date: 2026-11-10CHINA TELECOM CORP LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410396755.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-04-02
Publication Date
2026-11-10
Estimated Expiration
2044-04-02

AI Technical Summary

Technical Problem

Existing emotion state recognition technologies are mainly based on a single modality, resulting in poor emotion recognition performance. Furthermore, the interaction of multimodal information is insufficient, leading to inconsistencies and conflicts.

Method used

A multimodal pre-training and fine-tuning emotional state recognition method is adopted to obtain geometric modal information, texture modal information, and appearance modal information of facial key points. Feature extraction and fusion are performed using graph convolutional networks and transformer networks. Model pre-training and fine-tuning are carried out through geometric appearance matching, mask geometric reconstruction, and mask appearance reconstruction tasks.

Benefits of technology

It improves the accuracy and stability of emotion state recognition, enhances the complementarity of multimodal information and the generalization ability of the model, and achieves more accurate facial emotion state prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118152982B_ABST
    Figure CN118152982B_ABST
Patent Text Reader

Abstract

This invention provides a method and system for emotion state recognition based on multimodal pre-training and fine-tuning. The method includes: acquiring geometric modal information, texture modal information, and appearance modal information of multiple facial key points; constructing an emotion state analysis model based on multimodal pre-training and fine-tuning according to the geometric modal information, texture modal information, and appearance modal information; training the emotion state analysis model; and using the trained emotion state analysis model to predict facial emotion states. This invention can more accurately predict facial emotion states, providing a useful direction for exploration and practical experience for the further development of emotion recognition technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the fields of intelligent interaction, human-computer interaction, and intelligent customer service, and in particular to an emotion state recognition method based on multimodal pre-training and fine-tuning, an emotion state recognition system based on multimodal pre-training and fine-tuning, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Emotional state recognition refers to the analysis and understanding of various information such as human language, voice, and facial expressions to identify the emotional states expressed by people in communication. Emotional states can generally be divided into multiple categories, such as happiness, sadness, anger, and surprise. This technology is commonly used in fields such as natural language processing, speech recognition, and computer vision, aiming to help machines better understand and process human emotions, thereby improving the effectiveness and intelligence of human-computer interaction.

[0003] Traditional sentiment recognition technologies are primarily based on a single modality, such as text, speech, or image, failing to fully exploit multimodal information and resulting in poor sentiment recognition performance. Furthermore, existing multimodal sentiment analysis methods utilize post-processing for modal interaction, leading to insufficient multimodal information exchange and even inconsistencies and conflicts. Therefore, current sentiment recognition schemes generally exhibit poor performance. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide an emotion state recognition method based on multimodal pre-training and fine-tuning, an emotion state recognition system based on multimodal pre-training and fine-tuning, an electronic device, and a computer-readable storage medium to overcome or at least partially solve the above problems.

[0005] To address the aforementioned issues, this invention discloses an emotion state recognition method based on multimodal pre-training and fine-tuning. The method includes: acquiring geometric modal information, texture modal information, and appearance modal information of multiple facial key points; constructing an emotion state analysis model based on multimodal pre-training and fine-tuning according to the geometric modal information, texture modal information, and appearance modal information; training the emotion state analysis model; and using the trained emotion state analysis model to predict facial emotion states.

[0006] Optionally, acquiring the geometric modality information, texture modality information, and appearance modality information of multiple facial key points includes: selecting multiple target facial key points from a preset set of multiple sample facial key points; acquiring the two-dimensional coordinate information, histogram of oriented gradients (HOR) features, and facial image patches of the multiple target facial key points; using the two-dimensional coordinate information as the geometric modality information, the HOR features as the texture modality information, and the facial image patches as the appearance modality information.

[0007] Optionally, the step of building an emotional state analysis model based on multimodal pre-training and fine-tuning according to the geometric modality information, the texture modality information, and the appearance modality information includes: creating a graph convolutional network, a transformer network, and an activation function respectively; building the emotional state analysis model according to the graph convolutional network, the transformer network, and the activation function; wherein, the graph convolutional network is used to extract geometric modality encoding features of the geometric modality information and the texture modality information; the transformer network is used to perform feature encoding on the appearance modality information to obtain appearance modality encoding features; the transformer network is also used to fuse the geometric modality encoding features and the appearance modality encoding features to obtain a query matrix, a key matrix, and a value matrix; and the activation function is used to perform classification processing on the query matrix, the key matrix, and the value matrix to obtain a classification result.

[0008] Optionally, training the emotion state analysis model includes: setting a geometric appearance matching task, a mask geometric reconstruction task, and a mask appearance reconstruction task; pre-training the emotion state analysis model based on the geometric appearance matching task, the mask geometric reconstruction task, and the mask appearance reconstruction task; and fine-tuning the pre-trained emotion state analysis model using an expression dataset.

[0009] Optionally, the geometric appearance matching task includes: replacing the facial image patches corresponding to multiple target facial key points with other facial image patches corresponding to other target facial key points according to a first probability; and replacing the target facial key points and the histogram of oriented gradients (HOPs) corresponding to multiple facial image patches with other target facial key points and other HOPs corresponding to other facial image patches according to the first probability; determining whether the geometric modality information and the texture modality information match the appearance modality information based on the vector at the 0th position of the geometric modality encoding feature and the appearance modality encoding feature; wherein, the loss function of the geometric appearance matching task is the cross-entropy loss function.

[0010] Optionally, the mask geometry reconstruction task includes: masking multiple target facial key points according to a second probability; using the geometric modality encoding features to connect a fully connected layer to predict the masked multiple target facial key points; wherein, the loss function of the mask geometry reconstruction task is the mean squared error loss function.

[0011] Optionally, the mask appearance reconstruction task includes: masking multiple facial image patches according to a third probability; using the appearance modality coding features to connect a fully connected layer to predict the masked multiple facial image patches; wherein, the loss function of the mask appearance reconstruction task is the mean squared error loss function.

[0012] This invention also discloses an emotion state recognition system based on multimodal pre-training and fine-tuning. The system includes: an information acquisition module for acquiring geometric modal information, texture modal information, and appearance modal information of multiple facial key points; a model building module for building an emotion state analysis model based on multimodal pre-training and fine-tuning according to the geometric modal information, texture modal information, and appearance modal information; a model training module for training the emotion state analysis model; and an emotion prediction module for predicting facial emotion states using the trained emotion state analysis model.

[0013] Optionally, the information acquisition module includes: a key point filtering module, used to filter out multiple target facial key points from a preset multiple sample facial key points; an information extraction module, used to acquire two-dimensional coordinate information, histogram of directional gradients features, and facial image patches of the multiple target facial key points; and an information determination module, used to use the two-dimensional coordinate information as the geometric modality information, the histogram of directional gradients features as the texture modality information, and the facial image patches as the appearance modality information.

[0014] Optionally, the model building module includes: a network function creation module for creating a graph convolutional network, a transformer network, and an activation function; a sentiment analysis module building module for building the sentiment state analysis model based on the graph convolutional network, the transformer network, and the activation function; a feature extraction module for extracting geometric modal encoding features of the geometric modality information and the texture modality information using the graph convolutional network; and using the transformer network to perform feature encoding on the appearance modality information to obtain appearance modality encoding features; a feature fusion module for fusing the geometric modality encoding features and the appearance modality encoding features using the transformer network to obtain a query matrix, a key matrix, and a value matrix; and a matrix classification module for classifying the query matrix, key matrix, and value matrix using the activation function to obtain a classification result.

[0015] Optionally, the model training module includes: a task setting module for setting geometric appearance matching tasks, mask geometric reconstruction tasks, and mask appearance reconstruction tasks; a model pre-training module for pre-training the sentiment state analysis model based on the geometric appearance matching tasks, the mask geometric reconstruction tasks, and the mask appearance reconstruction tasks; and a model fine-tuning module for fine-tuning the pre-trained sentiment state analysis model using an expression dataset.

[0016] Optionally, the geometric appearance matching task includes: a replacement module, configured to replace the facial image patches corresponding to multiple target facial key points with other facial image patches corresponding to other target facial key points according to a first probability, and to replace the target facial key points and the histogram of oriented gradients (HOPs) corresponding to multiple facial image patches with other target facial key points and other HOPs corresponding to other facial image patches according to the first probability; and a matching judgment module, configured to determine whether the geometric modality information and the texture modality information match the appearance modality information based on the vector at the 0th position of the geometric modality encoding feature and the appearance modality encoding feature; wherein, the loss function of the geometric appearance matching task is the cross-entropy loss function.

[0017] Optionally, the mask geometric reconstruction task includes: a geometric occlusion module, used to occlude multiple target facial key points according to a second probability; and a geometric prediction module, used to predict the occluded multiple target facial key points by using the geometric modality encoding features connected to a fully connected layer; wherein the loss function of the mask geometric reconstruction task is the mean squared error loss function.

[0018] Optionally, the mask appearance reconstruction task includes: an appearance occlusion module, used to occlude multiple facial image patches according to a third probability; and an appearance prediction module, used to predict the occluded multiple facial image patches by using the appearance modality coding features connected to a fully connected layer; wherein the loss function of the mask appearance reconstruction task is the mean squared error loss function.

[0019] This invention also discloses an electronic device, comprising: one or more processors; and one or more machine-readable media storing instructions thereon, which, when executed by the one or more processors, cause the electronic device to perform an emotion state recognition method based on multimodal pre-training and fine-tuning as described above.

[0020] This invention also discloses a computer-readable storage medium storing a computer program that enables a processor to execute an emotion state recognition method based on multimodal pre-training and fine-tuning as described above.

[0021] The embodiments of the present invention have the following advantages:

[0022] This invention provides an emotion state recognition scheme based on multimodal pre-training and fine-tuning. It acquires geometric modal information, texture modal information, and appearance modal information of multiple facial key points. Then, it builds an emotion state analysis model based on multimodal pre-training and fine-tuning according to the geometric modal information, texture modal information, and appearance modal information. Subsequently, it trains the emotion state analysis model and finally uses the trained emotion state analysis model to predict facial emotion state.

[0023] Traditional sentiment recognition technologies are primarily based on a single modality, while this invention combines multiple modalities, including geometric, textural, and appearance modalities, thereby fully leveraging various information sources and improving the understanding and recognition of sentiment states. A sentiment analysis model based on multimodal pre-training and fine-tuning is constructed, enabling better interaction and fusion between different modalities. This fully utilizes the complementarity of multimodal information, improving the accuracy and stability of sentiment recognition. Training the sentiment analysis model through pre-training and fine-tuning allows for the early learning of general features using large-scale data, and fine-tuning adapts it to specific tasks, thus improving the model's generalization ability and recognition performance.

[0024] In summary, the embodiments of the present invention can more accurately predict facial emotional states, providing a useful direction for exploration and practical experience for the further development of emotion recognition technology. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the steps of an emotion state recognition method based on multimodal pre-training and fine-tuning according to an embodiment of the present invention.

[0026] Figure 2 This is a schematic diagram illustrating the relationship between facial key points, two-dimensional coordinate information, and HOG features in an embodiment of the present invention.

[0027] Figure 3 This is a schematic diagram of the structure of a graph convolutional network according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of a transformer network according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of the structure of a multimodal fusion module according to an embodiment of the present invention;

[0030] Figure 6 This is a schematic diagram illustrating the training process of an emotion state recognition model according to an embodiment of the present invention;

[0031] Figure 7This is a schematic diagram comparing the application effects of an emotion state recognition model according to an embodiment of the present invention with other models;

[0032] Figure 8 This is a structural block diagram of an emotion state recognition system based on multimodal pre-training and fine-tuning according to an embodiment of the present invention. Detailed Implementation

[0033] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] This invention proposes a multimodal pre-training and fine-tuning-based sentiment analysis model (MPF) by extracting geometric, texture, and appearance modal information from facial key points. The model utilizes a graph convolutional network (GCN) to extract geometric modal information and a transformer network to extract appearance modal information, fusing the geometric and appearance modal information through a multimodal fusion module. During model training, three pre-training tasks are proposed: geometric appearance matching, mask geometric reconstruction, and mask appearance reconstruction, to help the model learn better feature representations. Finally, the trained sentiment analysis model is used to predict facial sentiment states. This invention fully leverages multimodal information, improving the accuracy and stability of sentiment recognition, and providing valuable exploration directions and practical experience for the further development of sentiment recognition technology.

[0035] Reference Figure 1 The diagram illustrates a flowchart of an emotion state recognition method based on multimodal pre-training and fine-tuning according to an embodiment of the present invention. Specifically, this emotion state recognition method based on multimodal pre-training and fine-tuning may include the following steps:

[0036] Step 101: Obtain geometric modal information, texture modal information, and appearance modal information of multiple facial key points.

[0037] In embodiments of the present invention, acquiring geometric, textural, and appearance modal information of facial key points is crucial for emotion state recognition. Geometric modal information typically includes the location information of facial key points, textural modal information includes the texture features of facial skin, and appearance modal information includes the overall appearance features of the face. This information can be acquired through image processing and computer vision techniques.

[0038] For example, suppose we have an image containing a face. First, we can use a face detection algorithm to detect the location of the face, and then use a facial landmark detection algorithm to obtain the location information of facial landmarks. Next, we can use image processing techniques to extract texture features of the facial region, such as using filters to extract texture information. Finally, we can use the pixel values ​​of the face image patches as appearance modality information. In this way, we can obtain the geometric modality information, texture modality information, and appearance modality information of the face in the image.

[0039] Step 102: Build an emotion state analysis model based on multimodal pre-training and fine-tuning according to geometric modal information, texture modal information and appearance modal information.

[0040] In embodiments of the present invention, the purpose of building an emotion state analysis model based on multimodal pre-training and fine-tuning is to fully utilize multimodal information and improve the accuracy and stability of emotion state recognition. This step requires designing a model architecture that combines geometric modality information, texture modality information, and appearance modality information, and then pre-training and fine-tuning it.

[0041] For example, a neural network model can be designed where geometric modality information is input into a graph convolutional network (GCN) to extract features, texture modality information is input into a convolutional neural network (CNN) to extract features, and appearance modality information is input into a fully connected layer to extract features. Then, the features from these three modalities are fused to obtain the final sentiment analysis features. This model architecture can fully utilize multimodal information and improve the effectiveness of sentiment recognition.

[0042] Step 103: Train the emotional state analysis model.

[0043] In embodiments of the present invention, after the emotion state analysis model is built, it needs to be trained so that the model learns better feature representations. The training process can use a multimodal dataset, which contains rich emotion state information, such as a dataset of facial images containing different emotion states such as happiness, sadness, and anger.

[0044] For example, a multimodal dataset can be input into a sentiment analysis model. The model parameters can be continuously adjusted using the backpropagation algorithm to make the model's predictions on the training set as close as possible to the actual sentiment states. Through multiple iterations of training, the model can be gradually optimized, improving its ability to accurately identify sentiment states.

[0045] Step 104: Predict facial emotional states using the trained emotional state analysis model.

[0046] In embodiments of the present invention, the trained emotion state analysis model can be used to predict new facial emotion states. For a new facial image, it can be input into the trained model, which will output a corresponding emotion state probability distribution. The emotion state with the highest probability can be selected as the final prediction result based on the probability distribution.

[0047] For example, suppose we have a new facial image. We can input it into a trained emotion analysis model, and the model will output the probability distribution of different emotion states such as happiness, sadness, and anger. If the model outputs the highest probability of happiness, then we can determine that the emotional state expressed by the facial image is happiness.

[0048] This invention provides an emotion state recognition scheme based on multimodal pre-training and fine-tuning. It acquires geometric modal information, texture modal information, and appearance modal information of multiple facial key points. Then, it builds an emotion state analysis model based on multimodal pre-training and fine-tuning according to the geometric modal information, texture modal information, and appearance modal information. Subsequently, it trains the emotion state analysis model and finally uses the trained emotion state analysis model to predict facial emotion state.

[0049] Traditional sentiment recognition technologies are primarily based on a single modality, while this invention combines multiple modalities, including geometric, textural, and appearance modalities, thereby fully leveraging various information sources and improving the understanding and recognition of sentiment states. A sentiment analysis model based on multimodal pre-training and fine-tuning is constructed, enabling better interaction and fusion between different modalities. This fully utilizes the complementarity of multimodal information, improving the accuracy and stability of sentiment recognition. Training the sentiment analysis model through pre-training and fine-tuning allows for the early learning of general features using large-scale data, and fine-tuning adapts it to specific tasks, thus improving the model's generalization ability and recognition performance.

[0050] In summary, the embodiments of the present invention can more accurately predict facial emotional states, providing a useful direction for exploration and practical experience for the further development of emotion recognition technology.

[0051] In one exemplary embodiment of the present invention, one method for obtaining geometric modal information, texture modal information, and appearance modal information of multiple facial key points involves selecting multiple target facial key points from a preset set of sample facial key points, and obtaining the two-dimensional coordinate information, histogram of oriented gradients (HOR) features, and facial image patches of the multiple target facial key points. The two-dimensional coordinate information is used as geometric modal information, the HOR features as texture modal information, and the facial image patches as appearance modal information. The core of this embodiment is to select multiple target facial key points from a preset set of sample facial key points and obtain the two-dimensional coordinate information, HOR features, and facial image patches of these key points, which are respectively used as geometric, texture, and appearance modal information.

[0052] In practical applications, the pre-defined multiple sample facial keypoints refer to a pre-determined set of key points used to describe facial features, which can be obtained through face detection and keypoint localization algorithms. Selecting multiple target facial keypoints from these pre-defined keypoints can be achieved through certain selection rules or optimization algorithms to retain the most representative and informative keypoints. For each target facial keypoint, its two-dimensional coordinate information, i.e., the keypoint's position information in the image, can be obtained to describe the geometric structure of the face. This two-dimensional coordinate information can be directly obtained from the face keypoint detection algorithm. Histogram of Oriented Gradients (HOG) features can be extracted for each keypoint as texture modality information. These features can be obtained by calculating the gradient histogram of the region surrounding the keypoint, used to describe the texture features of the face, such as skin details and texture. For each keypoint, a facial image patch can be extracted centered on it as appearance modality information. These facial image patches can include the facial region surrounding the keypoint, used to describe the overall appearance features of the face.

[0053] For example, suppose we have an image containing a face. We use a face detection algorithm to detect the face region and locate several pre-defined facial key points. Then, based on filtering rules or optimization algorithms, we select target key points from these key points. For each target key point, we can obtain its two-dimensional coordinate information, histogram of oriented gradients (HARQ) features, and facial image patches as geometric modal information, texture modal information, and appearance modal information, respectively.

[0054] Reference Figure 2This diagram illustrates the relationship between facial key points, two-dimensional coordinate information, and HOG features according to an embodiment of the present invention. The preset sample facial key points can be 68 facial key points located using the Dlib library. After discarding 18 facial contour key points, the remaining 50 target facial key points are retained. For each target facial key point, the cell size is set to 8x8, and the block size is set to 16x16. The histogram of directional gradients for each cell is a 9-dimensional vector; therefore, the histogram of directional gradients for the target facial key points is a 36-dimensional vector. The two-dimensional coordinate information (X, Y) of the target facial key points represents geometric modality information, and the HOG features represent texture modality information. Facial movements not only cause changes in muscle position but also in texture; therefore, this embodiment of the present invention concatenates geometric modality information and texture modality information in the vector dimension to form a new geometric modality information.

[0055] This invention, by acquiring geometric modality information, texture modality information, and appearance modality information, can fully utilize multiple information sources to improve the accuracy and stability of emotion state recognition. Different modality information can provide rich facial feature representations, helping to more comprehensively describe facial emotional states. It can be applied to different emotion state recognition scenarios, possessing a certain degree of flexibility and versatility.

[0056] In an exemplary embodiment of the present invention, one implementation method for constructing an emotional state analysis model based on multimodal pre-training and fine-tuning using geometric modal information, texture modal information, and appearance modal information involves creating a graph convolutional network, a transformer network, and an activation function, respectively. The emotional state analysis model is then constructed based on the graph convolutional network, transformer network, and activation function. Specifically, the graph convolutional network is used to extract geometric modal encoding features from the geometric and texture modal information. The transformer network is used to encode the appearance modal information to obtain appearance modal encoding features. The transformer network is used to fuse the geometric and appearance modal encoding features to obtain a query matrix, a key matrix, and a value matrix. The activation function is used to classify the query matrix, key matrix, and value matrix to obtain a classification result. The core of this implementation method is to use graph convolutional networks, transformer networks, and activation functions to extract and fuse features from geometric, texture, and appearance modal information, thereby constructing an emotional state analysis model based on multimodal pre-training and fine-tuning.

[0057] In practical applications, graph convolutional networks (GCNNs) are used to extract features of geometric and textural modalities. GCNNs are neural networks suitable for graph-structured data and can effectively process relational data. Here, the two-dimensional coordinates of facial key points are used as node features. A graph structure is constructed using the relationships between facial key points, and then the GCNN extracts features of geometric and textural modalities. A transformer network is used to encode appearance modalities. The transformer network is a powerful attention mechanism model suitable for processing sequential data. Here, facial image patches are used as input sequences, and the transformer network encodes appearance modalities to obtain appearance modal encoded features. The transformer network is then used to fuse the geometric and appearance modal encoded features. By inputting the geometric and appearance modal encoded features into the transformer network, the attention mechanism is used to fuse and interact the two types of information, resulting in a query matrix, key matrix, and value matrix. An activation function is then used to classify the query matrix, key matrix, and value matrix to obtain the emotional state classification result. The activation function maps the query matrix, key matrix, and value matrix to the classification result space to achieve emotional state recognition.

[0058] For example, suppose we have an image containing a face. We use a graph convolutional network to extract geometric and texture modal features, and a transformer network to encode the appearance modal features. Next, the geometric and appearance modal features are input into the transformer network for fusion processing to obtain the final feature representation. Finally, an activation function is used to classify the feature representation to obtain the predicted sentiment state.

[0059] Reference Figure 3 The diagram shows a schematic representation of a graph convolutional network according to an embodiment of the present invention.

[0060] In this embodiment of the invention, the graph convolutional network is used to extract facial expression information contained in the geometric modality information. The geometric modality features obtained by concatenating facial keypoints and corresponding HOG features are input into a 10-layer GCN. For each GCN layer, the input features... The following calculations were performed to obtain

[0061] ①、

[0062] ②、f' in =conv(f in (M k +A k ))+conv(f in )

[0063] ③、f out=fc2(fc1(f') in ))

[0064] Where C in N is the dimension of the input features. in A is the number of input features. k It is a learnable adjacency matrix, whose initial adjacency is formed by sequentially connecting facial keypoints from index 0 to index 49. `conv` refers to a 1×1 convolution, and `fc1` converts `f'` into a single convolution. in Dimensional transformation (Transpose) fc2 transforms its dimensions into

[0065] In this embodiment of the invention, before inputting the geometric modal features into the GCN, an additional feature is introduced. This learnable vector, concatenated with geometric modality features, yields... Input features.

[0066] The input dimensions of a 10-layer GCN are [38×51, 64×51, 64×51, 128×51, 128×51, 256×51, 256×51, 512×51, 512×51, 768×51]. After obtaining the final output, it is transposed. The output vector.

[0067] It is worth mentioning that the learnable adjacency matrix, through continuous iterative optimization during the pre-training and fine-tuning tasks of the entire network, can ultimately express the connectivity relationships and importance of facial geometric features.

[0068] Reference Figure 4 The diagram shows a schematic representation of a transformer network according to an embodiment of the present invention.

[0069] For appearance modal features, an 8-layer transformer network is used for feature encoding. The initial input size is 50 bytes. The facial image blocks are first encoded into A two-dimensional image patch sequence, where 50 is the number of image patches and 768 is the length of the one-dimensional image patch vector. Furthermore, [the following is introduced]. This learnable vector, concatenated with the image patch sequence, yields... As input to multi-head self-attention blocks, a learnable one-dimensional positional code is added to the image block sequence to represent the positional information of different image blocks.

[0070] Three fully connected layers are used to process the input x. pWe calculate q, k, and v, which are the query matrix, key matrix, and value matrix, and then use the formula to calculate the output after self-attention tradeoff: Where d k Let k be the dimension of the k-sequence.

[0071] Eight heads are used to calculate the self-attention feature sequence separately. Then, the feature sequences calculated by each head are concatenated and linearly mapped to obtain a feature map of the same size as the input. The calculation formula is as follows:

[0072] O final =concat(O h1 O h2 ,…,O h8 W H

[0073] Finally, residual connections are performed on the output, and LayerNorm(LN) is used to adjust the O final Perform a normalization operation. The input to each layer can be represented as x. l The output of each layer can be represented as x l+1 .

[0074] This invention utilizes graph convolutional networks, transformer networks, and activation functions to fuse and process multimodal information, fully leveraging different modalities to improve the accuracy and robustness of emotion state recognition. It is applicable to various emotion state recognition scenarios, exhibiting flexibility and versatility. Furthermore, the transformer network can process sequential data, making it suitable for handling sequential information such as facial image patches. By encoding and fusing features using graph convolutional networks and transformer networks, the expressive power of features can be improved, better describing the geometric, textural, and appearance features of the face.

[0075] The MPF model in this embodiment of the invention mainly includes a feature encoding module, a multimodal fusion module, and a classification module. The feature encoding model comprises two parts: a graph convolutional network and a transformer network; the multimodal fusion module is composed of a transformer network; and the classification model is composed of an activation (softmax) function.

[0076] Reference Figure 5 The diagram shows a structural schematic of a multimodal fusion module according to an embodiment of the present invention.

[0077] For the multimodal fusion module, this embodiment of the invention uses a 4-layer transformer network for feature encoding. The outputs of the geometric feature encoding module and the appearance feature encoding module are also considered. and Two more modal vectors are introduced, respectively with... and They are added together to distinguish between the two modalities.

[0078] For these two modality vectors, the embodiments of the present invention pass them through three fully connected layers respectively to obtain [q G k G v G ] and [q A k A v A Then, let q G and k A v A Perform multi-head self-attention calculation, then pass through one residual connection layer, one LayerNorm layer, two fully connected layers, one residual connection layer, and one LayerNorm layer. A and k G v G Perform self-attention calculations, and subsequent calculations are as before. Among them, The input represents the geometric modality coding features of each layer. This represents the output of the geometric modality coding features for each layer. The input represents the appearance modality coding feature of each layer. This represents the output of the appearance modality coding features for each layer.

[0079] In one exemplary embodiment of the present invention, one method for training the sentiment state analysis model involves setting up a geometric appearance matching task, a mask geometry reconstruction task, and a mask appearance reconstruction task. The sentiment state analysis model is pre-trained based on these tasks. The pre-trained model is then fine-tuned using an expression dataset. The core of this embodiment is to pre-train the sentiment state analysis model using these tasks and then fine-tuning it using an expression dataset.

[0080] In practical applications, three tasks are set up: geometric appearance matching, masked geometric reconstruction, and masked appearance reconstruction. The geometric appearance matching task aims to determine whether the geometric modality and appearance modality match. This is done by randomly replacing image patches corresponding to facial key points with other image patches, and then using the vector at position 0 of the geometric modality features output by the model to determine if a match exists. The masked geometric reconstruction task aims to predict the masked key point information using geometric modality features, while the masked appearance reconstruction task predicts the pixel information of the masked facial image patches using appearance modality features. The sentiment analysis model is pre-trained based on these tasks. During pre-training, the model needs to learn the correlation between geometric modality, texture modality, and appearance modality, and be able to accurately reconstruct the masked information. This pre-training task design allows the model to be fully trained in understanding and fusing multimodal information. The pre-trained sentiment analysis model is then fine-tuned using an expression dataset. During fine-tuning, the model learns from samples in the expression dataset to further optimize model parameters, resulting in better performance in sentiment recognition. Fine-tuning improves the model's generalization ability and accuracy in practical applications.

[0081] For example, suppose we have an emotion state analysis model. First, we pre-train the model, setting up tasks such as geometric appearance matching, mask geometry reconstruction, and mask appearance reconstruction. During pre-training, the model needs to learn the relationships between different modal information and accurately reconstruct the masked information. Then, we fine-tune the pre-trained model using an facial expression dataset to improve its performance in emotion state recognition.

[0082] This invention, through pre-training with multiple tasks, enables the model to fully learn the correlations between multimodal information, improving the accuracy and robustness of emotion state recognition. Task setting and optimization during pre-training and fine-tuning enhance the model's generalization ability, allowing it to perform well across different scenarios and datasets. Through pre-training and fine-tuning, the model learns more stable and reliable feature representations, contributing to improved stability and reliability of emotion state recognition.

[0083] In an exemplary embodiment of the present invention, the geometric appearance matching task includes: replacing facial image patches corresponding to multiple target facial key points with other facial image patches corresponding to other target facial key points according to a first probability; and replacing the target facial key points and histogram features of oriented gradients corresponding to multiple facial image patches with other target facial key points and histogram features of oriented gradients corresponding to other facial image patches according to a first probability. Based on the vector at position 0 of the geometric modality encoding features and appearance modality encoding features, it is determined whether the geometric modality information and texture modality information match the appearance modality information. The loss function for the geometric appearance matching task is the cross-entropy loss function.

[0084] In practical applications, the geometric appearance matching task aims to determine whether geometric modality information and texture modality information match appearance modality information. Specifically, this involves replacing facial image patches corresponding to multiple target facial keypoints with other facial image patches corresponding to other target facial keypoints, and replacing the target facial keypoints and histogram features of the orientation gradients (HAR) corresponding to multiple facial image patches with other target facial keypoints and other HAR features corresponding to other facial image patches. Then, using the vectors at the 0th position of the geometric modality encoding features and appearance modality encoding features, a cross-entropy loss function is applied to determine whether the geometric modality information and texture modality information match the appearance modality information.

[0085] For example, suppose we have a sentiment analysis model that is pre-trained on a geometric appearance matching task. During pre-training, the model needs to learn the matching relationships between different modalities, i.e., whether geometric and texture modalities match appearance modalities. By setting task objectives and loss functions, the model can be progressively optimized, enabling it to more accurately determine the matching relationships between different modalities in practical applications, thereby improving the accuracy and robustness of sentiment recognition.

[0086] This invention, through a geometric appearance matching task, enables the model to learn matching relationships between different modalities, thereby improving the accuracy of emotion state recognition. The geometric appearance matching task has a clear and explicit objective, helping the model better learn the feature representations required for the task and improving the performance of emotion state recognition. Using the cross-entropy loss function effectively evaluates the difference between the model output and the real situation, helping the model converge faster during training and improving training efficiency and effectiveness.

[0087] In an exemplary embodiment of the present invention, the masked geometric reconstruction task includes: masking multiple target facial key points according to a second probability; and predicting the masked multiple target facial key points using a fully connected layer with geometric modality encoding features. The loss function for the masked geometric reconstruction task is the mean squared error loss function. The masked geometric reconstruction task is an important implementation method in emotion state recognition schemes. Its main purpose is to improve the model's understanding and representation ability of facial geometric modalities by masking part of the facial key point information and then using features learned by the model to predict the masked facial key point information.

[0088] In practical applications, the specific operation of the masked geometric reconstruction task involves masking multiple facial key points with a certain probability, i.e., hiding some key point information. Then, using the geometric modality encoding features learned by the model, a fully connected layer is used to predict the masked facial key points. In this way, the model can infer the location of the masked key points based on existing information, thereby achieving the reconstruction of geometric modality information.

[0089] For example, suppose we have an emotion state recognition model that is pre-trained for a masked geometry reconstruction task. During pre-training, the model needs to learn how to predict the locations of masked keypoints based on known geometric modal information. By continuously adjusting the model parameters, we can minimize the mean squared error loss during the prediction process, thereby improving the model's ability to understand and represent geometric modal information.

[0090] This invention, through a masked geometric reconstruction task, enables the model to learn how to predict the locations of occluded key points based on known geometric modal information, thereby improving its understanding and representation of geometric modal information. Training through this task allows the model to better adapt to different facial key point occlusion scenarios, enhancing its generalization ability and making it more robust in practical applications. Using the mean squared error loss function effectively assesses the difference between the model's predictions and the actual situation, helping the model converge faster during training and improving training efficiency and effectiveness.

[0091] In an exemplary embodiment of the present invention, the masked appearance reconstruction task includes: masking multiple facial image patches according to a third probability; and predicting the masked multiple facial image patches using appearance modality encoding features connected to a fully connected layer. The loss function for the masked appearance reconstruction task is the mean squared error loss function. The masked appearance reconstruction task is an important implementation method in emotion state recognition schemes. Its main purpose is to improve the model's understanding and representation ability of appearance modalities by masking parts of facial image patches and then using features learned by the model to predict the pixel information of the masked parts of the facial image patches.

[0092] In practical applications, the specific operations of the masked appearance reconstruction task include partially occluding facial image patches with a certain probability, i.e., hiding some image information. Then, using the appearance modality coding features already learned by the model, a fully connected layer is added to predict the pixel information of the occluded facial image patches. In this way, the model can infer the pixel information of the occluded image patches based on existing information, thereby achieving the reconstruction of appearance modality information.

[0093] For example, suppose we have an emotion state recognition model that is pre-trained for a masked appearance reconstruction task. During pre-training, the model needs to learn how to predict the pixel information of masked image patches based on known appearance modal information. By continuously adjusting the model parameters, we can minimize the mean squared error loss during the prediction process, thereby improving the model's ability to understand and represent appearance modal information.

[0094] This invention, through a mask appearance reconstruction task, enables the model to learn how to predict the pixel information of occluded image patches based on known appearance modality information, thereby improving its understanding and representation of appearance modality information. Training through the mask appearance reconstruction task allows the model to better adapt to different image patch occlusion situations, improving its generalization ability and making it more robust in practical applications. Using the mean squared error loss function effectively evaluates the difference between the model's predictions and the actual situation, helping the model converge faster during training and improving training efficiency and effectiveness.

[0095] Reference Figure 6 The diagram illustrates the training process of an emotion state recognition model according to an embodiment of the present invention.

[0096] The geometric modality is input into the emotion recognition model to obtain geometric feature encoding, while the appearance modality is input to the emotion recognition model to obtain appearance feature encoding. The geometric feature encoding, along with the modality type encoding and learnable parameter encoding, is input into the Transformer Encoder. The appearance feature encoding, along with the modality type encoding and learnable parameter encoding, is also input into the Transformer Encoder. The fully connected layer (FC) of the Transformer Encoder determines whether the geometric modality and appearance modality match (True). The multilayer perceptron (MLP) of the Transformer Encoder predicts keypoint location information, such as coordinates (10, 20, 30). The multilayer perceptron (MLP) of the Transformer Encoder predicts keypoint appearance information.

[0097] The aforementioned geometric appearance matching task involves randomly replacing image patches (facial keypoints and HOG features) corresponding to facial keypoints (facial image patches) with other image patches (facial keypoints and HOG features) with a probability of 0.5. Then, the vector at position 0 of the geometric modality feature (appearance modality feature) output by multimodal feature fusion is mapped to a binary logits (a neural network prediction value representing the confidence level of the input data), used to determine whether the geometric modality and appearance modality match. This task uses the cross-entropy loss function loss1.

[0098] The aforementioned mask appearance reconstruction task aims to predict the location information of masked facial key points using geometric modal features obtained through multimodal fusion. Key points are randomly masked with a probability of 0.15, and then the final geometric modal features are connected to a fully connected layer to predict the masked key point information. This task uses the mean squared error loss function loss².

[0099] The aforementioned mask appearance reconstruction task aims to predict the pixel information of masked facial image patches using appearance modal features obtained through multimodal fusion. Facial image patches are randomly masked with a probability of 0.15, and then the final appearance modal features are connected to a fully connected layer to predict the masked pixel information. This task uses the mean squared error loss function loss3.

[0100] The loss function used throughout the pre-training process is: loss = loss1 + loss2 + loss3.

[0101] In this embodiment of the invention, a dataset (such as MS-Celeb-1M) is used for pre-training, and an optimization algorithm (such as AdamW) is used. The minimum batch size for training is 64, the iteration period is 200 rounds, the initial learning rate is 0.0005, and the learning rate decays using a cosine annealing strategy, decaying once every 30 rounds with a decay rate of 0.05.

[0102] For the pre-trained MPF model, fine-tuning is performed using the facial expression dataset alignment. The loss function used for fine-tuning is cross-entropy loss, and other settings remain consistent with the pre-training settings.

[0103] Reference Figure 7 The diagram illustrates a comparison of the application effects of an emotion state recognition model according to an embodiment of the present invention with other models.

[0104] To verify the model's effectiveness, this embodiment of the invention sampled over 2000 users, capturing a total of 20,000 facial images. Emotional states were categorized into seven types: happy, angry, neutral, surprised, disgusted, afraid, and sad. Each of the 20,000 facial images was labeled according to these seven expression categories, with 16,000 images selected as the training set and 4,000 as the test set. The test data was then input into the model for validation. The results showed that the MPF model significantly improved accuracy, recall, and F-measure compared to other models (such as ResNet50, RegNetY 16GF, ViT-B / 16, and DeiT-B).

[0105] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0106] Reference Figure 8 The diagram illustrates a structural block diagram of an emotion state recognition system based on multimodal pre-training and fine-tuning according to an embodiment of the present invention. This emotion state recognition system based on multimodal pre-training and fine-tuning may specifically include the following modules.

[0107] The information acquisition module 81 is used to acquire geometric modal information, texture modal information and appearance modal information of multiple facial key points;

[0108] The model building module 82 is used to build an emotion state analysis model based on multimodal pre-training and fine-tuning according to the geometric modal information, texture modal information and appearance modal information;

[0109] The model training module 83 is used to train the emotional state analysis model;

[0110] The emotion prediction module 84 is used to predict facial emotion states using the trained emotion state analysis model.

[0111] In an exemplary embodiment of the present invention, the information acquisition module 81 includes:

[0112] The key point filtering module is used to filter out multiple target facial key points from a set of preset sample facial key points.

[0113] The information extraction module is used to obtain the two-dimensional coordinate information, directional gradient histogram features, and facial image patches of multiple target facial key points;

[0114] The information determination module is used to use the two-dimensional coordinate information as the geometric modality information, the directional gradient histogram feature as the texture modality information, and the facial image block as the appearance modality information.

[0115] In an exemplary embodiment of the present invention, the model building module 82 includes:

[0116] The network function creation module is used to create graph convolutional networks, transformer networks, and activation functions, respectively.

[0117] The sentiment analysis module is used to build the sentiment state analysis model based on the graph convolutional network, the transformer network, and the activation function.

[0118] The feature extraction module is used to extract geometric modal encoding features of the geometric modal information and the texture modal information using the graph convolutional network; and to perform feature encoding on the appearance modal information using the transformer network to obtain appearance modal encoding features;

[0119] The feature fusion module is used to fuse the geometric mode coding features and the appearance mode coding features using the transformer network to obtain a query matrix, a key matrix, and a value matrix;

[0120] The matrix classification module is used to classify the query matrix, key matrix, and value matrix using the activation function to obtain the classification result.

[0121] In an exemplary embodiment of the present invention, the model training module 83 includes:

[0122] The task settings module is used to set up geometric appearance matching tasks, mask geometry reconstruction tasks, and mask appearance reconstruction tasks.

[0123] The model pre-training module is used to pre-train the sentiment state analysis model based on the geometric appearance matching task, the mask geometric reconstruction task, and the mask appearance reconstruction task.

[0124] The model fine-tuning module is used to fine-tune the pre-trained emotion state analysis model using the expression dataset.

[0125] In one exemplary embodiment of the present invention, the geometric appearance matching task includes:

[0126] The replacement module is used to replace the facial image blocks corresponding to multiple target facial key points with other facial image blocks corresponding to other target facial key points according to a first probability, and to replace the target facial key points and the orientation gradient histogram features corresponding to multiple facial image blocks with other target facial key points and other orientation gradient histogram features corresponding to other facial image blocks according to the first probability.

[0127] The matching and judgment module is used to determine whether the geometric modality information and the texture modality information match the appearance modality information based on the vector at the 0th position of the geometric modality coding feature and the appearance modality coding feature.

[0128] The loss function for the geometric appearance matching task is the cross-entropy loss function.

[0129] In one exemplary embodiment of the present invention, the mask geometry reconstruction task includes:

[0130] A geometric occlusion module is used to occlude multiple key facial features of the target according to a second probability;

[0131] A geometric prediction module is used to predict multiple occluded facial key points of the target by using the geometric modality encoding features connected to a fully connected layer;

[0132] The loss function for the mask geometry reconstruction task is the mean squared error loss function.

[0133] In one exemplary embodiment of the present invention, the mask appearance reconstruction task includes:

[0134] An appearance masking module is used to mask multiple facial image blocks according to a third probability;

[0135] An appearance prediction module is used to predict multiple occluded facial image patches by connecting a fully connected layer with the appearance modality coding features.

[0136] The loss function for the mask appearance reconstruction task is the mean squared error loss function.

[0137] As the system implementation is basically similar to the method implementation, it is described in a relatively simple way. For relevant details, please refer to the description of the method implementation.

[0138] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0139] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0141] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0143] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0144] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0145] The foregoing has provided a detailed description of an emotion state recognition method and a emotion state recognition system based on multimodal pre-training and fine-tuning provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for emotion state recognition based on multimodal pre-training and fine-tuning, characterized in that, The method includes: Obtain geometric modal information, texture modal information, and appearance modal information of multiple facial key points; Based on the geometric modal information, the texture modal information, and the appearance modal information, an emotion state analysis model based on multimodal pre-training and fine-tuning is constructed; The emotional state analysis model is trained. Predict facial emotional states using the trained emotional state analysis model; The step of building an emotion state analysis model based on multimodal pre-training and fine-tuning according to the geometric modal information, the texture modal information, and the appearance modal information includes: Create graph convolutional networks, transformer networks, and activation functions respectively; The sentiment state analysis model is constructed based on the graph convolutional network, the transformer network, and the activation function. The graph convolutional network is used to extract the geometric modality coding features of the geometric modality information and the texture modality information; The transformer network is used to perform feature encoding on the appearance modal information to obtain appearance modal encoded features; The transformer network is also used to fuse the geometric mode coding features and the appearance mode coding features to obtain a query matrix, a key matrix, and a value matrix; The activation function is used to classify the query matrix, key matrix, and value matrix to obtain classification results; Training the emotional state analysis model includes: The system includes a geometric appearance matching task, a mask geometric reconstruction task, and a mask appearance reconstruction task. The geometric appearance matching task determines whether the geometric modality and the appearance modality match. The mask geometric reconstruction task predicts the masked key point information through geometric modality features. The mask appearance reconstruction task predicts the pixel information of the masked facial image block through appearance modality features. The sentiment state analysis model is pre-trained based on the geometric appearance matching task, the mask geometric reconstruction task, and the mask appearance reconstruction task; The pre-trained emotion state analysis model was fine-tuned using an expression dataset.

2. The method according to claim 1, characterized in that, The acquisition of geometric modal information, texture modal information, and appearance modal information of multiple facial key points includes: Multiple target facial key points are selected from a set of preset sample facial key points. Two-dimensional coordinate information, oriented gradient histogram features, and facial image patches of multiple target facial key points are obtained; The two-dimensional coordinate information is used as the geometric modality information, the directional gradient histogram feature is used as the texture modality information, and the facial image patch is used as the appearance modality information.

3. The method according to claim 2, characterized in that, The geometric appearance matching task includes: According to a first probability, the facial image blocks corresponding to multiple target facial key points are replaced with other facial image blocks corresponding to other target facial key points, and the target facial key points and the orientation gradient histogram features corresponding to multiple facial image blocks are replaced with other target facial key points and other orientation gradient histogram features corresponding to other facial image blocks according to the first probability. Based on the vector at position 0 of the geometric modality coding feature and the appearance modality coding feature, determine whether the geometric modality information and the texture modality information match the appearance modality information; The loss function for the geometric appearance matching task is the cross-entropy loss function.

4. The method according to claim 2, characterized in that, The mask geometry reconstruction task includes: Multiple key facial features of the target are covered according to a second probability; The geometric modality encoding features are used to form a fully connected layer to predict multiple occluded facial key points of the target. The loss function for the mask geometry reconstruction task is the mean squared error loss function.

5. The method according to claim 1, characterized in that, The mask appearance reconstruction task includes: Multiple facial image blocks are covered according to a third probability; The appearance modality coding features are used to form a fully connected layer to predict multiple occluded facial image patches; The loss function for the mask appearance reconstruction task is the mean squared error loss function.

6. An emotion state recognition system based on multimodal pre-training and fine-tuning, characterized in that, The system includes: The information acquisition module is used to acquire geometric modal information, texture modal information, and appearance modal information of multiple facial key points; The model building module is used to build an emotion state analysis model based on multimodal pre-training and fine-tuning according to the geometric modal information, texture modal information and appearance modal information; The model training module is used to train the sentiment state analysis model; The emotion prediction module is used to predict facial emotion states using the trained emotion state analysis model. The model building module includes: a network function creation module for creating a graph convolutional network, a transformer network, and an activation function; a sentiment analysis module for building a sentiment state analysis model based on the graph convolutional network, the transformer network, and the activation function; a feature extraction module for extracting geometric modal encoding features of the geometric modal information and the texture modal information using the graph convolutional network; and using the transformer network to perform feature encoding on the appearance modal information to obtain appearance modal encoding features; a feature fusion module for fusing the geometric modal encoding features and the appearance modal encoding features using the transformer network to obtain a query matrix, a key matrix, and a value matrix; and a matrix classification module for classifying the query matrix, key matrix, and value matrix using the activation function to obtain classification results. The model training module includes: a task setting module for setting geometric appearance matching, mask geometric reconstruction, and mask appearance reconstruction tasks; wherein the geometric appearance matching task determines whether the geometric modality and appearance modality match; the mask geometric reconstruction task predicts masked key point information through geometric modality features; and the mask appearance reconstruction task predicts pixel information of masked facial image blocks through appearance modality features; a model pre-training module for pre-training the emotion state analysis model based on the geometric appearance matching, mask geometric reconstruction, and mask appearance reconstruction tasks; and a model fine-tuning module for fine-tuning the pre-trained emotion state analysis model using an expression dataset.

7. An electronic device, characterized in that, include: One or more processors; and One or more machine-readable media storing instructions thereon, when executed by the one or more processors, cause the electronic device to perform the emotion state recognition method based on multimodal pre-training and fine-tuning as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The stored computer program enables the processor to execute the emotional state recognition method based on multimodal pre-training and fine-tuning as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Facial expression recognition method and device

    CN107045618A

  • Expression image data processing method and device and server

    CN117079326A