A micro-expression recognition method based on multimodal fusion

Abstract

The present invention discloses a micro-expression recognition method based on multimodal fusion, which relates to intelligent micro-expression recognition, including obtaining input data of a model from a micro-expression database, extracting features from each input data using ViT and a convolutional neural network CNN, and obtaining a feature vector V a and the eigenvector V f , for the eigenvector V a and the eigenvector V f Similarity measurement is performed to obtain output features. By fusing peak frames, optical flow maps, and key point classification micro-expression data, and utilizing data features of different modalities, the problem of insufficient utilization of micro-expression data in the existing technology is solved; for data of different modalities, a network architecture is adopted: Vision Transformer (ViT) is used to extract global features of peak frames, and convolutional neural network (CNN) is used to extract local features of optical flow maps. The fused features are optimized through contrast loss and classification loss, which improves the similarity of features of the same expression sample and enhances the accuracy and robustness of recognition.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent micro-expression recognition, and in particular to a micro-expression recognition method based on multimodal fusion. Background Art

[0002] Human facial expressions can be divided into two categories: macro-expressions and micro-expressions. The main characteristics of macro-expressions are long duration, very obvious facial movements, and easy to be recognized by humans and computers. However, in contrast, micro-expressions are spontaneous, with very low facial movements, lasting from 0.065 to 0.5 seconds, and can reveal a person's true emotional state. However, it is difficult for people who have not received long-term training in this area to recognize changes in expressions. Micro-expressions are widely used in lie detection, education, security and other fields.

[0003] Micro-expression recognition mainly includes feature extraction and emotion classification. Feature extraction is a key step, mainly based on traditional feature extraction and feature representation based on deep learning. At present, deep learning methods play an important role in the field of micro-expressions. Convolutional neural networks (CNN) and visual transformers (ViT) are powerful feature extraction tools. Many studies use CNN architecture as the backbone network for learning representation. In addition, some micro-expression recognition methods based on ViT have also been studied to show its ability to capture global features. Due to the short-term, weak and local nature of micro-expression data itself, relying solely on a single modality for recognition often cannot meet the requirements of accuracy and robustness. Summary of the invention

[0004] The purpose of the present invention is to provide a micro-expression recognition method based on multimodal fusion, which solves the technical problem that micro-expression data cannot be fully utilized in the prior art.

[0005] The technical solution of the present invention is: a micro-expression recognition method based on multimodal fusion, comprising the following steps:

[0006] Step 1: Obtain the input data of the model from the micro-expression database;

[0007] Step 2: Use ViT and convolutional neural network CNN to extract features for each input data to obtain feature vectors and the eigenvector ;

[0008] Step 3: Eigenvector and the eigenvector Perform similarity measurement and obtain output features;

[0009] Step 4: Fuse the output features at the feature level and decision level, and output the category of micro-expression;

[0010] Step 1 includes: Step 1.1, using the face recognition toolkit Dlib to perform face cropping and alignment operations on the micro-expression database, delete the background and other irrelevant information, and obtain an aligned facial sequence;

[0011] Step 1.2: Peak frame Acquisition: Acquisition of peak frames of CASME II, SAMM and SMIC-HS databases , CASME II and SAMM databases for peak frames Marking is performed. The SMIC-HS database does not have marked peak frames. , the peak frame is replaced by the intermediate frame of the micro-expression sequence ;

[0012] Step 1.3, obtaining the optical flow map: use TV-L1 to extract the optical flow between the start frame and the peak frame;

[0013] Step 1.4: Key points Acquisition: Select key points in the area related to micro-expressions ;

[0014] Step 2 includes:

[0015] Step 2.1: Peak frame Input into ViT to get a series of feature vectors with position information , then Input into the Transformer encoder;

[0016] Step 2.2: After encoding by 3 layers of Transformer encoder, we get The features of The eigenvector of ;

[0017] Step 2.3: Input the optical flow map into the convolutional neural network CNN, and then use The maximum pooling layer of The eigenvector of ;

[0018] Step 2.4: For each key point , using the key point coordinates of the starting frame image and the key point coordinates of the peak frame image The displacement between is calculated as follows:

[0019] ;

[0020] in, represents the difference in the horizontal axis, Represents the difference in ordinates, using the eigenvector constructed using displacement , and then input into two fully connected layers, and finally output the optimized selection result .

[0021] Furthermore, in step 3, the feature vector obtained for the same micro-expression sample and , measured using L2 normalized cosine similarity:

[0022] ;

[0023] in, Indicates the similarity between the two.

[0024] Further, step 4 includes:

[0025] Step 4.1: In classification fusion, the feature vector obtained in step 2 and The fusion is performed by convex combination, and the fusion method is as follows: ;

[0026] in, is the feature vector and the eigenvector The fused feature vector, is a learnable weight parameter used for convex combination operation fusion;

[0027] Step 4.2: It is sent to two fully connected layers. The number of fully connected outputs in the last layer is 3, and the feature fusion classification result is obtained. ;

[0028] Step 4.3: Fusion of features and classification results And key point classification results After decision-level fusion, the overall classification results are as follows:

[0029] ;

[0030] in, is the overall classification result, It is the weight parameter that controls the classification results of the feature fusion branch and the key point branch. By adjusting these two weight parameters, the model can dynamically optimize the overall performance and adapt to different task requirements according to the characteristics of the data.

[0031] Furthermore, it also includes the setting of model parameters, and the setting of model parameters includes data sets;

[0032] The CASME II, SAMM, and SMIC-HS datasets were classified into negative, positive, and surprise;

[0033] MER-MFNet performs end-to-end training for multiple iterations, with AdamW as the optimizer;

[0034] MER-MFNet is a micro-expression recognition network based on multimodal fusion.

[0035] Furthermore, the contrast loss function is set as: , so that the characteristics of the two modes are closer.

[0036] Furthermore, the model is optimized using end-to-end joint training according to the contrastive loss function And the classification loss function , classification loss function Use the cross entropy loss function for calculation, the overall loss function The calculation results are as follows:

[0037] ;

[0038] Among them, m is the function that controls the contrast loss In the overall loss function The contribution parameter in is the overall loss function.

[0039] By adopting the above technical solution, the present invention has the following beneficial effects:

[0040] By fusing peak frames, optical flow maps, and key point classification micro-expression data, the data features of different modalities are fully utilized to solve the problem of insufficient utilization of micro-expression data in existing technologies; for data of different modalities, a specially designed network architecture is used: Vision Transformer (ViT) is used to extract global features of peak frames, convolutional neural network (CNN) is used to extract local features of optical flow maps, and geometric analysis is used to extract local features of key points. The fused features are optimized through contrast loss and classification loss, which effectively improves the similarity of features of the same expression sample, enhances the accuracy and robustness of recognition, and performs well in micro-expression recognition in complex scenes, with broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are part of this application and are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an improper limitation of the present invention. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0042] Figure 1 The overall framework diagram of the micro-expression recognition method based on multimodal fusion provided in this embodiment of the present application;

[0043] Figure 2 This is a structural diagram of the CNN provided in this embodiment of the present application.

[0044] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0045] The specific implementation modes of the present invention are further described in detail with reference to the accompanying drawings.

[0046] See also Figure 1 to Figure 2 As shown, the embodiment of the present application provides a micro-expression recognition method based on multimodal fusion, comprising the following steps:

[0047] Step 1. Get the input data of the model from the micro-expression database. Before that, you need to determine the experimental data, training framework, number of training rounds and optimizer selection. The database includes the Chinese Academy of Sciences Micro-Expression Database II (CASME II), the Spontaneous Micro-Facial Movement Dataset (SAMM) and the Spontaneous Micro-expression Database-High Speedcamera (SMIC-HS).

[0048] Step 2: Use ViT and convolutional neural network CNN to extract features for each input data to obtain feature vectors and the eigenvector ;

[0049] Step 3: Eigenvector and the eigenvector Perform similarity measurement and obtain output features;

[0050] Step 4: Fuse the output features at the feature level and decision level, and output the category of micro-expression.

[0051] In the above scheme, a suitable feature extraction network is selected for feature extraction according to the characteristics of each input data itself, so as to better obtain discriminative features, especially in capturing global information, local features and subtle changes, thereby improving the accuracy of micro-expression recognition and making up for the shortcomings of existing technologies in insufficient data utilization, thereby greatly improving the accuracy and robustness of recognition, especially achieving more accurate sentiment analysis in complex scenarios.

[0052] By fusing peak frames, optical flow maps and key point classification micro-expression data, the data features of different modalities are fully utilized, which solves the problem of insufficient utilization of micro-expression data in existing technologies. For data of different modalities, a specially designed network architecture is adopted: Vision Transformer (ViT) is used to extract the global features of peak frames, convolutional neural networks (CNN) are used to extract local features of optical flow maps, and geometric analysis is used to extract local features of key points. The fused features are optimized through contrast loss and classification loss, which effectively improves the similarity of features of the same expression samples and enhances the accuracy and robustness of recognition.

[0053] Step 1 includes:

[0054] Step 1.1: Use the face recognition toolkit Dlib to perform face cropping and alignment operations on the micro-expression database, delete the background and other irrelevant information, and obtain the aligned facial sequence;

[0055] Step 1.2: Peak frame Acquisition: Acquisition of peak frames of CASME II, SAMM and SMIC-HS databases , CASME II and SAMM databases for peak frames Marking is performed. The SMIC-HS database does not have marked peak frames. , the peak frame is replaced by the intermediate frame of the micro-expression sequence ;

[0056] Step 1.3, obtaining the optical flow map: Use TV-L1 (a method for extracting the optical flow between the start frame and the peak frame in the video) to extract the optical flow between the start frame and the peak frame;

[0057] Step 1.4: Key points Micro-expressions mostly occur in the eyebrows, eyes and mouth areas of the face, so at 68 key points Select 42 points related to this area.

[0058] Step 2 includes:

[0059] Step 2.1: Peak frame Input into ViT to get a series of feature vectors with position information , then Input into the Transformer encoder, specifically Divided into 49 pieces of size Each image block is flattened into a 48-dimensional vector , and then these vectors are mapped through a trainable linear projection layer to obtain , and add a learnable position embedding , get the feature vector with position information ;

[0060] Step 2.2, the Transformer encoder consists of multiple Transformer layers, each of which consists of two normalization layers, a multi-head attention layer and a feedforward neural network, and a residual connection is used between the two. After encoding by three layers of Transformer encoder, we get The features of The eigenvector of ;

[0061] Step 2.3: Input the optical flow map into the convolutional neural network CNN. The convolutional neural network CNN uses the structure in Inception, uses two Inception blocks, and then uses The maximum pooling layer of The eigenvector of ;

[0062] Step 2.4: Use 42 key points related to micro-expressions , for each key point , using the key point coordinates of the starting frame image and the key point coordinates of the peak frame image The displacement between is calculated as follows:

[0063] ;

[0064] in, represents the difference in the horizontal axis, Represents the difference in ordinates, using the eigenvector constructed using displacement , and then input into two fully connected layers, and finally output the optimized selection result .

[0065] It should be noted that if the number of input data samples is small, other Transformer variants and CNNs can also be used in the feature extraction network architecture, and shallow ones should be selected.

[0066] In step 3, the feature vector obtained for the same micro-expression sample and the eigenvector , measured using L2 normalized cosine similarity:

[0067] ;

[0068] in, Indicates the similarity between the two.

[0069] Step 4 includes:

[0070] Step 4.1: In classification fusion, the feature vector obtained in step 2 and the eigenvector The fusion is performed by convex combination, and the fusion method is as follows: ;

[0071] in, is the feature vector and the eigenvector The fused feature vector, is a learnable weight parameter used for convex combination operation fusion;

[0072] Step 4.2: It is sent to two fully connected layers. The number of fully connected outputs in the last layer is 3, and the feature fusion classification result is obtained. ;

[0073] Step 4.3: Fusion of features and classification results And key point classification results After decision-level fusion, the overall classification results are as follows:

[0074] ;

[0075] in, is the overall classification result, It is the weight parameter that controls the classification results of the feature fusion branch and the key point branch. By adjusting these two weight parameters, the model can dynamically optimize the overall performance and adapt to different task requirements according to the characteristics of the data.

[0076] The micro-expression recognition method based on multimodal fusion provided in the embodiment of the present application also includes setting of model parameters, and the setting of model parameters includes a data set;

[0077] The CASME II, SAMM, and SMIC-HS datasets are divided into negative, positive, and surprise, which is the same as the 2nd Micro-Expression Grand Challenge Composite Database (MEGC2019-CD);

[0078] MER-MFNet was trained end-to-end for 60 iterations, with AdamW as the optimizer. MER-MFNet is a Micro-Expression Recognition Based on Multimodal FusionNetwork, initialized with a learning rate of 0.001, and cosine rate scheduling was applied. The batch size was set to 128. All experiments were implemented in the Pytorch framework and accelerated on the NVIDIA RTX 4060 GPU (with 16GB of video memory).

[0079] Set the contrast loss function to: , so that the characteristics of the two modes are closer.

[0080] Optimize the model using end-to-end joint training and compare the loss function And the classification loss function , classification loss function Use the cross entropy loss function for calculation, the overall loss function The calculation results are as follows:

[0081] ;

[0082] Among them, m is the function that controls the contrast loss In the overall loss function The contribution parameter in is the overall loss function. In the setting, m takes a smaller value to facilitate the output of micro-expression categories.

[0083] Comparative experimental example:

[0084] The experimental comparison results of the unweighted F1-score (UF1) and unweighted average recall (UAR) evaluation indicators of the micro-expression recognition method based on multimodal fusion network (MER-MFNet) provided in the embodiment of the present application with other methods on the MEGC2019-CD, CASMEII, SMIC-HS and SAMM databases are shown in Table 1.

[0085] Table 1: Experimental comparison of MER-MFNet with other methods on composite database

[0086]

[0087] It can be seen from Table 1 that the MER-MFNet model has achieved better recognition results. It can be observed that compared with the hand-crafted method LBP-TOP, MER-MFNet has improved UF1 and UAR by 0.119 and 0.129 on the composite dataset MEGC2019-CD, while maintaining an overall advantage in single dataset verification. In addition, the MER-MFNet method is superior to other methods. It is worth noting that compared with the Dual-ATME method, the evaluation indicators of UF1 and UAR have improved on the CASME II, SMIC-HS, SAMM and MEGC2019-CD datasets, which are higher than 0.065 / 0.074, 0.029 / 0.017, 0.007 / 0.021 and 0.028 / 0.027, respectively. The micro-expression recognition method based on multimodal fusion provided in the embodiment of the present application has the characteristics of extracting global vertex frames and local optical flow maps, and also integrates the geometric features of micro-expression key points. However, compared to STSTNet, the MER-MFNet model achieves slightly lower performance in MEGC2019-CD and CASME II.

[0088] Ablation experiment verification:

[0089] The rationality of each part of MER-MFNet is analyzed and verified. Ablation experiments are conducted on the MEGC2019-CD dataset and the CASME II dataset using accuracy (ACC), UAR, and UF1 as evaluation indicators. Table 2 shows the selection of feature extraction networks used for peak frames (Apex) and optical flow maps (Flow).

[0090] Table 2: Ablation experiments of network architectures with different input modalities

[0091]

[0092] From Table 2, we can see that on both CASME II and MEGC2019-CD datasets, although Apex-CNN is slightly better in ACC on CASMEII, Apex-ViT performs better in all indicators (ACC, UAR, UF1) on the MEGC2019-CD dataset and provides more consistent performance between the two datasets. Considering that Apex-ViT exhibits better generalization and stability, it is reasonable to choose Apex-ViT as the basic model for further research or application. Specifically, in the MEGC2019-CD dataset, Apex-ViT improves the accuracy by about 5.6% and UAR by about 12.3% compared to Apex-CNN, and UF1 also has a similar improvement, which shows that Apex-ViT has a clear advantage in processing these two datasets, so ViT is used for peak frames. Similarly, we can analyze that the optical flow map works better with CNN.

[0093] Verification of the effectiveness of multimodal fusion:

[0094] Table 3 shows the effectiveness of the three-branch fusion, ATN means using ViT to extract peak frame features, and FCN means using CNN to extract optical flow map features. Three groups of experiments were conducted. The first group used only one branch ATN, the second group was the fusion of two branches, and the third group was the method MER-MFNet provided in the embodiment of this application.

[0095] Table 3: Effectiveness of multimodal fusion methods

[0096]

[0097] Table 3 shows that on the CASME II dataset, MER-MFNet improves ACC by 29.6% (from 0.552 to 0.848), UAR by 95.0% (from 0.423 to 0.825), and UF1 by 92.6% (from 0.431 to 0.830) compared with the single-branch ATN method. Compared with ATN+FCN, MER-MFNet's ACC, UAR, and UF1 also improve by 3.3%, 3.6%, and 3.9%, respectively. For the MEGC2019-CD dataset, the performance improvement of MER-MFNet relative to ATN is more obvious: ACC improves by 53.2% (from 0.483 to 0.740), UAR improves by 78.1% (from 0.397 to 0.707), and UF1 also improves by 76.8% (from 0.400 to 0.707). Compared with ATN+FCN, these indicators of MER-MFNet are improved by 2.6%, 3.1% and 3.1% respectively. MER-MFNet not only shows better recognition ability than ATN and ATN+FCN in both datasets, but also has significant performance improvement, especially when dealing with the more challenging MEGC2019-CD dataset, which shows that MER-MFNet has stronger data adaptability and robustness. The experimental results show the effectiveness of the fusion of three branches.

[0098] Experimental verification of different feature fusion methods:

[0099] Table 4 shows different feature fusion methods, four fusion methods: element multiplication (mul), addition (add), convex combination (convex), and combination of element-wise addition and multiplication (add and mul).

[0100] Table 4: Different feature fusion methods

[0101]

[0102] Table 4 shows that: According to the experimental data in Table 4, on the CASME II dataset, the feature fusion method using the convex method performs best, with an ACC of 0.848, which is about 3.3% higher than the add method and nearly 7% higher than the mul and Add and mul methods; UAR is 0.825, which is about 3.6% higher than the add method and about 9.3% higher than the mul and Add and mul methods; UF1 is 0.830, which is also higher than the other three methods. In the MEGC2019-CD dataset, although the add method has a slight advantage in accuracy, the convex method remains competitive in UAR and UF1, which are 0.707 respectively, slightly higher than the 0.704 of the add method. Overall, the convex method has the best comprehensive performance on the two datasets, especially in key indicators such as UAR and UF1. Therefore, convex combination is the best choice among the four feature fusion strategies.

[0103] Selection of contrast loss coefficient m:

[0104] Table 5 shows the effectiveness of choosing the coefficient m for the contrastive loss.

[0105] Table 5: Contrast loss coefficient

[0106]

[0107] Table 5 shows that: According to the data in Table 5, it can be seen that the parameter m has a significant impact on the model performance on the CASME II and MEGC2019-CD datasets. Specifically: On the CASME II dataset, when m is 0.0001, the ACC of the model reaches a maximum value of 0.848, UAR is 0.825, and UF1 is 0.830. Compared with the case of m=0, ACC is increased by about 2.1%, UAR is increased by about 2.5%, and UF1 is increased by about 2.8%. On the MEGC2019-CD dataset, the performance is also better when m=0.0001, with ACC of 0.739, UAR of 0.707, and UF1 of 0.707. Compared with m=0, ACC and UAR are increased by about 0.7%, and UF1 is increased by about 0.9%. When m=0.00001, the three performance indicators on the CASME II dataset and MEGC2019-CD data have decreased. It can be seen that a reasonable choice of m value can effectively improve the performance of the model on both datasets. Especially for the CASME II dataset, m=0.0001 is a relatively ideal parameter configuration point. Experiments show that m is optimal when it is set to 0.0001. In the parameter experiment, we set m very small because our main task is classification.

[0108] Choice of classification loss function:

[0109] Table 6 shows the results of different classification loss functions for micro-expression recognition. We compare three types of classification loss functions, namely Cross Entropy Loss (CE), Focal Loss (Focal) and Weighted Cross Entropy Loss (W-CE). The experimental results show that the weighted cross entropy loss works best.

[0110] Table 6: Selection of classification loss functions

[0111]

[0112] It can be obtained from Table 6: According to the data in Table 6, on the CASME II dataset, the model using the W-CE loss function achieved the highest accuracy of 0.848, UAR of 0.825, and UF1 of 0.830, which is significantly improved compared with other loss functions such as CE and FL; in the MEGC2019-CD dataset, W-CE also performed well. Although the improvement was relatively small, its ACC of 0.739, UAR of 0.707, and UF1 of 0.707 were still better than or equal to other loss functions. This shows that the W-CE loss function can provide better performance on both datasets.

[0113] This specific embodiment is merely an explanation of the invention and is not a limitation of the invention. After reading this specification, those skilled in the art may make non-creative modifications to this embodiment as needed. However, as long as they are within the scope of protection of the invention, they are protected by patent law.

Claims

1. A micro-expression recognition method based on multimodal fusion, characterized in that: The following steps are involved: Step 1: Obtain the input data of the model from the micro-expression database; Step 2: Use ViT and convolutional neural network CNN to extract features for each input data to obtain feature vectors and the eigenvector ; Step 3: Eigenvector and the eigenvector Perform similarity measurement and obtain output features; Step 4: Fuse the output features at the feature level and decision level, and output the category of micro-expression; The step 1 comprises: Step 1.1: Use the face recognition toolkit Dlib to perform face cropping and alignment operations on the micro-expression database, delete the background and other irrelevant information, and obtain the aligned facial sequence; Step 1.2: Peak frame Acquisition: Acquisition of peak frames of CASME II, SAMM and SMIC-HS databases , CASMEII and SAMM databases for peak frames Marking is performed. The SMIC-HS database does not have marked peak frames. , the peak frame is replaced by the intermediate frame of the micro-expression sequence ; Step 1.3, obtaining the optical flow map: use TV-L1 to extract the optical flow between the start frame and the peak frame; Step 1.4: Key points Acquisition: Select key points in the area related to micro-expressions ; The step 2 comprises: Step 2.1: Peak frame Input into ViT to get a series of feature vectors with position information , then Input into the Transformer encoder; Step 2.2: After encoding by 3 layers of Transformer encoder, we get The features of The eigenvector of ; Step 2.3: Input the optical flow map into the convolutional neural network CNN, and then use The maximum pooling layer of The eigenvector of ; Step 2.4: For each key point , using the key point coordinates of the starting frame image and the key point coordinates of the peak frame image The displacement between is calculated as follows: in, represents the difference in the horizontal axis, Represents the difference in ordinates, using the eigenvector constructed using displacement , and then input into two fully connected layers, and finally output the optimized selection result .

2. The micro-expression recognition method based on multimodal fusion according to claim 1, characterized in that: In step 3, the feature vector obtained for the same micro-expression sample and the eigenvector , measured using L2 normalized cosine similarity: ； in, Indicates the similarity between the two.

3. The micro-expression recognition method based on multimodal fusion according to claim 2, characterized in that: The step 4 comprises: Step 4.1: In classification fusion, the feature vector obtained in step 2 and the eigenvector The fusion is performed by convex combination, and the fusion method is as follows: ; in, is the feature vector and the eigenvector The fused feature vector, is a learnable weight parameter used for convex combination operation fusion; Step 4.2: It is sent to two fully connected layers. The number of fully connected outputs in the last layer is 3, and the feature fusion classification result is obtained. ; Step 4.3: Fusion of features and classification results And key point classification results After decision-level fusion, the overall classification results are as follows: ； in, is the overall classification result, It is the weight parameter that controls the classification results of the feature fusion branch and the key point branch. By adjusting these two weight parameters, the model can dynamically optimize the overall performance and adapt to different task requirements according to the characteristics of the data.

4. The micro-expression recognition method based on multimodal fusion according to claim 2, characterized in that: Also included is the setting of model parameters, wherein the setting of model parameters includes a data set; The CASME II, SAMM, and SMIC-HS datasets were classified into negative, positive, and surprise; MER-MFNet performs end-to-end training for multiple iterations, with AdamW as the optimizer; The MER-MFNet is a micro-expression recognition network based on multimodal fusion.

5. The micro-expression recognition method based on multimodal fusion according to claim 3, characterized in that: Set the contrast loss function to: , so that the characteristics of the two modes are closer.

6. The micro-expression recognition method based on multimodal fusion according to claim 5, characterized in that: The model is optimized using end-to-end joint training according to the contrastive loss function And the classification loss function , classification loss function Use the cross entropy loss function for calculation, the overall loss function The calculation results are as follows: ； Among them, m is the function that controls the contrast loss In the overall loss function The contribution parameter in is the overall loss function.

Images