Expression recognition method and system based on multi-feature fusion and triple cross attention mechanism

Abstract

The present invention discloses an expression recognition method and system based on multi-feature fusion and a three-way cross attention mechanism, comprising: obtaining an LBP feature image and a HOG feature image through a feature extraction algorithm, and obtaining an LBPHOG fused feature image through a fusion operation; inputting the original expression image and the LBPHOG fused feature image into a multi-feature extraction module, wherein the two input streams are respectively subjected to different pre-trained convolutional neural networks to extract an lbphog-landmark feature sub-block, an image feature sub-block, and an image-landmark feature sub-block; using a convolutional layer to align the dimensions of the three feature sub-blocks; expanding the word vector dimension for each feature sub-block and then splicing the three feature sub-blocks together, fusing position encoding parameters into the spliced ​​feature blocks, inputting the spliced ​​feature blocks into a three-way cross attention module, obtaining an attention matrix calculated by the three-way cross attention module; and inputting the attention matrix into an SE-Block and a fully connected layer network to obtain an expression recognition result. The present invention improves the accuracy of expression recognition under natural conditions.

Description

Technical Field

[0001] The present invention belongs to the field of image classification in computer vision, and specifically relates to an expression recognition method and system based on multi-feature fusion and triple-cross attention mechanism. Background Art

[0002] Facial expression recognition technology involves using computers to identify and judge facial expressions, thereby determining a person's emotional state. This technology has widespread applications in fields such as computer vision, human-computer interaction, and psychology. With the development of computer technology, facial expression recognition technology has also been gradually introduced into the computer field.

[0003] With the rapid development of computer hardware and software, modern facial expression recognition technology has made significant progress. Currently, there are two mainstream facial expression recognition technologies: those based on manual feature extraction and those based on deep learning. Manual feature extraction was the first method developed. It uses image processing techniques to extract features from facial images and classify them. However, this method has the disadvantages of requiring manual design of feature extraction algorithms, requiring specialized knowledge and skills, and is susceptible to factors such as lighting and facial occlusion, resulting in limited accuracy. Deep learning is currently the mainstream facial expression recognition technology. It uses deep neural networks to automatically extract image features and perform classification predictions. This method does not require manual design of feature extraction algorithms, can process large amounts of data samples, and achieves high accuracy. However, facial expressions in natural conditions are easily affected by a variety of factors. In addition to the aforementioned issues of lighting and facial occlusion, there are also issues such as head posture variations and resolution, which affect the accuracy of expression recognition. Furthermore, compared to expression datasets generated in the laboratory, the intensity of expressions in natural conditions is weaker than that in staged images, making it easier for expression categories to be confused.

[0004] To address these issues, an effective approach is to leverage multiple features extracted from expression images and deep semantic features, fusing these features to increase information and enrich the semantics. Furthermore, different features have their own unique characteristics. LBP features are robust to illumination and rotation, while landmark features can enhance the differentiation of expressions between different categories. Image features can compensate for the lack of image information in the first two. Furthermore, the edge information inherent in the LBP feature image can help the landmark extraction model obtain more and better landmark points.

[0005] However, simply fusing multiple features cannot obtain high-level semantic features. Considering that different features have their own characteristics, the three features can be complemented by the three-way cross-attention mechanism, and the accuracy of expression recognition can be further improved with the help of the structure of the multi-level three-way cross-attention module; in addition, through data enhancement, the model's robustness to local occlusion and head posture changes can be enhanced. Summary of the Invention

[0006] The main technical problem solved by this invention is that, during the training process of facial expression recognition models, some images may be partially occluded, experience changes in lighting, and have head posture issues, resulting in low recognition accuracy for some images after the model is trained. This invention provides a technical solution for facial expression recognition based on multi-feature fusion and a three-way cross attention mechanism as follows:

[0007] Expression recognition based on multi-feature fusion and triple-cross attention mechanism includes the following steps:

[0008] Step 1: Build an expression recognition neural network model, including a multi-feature input layer, a feature extraction layer, a feature fusion layer, a cross-attention layer, and an output layer;

[0009] The multi-feature input layer extracts the LBP feature image and the HOG feature image of the original expression picture through a feature extraction algorithm, and fuses the two feature images to obtain an LBPHOG fused feature image;

[0010] Preprocess the original expression image and the LBPHOG fusion feature image, input the two types of preprocessed images into the feature extraction layer to obtain the lbphog-landmark feature sub-block, image feature sub-block, and image-landmark feature sub-block;

[0011] The feature fusion layer is used to align the dimensions of the three feature sub-blocks output by the feature extraction layer. The feature dimension parameters are predefined, and the three feature sub-blocks are aligned on the feature dimension through the convolution layer. After expanding the word vector dimensions of the three feature sub-blocks, the three feature sub-blocks are spliced ​​together and the fused feature matrix is ​​obtained after position encoding.

[0012] The cross-attention layer includes a multi-level three-cross attention module. The three-cross attention module at each level is used to calculate the cross-attention matrix of each feature block after alignment and splicing. By exchanging the Q matrices between different attention matrices, the Q matrices of the other two feature sub-blocks are merged during the attention calculation of the image feature sub-block to achieve the purpose of calculating cross attention. The final three-category cross-attention matrix output by the multi-level three-cross attention module is spliced ​​again to obtain a fused cross-attention matrix.

[0013] The fused cross attention matrix is ​​subjected to feature compression extraction and a fully connected layer to obtain the final expression probability output;

[0014] Step 2: Construct a data set to train the expression recognition neural network model in step 1. During the training process, the loss function calculates the loss value between the predicted expression label and the true label.

[0015] Step 3: Use the trained expression recognition neural network model to realize expression recognition.

[0016] Furthermore, the Circle LBP algorithm is used to extract LBP feature images from the original expression images.

[0017] Furthermore, the preprocessing method is to uniformly scale and standardize the original input expression image and the fused feature image, and perform data enhancement respectively. The data enhancement method includes random horizontal flipping, random vertical flipping, random addition of Gaussian noise, and random erasing of image areas.

[0018] Furthermore, the feature extraction layer includes two convolutional neural network models, MobileFaceNet and IR50. The MobileFaceNet network model is pre-trained with the ImageNet dataset, and the IR50 network model is pre-trained with the Ms-Celeb-1M dataset. During the training process, the training parameters of MobileFaceNet are frozen, and the training parameters of IR50 are fine-tuned. The landmark features of the original expression image and the landmark features of the LBPHOG feature image are extracted through MobileFaceNet to obtain image-landmark feature sub-blocks and lbphog-landmark feature sub-blocks respectively. The image features are extracted through the IR50 model to obtain the image feature sub-block.

[0019] Furthermore, after extracting the three types of features, a dimension alignment convolution layer is used to align the three dimensions. After the dimension alignment, the average value of the first dimension of the three feature sub-blocks is extracted and expanded into the word vector dimension of the sub-block. Then, the three feature sub-blocks are spliced ​​together through a splicing operation, and position encoding parameters are added to the spliced ​​feature sub-blocks through an addition operation.

[0020] Furthermore, in the cross attention layer, the three cross attention modules at each level split the lbphog-landmark feature matrix x according to the aligned dimensions. lbp 、image feature matrix x img And the image-landmark feature matrix x lm , calculate the cross attention of the three types of feature matrices in parallel to obtain their respective cross attention matrices;

[0021] The implementation process of calculating the respective cross attention is as follows: Calculate x in the lbphog-landmark attention calculation module lbp K matrix K lbp and V matrix V lbp , and x img The Q matrix Q img , calculate x in the image attention calculation module img K matrix K img and V matrix V img , and x lbp and x lm The Q matrix Q lbp , Q lm , calculate x in the image-landmark attention calculation module lm K matrix K lm and V matrix V lm , and x img The Q matrix Q img , thus obtaining the three cross attentions output by the three cross attention modules:

[0022]

[0023]

[0024]

[0025] Among them, Attention lbp , Attention img , Attention lm are the cross attention matrices of lbphog-landmark, image, and image-landmark feature sub-blocks, Softmax is the Softmax function, and d is the scaling parameter.

[0026] Furthermore, the specific implementation method of obtaining the fused cross attention matrix is ​​as follows;

[0027] After the three-cross attention module at each level outputs three cross attention matrices, they are spliced ​​again after passing through the multi-layer perceptron and regularization layer, as well as the residual operation, and then input into the three-cross attention module of the next layer. The formula of a single cross attention module is as follows:

[0028] x′ lbp =Attention lbp +x lbp

[0029] x lbp_out =MLP(Norm(x′ lbp))+x′ lbp

[0030] x′ img =Attention img +x img

[0031] x img_out =MLP(Norm(x′ img ))+x′ img

[0032] x′ lm =Attention lm +x lm

[0033] x lm_out =MLP(Norm(x′ lm ))+x′ lm

[0034] Among them, MLP is a multi-layer perceptron, Norm is a regularization layer, and x lbp_out , x img_out , x lm_out They are the three types of cross attention matrices output by the three-cross attention module in a certain layer, and are also the input of the three-cross attention module in the next layer.

[0035] Furthermore, feature compression extraction is performed through SE-Block.

[0036] Furthermore, the loss function includes the cross entropy loss function L CE And the label smoothing loss function L LS , the formula of the cross entropy loss function is Where n represents the number of samples; the formula of the label smoothing loss function is Where y is the true value, is the predicted value, α is the smoothing parameter, and KLDivergence is the KL divergence.

[0037] The present invention also provides an expression recognition system based on multi-feature fusion and triple-cross attention mechanism, comprising the following modules:

[0038] The network model building module is used to build the expression recognition neural network model, including the multi-feature input layer, feature extraction layer, feature fusion layer, cross attention layer, and output layer;

[0039] The multi-feature input layer extracts the LBP feature image and the HOG feature image of the original expression picture through a feature extraction algorithm, and fuses the two feature images to obtain an LBPHOG fused feature image;

[0040] Preprocess the original expression image and the LBPHOG fusion feature image, input the two types of preprocessed images into the feature extraction layer to obtain the lbphog-landmark feature sub-block, image feature sub-block, and image-landmark feature sub-block;

[0041] The feature fusion layer is used to align the dimensions of the three feature sub-blocks output by the feature extraction layer. The feature dimension parameters are predefined, and the three feature sub-blocks are aligned on the feature dimension through the convolution layer. After expanding the word vector dimensions of the three feature sub-blocks, the three feature sub-blocks are spliced ​​together and the fused feature matrix is ​​obtained after position encoding.

[0042] The cross-attention layer includes a multi-level three-cross attention module. The three-cross attention module at each level is used to calculate the cross-attention matrix of each feature block after alignment and splicing. By exchanging the Q matrices between different attention matrices, the Q matrices of the other two feature sub-blocks are merged during the attention calculation of the image feature sub-block to achieve the purpose of calculating cross attention. The final three-category cross-attention matrix output by the multi-level three-cross attention module is spliced ​​again to obtain a fused cross-attention matrix.

[0043] The fused cross attention matrix is ​​subjected to feature compression extraction and a fully connected layer to obtain the final expression probability output;

[0044] The network model training module is used to build a data set to train the expression recognition neural network model. During the training process, the loss function calculates the loss value between the predicted expression label and the true label;

[0045] The expression recognition module is used to realize expression recognition using the trained expression recognition neural network model.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) The two-stream input proposed in the present invention can provide sufficient image information for the model, and the feature blocks after passing through the feature extraction module can have rich high-level semantic information, focusing more on using landmark features to distinguish subtle differences between expressions, which can improve the recognition accuracy of expressions.

[0048] (2) In the present invention, by combining LBP features, HOG features, image features and landmark features, and cross-utilizing their respective characteristics in the process of calculating attention, the shortcomings of each feature can be compensated, the accuracy of expression recognition can be further improved, and the robustness of the model can be enhanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 1 is a model structure diagram of a facial expression recognition method according to an embodiment of the present invention;

[0050] Figure 2 It is a schematic diagram of the structure of the three-cross attention module in the model of the present invention;

[0051] Figure 3 4 is a flowchart of a facial expression recognition method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0053] The present invention discloses an expression recognition method based on multi-feature fusion and three-cross attention mechanism, which comprises the following steps: screening a facial expression dataset in a natural environment, obtaining an LBPHOG fused feature image of the original image of the dataset through an algorithm, preprocessing the original image and the fused feature image, inputting the preprocessed original image and the fused feature image into a network model, the model first uses the MobileFaceNet pre-trained network model in the feature extraction module to obtain the landmark features of the original feature image and the fused feature image, obtains the image features of the original image through the IR50 pre-trained network model in the feature extraction module, and then performs feature alignment layer The three types of feature sub-blocks obtained are aligned and spliced, and then the spliced ​​feature blocks are input into a three-cross attention module with a depth of 8 to calculate the cross attention matrix. Finally, the SE-Block in the output layer is used for special compression extraction, and the predicted expression label is output after the fully connected layer. The present invention utilizes multiple features of the face image and cross-fuses multiple features to enable the network model to obtain more feature information. The three-cross attention mechanism is used to make up for the shortcomings between each feature, and the landmark feature is used to guide the image feature to output high-level semantic information, thereby solving the problems of similar expression categories, low expression recognition rate, and poor robustness to illumination, partial occlusion, and head posture changes. The details are as follows:

[0054] See also Figure 1-2 , an expression recognition method based on multi-feature fusion and triple cross attention mechanism, characterized by including:

[0055] S1. Predefine feature dimension parameters and obtain a training data set. The present invention uses the facial expression data sets RAF-DB, FER+, and AffectNet in natural environments, and uses numbers 0-6 to represent expressions of happiness, surprise, sadness, anger, disgust, fear, and neutrality, respectively.

[0056] Furthermore, the predefined feature dimension parameter refers to the feature dimension that the three feature sub-blocks need to be aligned during the feature alignment process. This parameter will affect the subsequent position encoding (position embedding) and feature compression extraction and the initialization parameters of the fully connected layer. The predefined parameter in the present invention is 256.

[0057] S2. Obtain the LBPHOG feature fusion image through the algorithm, perform the same preprocessing operation on the feature fusion image and the original image, input the fusion feature image and the original image into the model at the same time for feature extraction, and obtain the lbphog-landmark feature sub-block, image feature sub-block and image-landmark feature sub-block.

[0058] Specifically, the LBP feature image and the HOG feature image of the original expression picture are extracted by a feature extraction algorithm, and the two feature images are fused to obtain the LBPHOG fused feature image.

[0059] Furthermore, in the present invention, the LBPHOG fusion feature image is extracted before training to avoid the waste of computing power caused by feature extraction during training and slow down the model training speed; in image preprocessing, the probabilities of random horizontal flipping, random vertical flipping, random addition of Gaussian noise, and random erasing of regions are set to 0.25, 0.5, 0.25, and 0.5 respectively, and the width and height ratios of the randomly erased regions are set to 0.1 and 0.02. The image size parameter set in the image preprocessing image alignment is 224x224. This operation makes the input fusion feature image and the original image have the same width and height as the original image. The initial feature image is scaled to 224x224 size. The parameters of the image normalization settings in image preprocessing are derived from ImageNet. The mean of each channel is set to 0.485, 0.456, and 0.406, and the variance of each channel is set to 0.229, 0.224, and 0.225, respectively. The image normalization operation scales the pixel values ​​of the image to between (-1, 1), which can accelerate the training and convergence of the model. In data augmentation, random flipping, random noise, and region erasure can enhance the robustness of the model to posture changes and local occlusion.

[0060] The feature extraction in the present invention uses a MobileFaceNet model pre-trained on the ImageNet dataset and an IR50 model trained on the Ms-Celeb-1M dataset. During the training process, the training parameters of MobileFaceNet are frozen and the training parameters of IR50 are fine-tuned. The landmark features of the original image and the landmark features of the LBPHOG feature image are extracted by MobileFaceNet, and the image features are extracted by the IR50 model. By freezing the training parameters, the number of parameters in the model training process is effectively reduced. The pre-trained model can have good feature extraction capabilities, and after feature extraction, the model can obtain high-level semantic information of the image.

[0061] The landmark features extracted from the original image and the LBPHOG fusion feature image can accurately locate the significant areas related to facial expressions. The image features and LBPHOG features can provide global information of the face other than the landmarks, such as cheeks, wrinkled forehead and other parts. At the same time, the LBPHOG features can also provide more accurate landmark feature points.

[0062] S3. Input the three extracted feature sub-blocks into the feature alignment convolution layer. The alignment dimension of each convolution layer is set to 256. The three feature sub-blocks are dimensionally aligned through the convolution layer. After dimension alignment, the average value of the first dimension of each feature sub-block is extracted and expanded into the token dimension of the sub-block. After the three feature sub-blocks are spliced ​​together through the concatenation operation, the position encoding (position embedding) parameters are added to the spliced ​​feature sub-block through the addition operation. The feature blocks after feature alignment and splicing are the input of the cross attention layer.

[0063] Furthermore, the feature alignment convolution layer described in step S3 adopts three independent convolution blocks, and the primary features extracted by the feature extraction module are upgraded or reduced in dimension according to the set dimension alignment parameters, so that the three feature sub-blocks are consistent in feature dimension; the average value of the three feature sub-blocks in the feature dimension is used as the initial value of the expanded word vector (token) dimension, which can maintain the semantic information of the three feature sub-blocks themselves and is conducive to training fitting; the three feature sub-blocks after the expanded dimension are connected through a connection operation, and the position encoding (position embedding) parameter is added to give them position information.

[0064] S4. Input the feature fusion matrix into the multi-level three-cross attention module of the cross-attention layer. In the three-cross attention module, the lbphog-landmark feature matrix, image feature matrix and image-landmark feature matrix are divided according to the aligned dimensions. The three types of feature matrices are calculated in parallel for their respective cross attentions to obtain their respective cross attention matrices; the final three types of cross attention matrices output by the three-cross attention module are spliced ​​again to obtain a fused cross attention matrix.

[0065] Furthermore, in the three-cross attention module, the lbphog-landmark feature matrix, the image feature matrix, and the image-landmark feature matrix x are split. lbp , x img , x lm And perform cross attention calculation, calculate x in the lbphog-landmark attention calculation module lbp K matrix K lbp and V matrix V lbp , and x img The Q matrix Q img , calculate x in the image attention calculation module img K matrix K img and V matrix V img , and x lbp and x lm The Q matrix Q lbp , Q lm , calculate x in the image-landmark attention calculation module lm K matrix K lm and V matrix V lm , and x img The Q matrix Q img , so as to obtain the three cross attention modules output

[0066]

[0067]

[0068]

[0069] Among them, Attention lbp , Attention img , Attention lm are the cross attention matrices of lbphog-landmark, image, and image-landmark feature sub-blocks, Softmax is the Softmax function, and d is the scaling parameter.

[0070] S5. Pass the above fused cross attention matrix through three cross attention modules with a depth of 8 to obtain the final attention matrix, input the attention matrix into SE-Block for feature compression extraction, and then input it into the final fully connected layer network to obtain the expression category output. The loss between the predicted value and the true value is calculated through the cross entropy loss function and the label smoothing loss function, and the next training iteration is performed.

[0071] Furthermore, after outputting the three cross attention matrices, the three types of cross attention matrices are spliced ​​again after passing through the multi-layer perceptron and regularization layer, as well as the residual operation, and then input into the three cross attention modules of the next layer. The formula of a single cross attention module is as follows:

[0072] x′ lbp =Attention lbp +x lbp

[0073] x lbp_out =MLP(Norm(x′ lbp ))+x′ lbp

[0074] x′ img =Attention img +x img

[0075] x img_out =MLP(Norm(x′ img ))+x′ img

[0076] x′ lm =Attention lm +x lm

[0077] x lm_out =MLP(Norm(x′ lm ))+x′ lm

[0078] Among them, MLP is a multi-layer perceptron, Norm is a regularization layer, and x lbp_out , x img_out , x lm_out They are the three types of cross attention matrices output by the three cross attention modules of this layer, and are also the input of the three cross attention modules of the next layer.

[0079] Finally, after passing through the 8-layer triple cross attention module, the final cross attention matrix is ​​input into the SE-Block and the fully connected layer in sequence to obtain the predicted expression label.

[0080] The loss function used in this embodiment includes the cross entropy loss function L CE And the label smoothing loss function L LS , the formula of the cross entropy loss function is Where n represents the number of samples; the formula of the label smoothing loss function is Where y is the true value, is the predicted value, α is the smoothing parameter, and KLDivergence is the KL divergence.

[0081] Furthermore, the accuracy of the method of the present invention obtained by the model proposed in this patent on the RAF-DB and Affect datasets was compared with other expression recognition algorithms SCN, PSR, RAN, KTN, VTFF, TransFER, Meta-Face2Exp, EAC, and POSTER. The results are shown in Table 1:

[0082] Table 1. Comparison of experimental results of expression recognition algorithms

[0083]

[0084]

[0085] As can be seen from Table 1, the expression recognition method proposed in the present invention has a higher accuracy than the common expression recognition algorithms SCN, PSR, RAN, KTN, VTFF, TransFER, Meta-Face2Exp, EAC, and POSTER.

[0086] In summary, the two-stream input proposed in the present invention can provide sufficient image information for the model, and the feature blocks after passing through the feature extraction module can have rich high-level semantic information, with a greater emphasis on using landmark features to distinguish subtle differences between expressions, which can improve the recognition accuracy of expressions. In the present invention, by combining LBP features, HOG features, image features, and landmark features, and cross-utilizing their respective characteristics in the process of calculating attention, the shortcomings of each feature can be compensated, further improving the accuracy of expression recognition and enhancing the robustness of the model.

[0087] In another embodiment, the present invention further provides an expression recognition system based on multi-feature fusion and triple-cross attention mechanism, comprising the following modules:

[0088] The network model building module is used to build the expression recognition neural network model, including the multi-feature input layer, feature extraction layer, feature fusion layer, cross attention layer, and output layer;

[0089] The multi-feature input layer extracts the LBP feature image and the HOG feature image of the original expression picture through a feature extraction algorithm, and fuses the two feature images to obtain an LBPHOG fused feature image;

[0090] Preprocess the original expression image and the LBPHOG fusion feature image, input the two types of preprocessed images into the feature extraction layer to obtain the lbphog-landmark feature sub-block, image feature sub-block, and image-landmark feature sub-block;

[0091] The feature fusion layer is used to align the dimensions of the three feature sub-blocks output by the feature extraction layer. The feature dimension parameters are predefined, and the three feature sub-blocks are aligned on the feature dimension through the convolution layer. After expanding the word vector dimensions of the three feature sub-blocks, the three feature sub-blocks are spliced ​​together and the fused feature matrix is ​​obtained after position encoding.

[0092] The cross-attention layer includes a multi-level three-cross attention module. The three-cross attention module at each level is used to calculate the cross-attention matrix of each feature block after alignment and splicing. By exchanging the Q matrices between different attention matrices, the Q matrices of the other two feature sub-blocks are merged during the attention calculation of the image feature sub-block to achieve the purpose of calculating cross attention. The final three-category cross-attention matrix output by the multi-level three-cross attention module is spliced ​​again to obtain a fused cross-attention matrix.

[0093] The fused cross attention matrix is ​​subjected to feature compression extraction and a fully connected layer to obtain the final expression probability output;

[0094] The network model training module is used to build a data set to train the expression recognition neural network model. During the training process, the loss function calculates the loss value between the predicted expression label and the true label;

[0095] The expression recognition module is used to realize expression recognition using the trained expression recognition neural network model.

[0096] The specific implementation method of each module is the same as that of each step and will not be described in detail in the present invention.

[0097] The above-described embodiments represent several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the patent. It should be noted that those skilled in the art may make various modifications and improvements without departing from the spirit of the present invention, and such modifications and improvements fall within the scope of protection of the present invention. Therefore, the scope of protection of the patent for this invention shall be based on the appended claims.

Claims

1. Expression recognition method based on multi-feature fusion and triple-cross attention mechanism, characterized by: The steps include: Step 1: Build an expression recognition neural network model, including a multi-feature input layer, a feature extraction layer, a feature fusion layer, a cross-attention layer, and an output layer; The multi-feature input layer extracts the LBP feature image and the HOG feature image of the original expression picture through a feature extraction algorithm, and fuses the two feature images to obtain an LBPHOG fused feature image; Preprocess the original expression image and the LBPHOG fusion feature image, input the two types of preprocessed images into the feature extraction layer to obtain the lbphog-landmark feature sub-block, image feature sub-block, and image-landmark feature sub-block; The feature fusion layer is used to align the dimensions of the three feature sub-blocks output by the feature extraction layer. The feature dimension parameters are predefined, and the three feature sub-blocks are aligned on the feature dimension through the convolution layer. After expanding the word vector dimensions of the three feature sub-blocks, the three feature sub-blocks are spliced ​​together and the fused feature matrix is ​​obtained after position encoding. The cross-attention layer includes a multi-level three-cross attention module. The three-cross attention module at each level is used to calculate the cross-attention matrix of each feature block after alignment and splicing. By exchanging the Q matrices between different attention matrices, the Q matrices of the other two feature sub-blocks are merged during the attention calculation of the image feature sub-block to achieve the purpose of calculating cross attention. The final three-category cross-attention matrix output by the multi-level three-cross attention module is spliced ​​again to obtain a fused cross-attention matrix. The fused cross attention matrix is ​​subjected to feature compression extraction and a fully connected layer to obtain the final expression probability output; Step 2: Construct a data set to train the expression recognition neural network model in step 1. During the training process, the loss function calculates the loss value between the predicted expression label and the true label. Step 3: Use the trained expression recognition neural network model to realize expression recognition.

2. The facial expression recognition method based on multi-feature fusion and triple-cross attention mechanism as claimed in claim 1, characterized in that: The Circle LBP algorithm is used to extract LBP feature images from the original expression images.

3. The facial expression recognition method based on multi-feature fusion and triple-cross attention mechanism as claimed in claim 1, characterized in that: The preprocessing method is to uniformly scale and standardize the original input expression image and the fused feature image, and perform data enhancement respectively. The data enhancement method includes random horizontal flipping, random vertical flipping, random addition of Gaussian noise, and random erasing of image areas.

4. The facial expression recognition method based on multi-feature fusion and triple-cross attention mechanism as claimed in claim 1, characterized in that: The feature extraction layer includes two convolutional neural network models, MobileFaceNet and IR50. The MobileFaceNet network model is pre-trained with the ImageNet dataset, and the IR50 network model is pre-trained with the Ms-Celeb-1M dataset. During the training process, the training parameters of MobileFaceNet are frozen, and the training parameters of IR50 are fine-tuned. The landmark features of the original expression image and the landmark features of the LBPHOG feature image are extracted through MobileFaceNet to obtain image-landmark feature sub-blocks and lbphog-landmark feature sub-blocks, respectively. The image features are extracted through the IR50 model to obtain image feature sub-blocks.

5. The facial expression recognition method based on multi-feature fusion and triple-cross attention mechanism as claimed in claim 1, characterized in that: After extracting the three types of features, a dimension alignment convolution layer is used to align the three dimensions. After the dimension alignment, the average value of the first dimension of the three feature sub-blocks is extracted and expanded into the word vector dimension of the sub-block. The three feature sub-blocks are then spliced ​​together through a splicing operation, and position encoding parameters are added to the spliced ​​feature sub-blocks through an addition operation.

6. The facial expression recognition method based on multi-feature fusion and triple-cross attention mechanism as claimed in claim 1, characterized in that: In the cross attention layer, the three cross attention modules at each level split the lbphog-landmark feature matrix x according to the aligned dimensions. lbp 、image feature matrix x img And the image-landmark feature matrix x lm , calculate the cross attention of the three types of feature matrices in parallel to obtain their respective cross attention matrices; The implementation process of calculating the respective cross attention is as follows: Calculate x in the lbphog-landmark attention calculation module lbp K matrix K lbp and V matrix V lbp , and x img The Q matrix Q img , calculate x in the image attention calculation module img K matrix K img and V matrix V img , and x lbp and x lm The Q matrix Q lbp ,Q lm , calculate x in the image-landmark attention calculation module lm K matrix K lm and V matrix V lm , and x img The Q matrix Q img , thus obtaining the three cross attentions output by the three cross attention modules: Among them, Attention lbp ,Attention img ,Attention lm are the cross attention matrices of lbphog-landmark, image, and image-landmark feature sub-blocks, Softmax is the Softmax function, and d is the scaling parameter.

7. The facial expression recognition method based on multi-feature fusion and triple-cross attention mechanism as claimed in claim 6, characterized in that: The specific implementation method of obtaining the fused cross attention matrix is ​​as follows; After the three-cross attention module at each level outputs three cross attention matrices, they are spliced ​​again after passing through the multi-layer perceptron and regularization layer, as well as the residual operation, and then input into the three-cross attention module of the next layer. The formula of a single cross attention module is as follows: x′ lbp =Attention lbp +x lbp x lbp_out =MLP(Norm(x′ lbp ))+x′ lbp x′ img =Attention img +x img x img_out =MLP(Norm(x′ img ))+x′ img x′ lm =Attention lm +x lm x lm_out =MLP(Norm(x′ lm ))+x′ km Among them, MLP is a multi-layer perceptron, Norm is a regularization layer, and x lbp_out ,x img_out ,x lm_out They are the three types of cross attention matrices output by the three-cross attention module in a certain layer, and are also the input of the three-cross attention module in the next layer.

8. The facial expression recognition method based on multi-feature fusion and triple-cross attention mechanism as claimed in claim 1, characterized in that: Feature compression extraction is performed through SE-Block.

9. The facial expression recognition method based on multi-feature fusion and triple-cross attention mechanism as claimed in claim 1, characterized in that: The loss function includes the cross entropy loss function L CE And the label smoothing loss function LS , the formula of the cross entropy loss function is Where n represents the number of samples; the formula of the label smoothing loss function is Where y is the true value, is the predicted value, α is the smoothing parameter, and KLDivergence is the KL divergence.

10. Expression recognition system based on multi-feature fusion and triple-cross attention mechanism, characterized by: Includes the following modules: The network model building module is used to build the expression recognition neural network model, including the multi-feature input layer, feature extraction layer, feature fusion layer, cross attention layer, and output layer; The multi-feature input layer extracts the LBP feature image and the HOG feature image of the original expression picture through a feature extraction algorithm, and fuses the two feature images to obtain an LBPHOG fused feature image; Preprocess the original expression image and the LBPHOG fusion feature image, input the two types of preprocessed images into the feature extraction layer to obtain the lbphog-landmark feature sub-block, image feature sub-block, and image-landmark feature sub-block; The feature fusion layer is used to align the dimensions of the three feature sub-blocks output by the feature extraction layer. The feature dimension parameters are predefined, and the three feature sub-blocks are aligned on the feature dimension through the convolution layer. After expanding the word vector dimensions of the three feature sub-blocks, the three feature sub-blocks are spliced ​​together and the fused feature matrix is ​​obtained after position encoding. The cross-attention layer includes a multi-level three-cross attention module. The three-cross attention module at each level is used to calculate the cross-attention matrix of each feature block after alignment and splicing. By exchanging the Q matrices between different attention matrices, the Q matrices of the other two feature sub-blocks are merged during the attention calculation of the image feature sub-block to achieve the purpose of calculating cross attention. The final three-category cross-attention matrix output by the multi-level three-cross attention module is spliced ​​again to obtain a fused cross-attention matrix. The fused cross attention matrix is ​​subjected to feature compression extraction and a fully connected layer to obtain the final expression probability output; The network model training module is used to build a data set to train the expression recognition neural network model. During the training process, the loss function calculates the loss value between the predicted expression label and the true label; The expression recognition module is used to realize expression recognition using the trained expression recognition neural network model.

Images