Facial expression recognition method and device based on multi-modal and multi-scale feature fusion

Abstract

The application provides a facial expression recognition method and device based on multi-modal and multi-scale feature fusion, which inputs a to-be-recognized expression picture into a trained facial expression recognition network, extracts abstract features, face key point features, face edge features and face texture features; features of the same scale are respectively subjected to feature extraction by using an attention mechanism to obtain deep features and shallow features, and all features after linear mapping are fused and classified. When the deep features and the shallow features are extracted, the features of various resolutions output by the convolutional neural network are subjected to multi-scale feature fusion, so that the loss of original image information can be reduced; the four kinds of face features are fused, so that the auxiliary guiding effect of artificial features on abstract feature extraction can be improved in more dimensions, so that the classification accuracy of the network can be improved to a greater extent. Meanwhile, the attention mechanism promotes the network to accurately locate the key area highly related to the expression, so that the classification ability of the network can be improved.

Description

Technical Field

[0001] This invention belongs to the field of facial expression recognition technology, specifically relating to a facial expression recognition method and device based on multimodal and multi-scale feature fusion. Background Technology

[0002] In recent years, with the development of artificial intelligence technology, especially the advancement of deep learning technology, more and more researchers have chosen to use deep learning to extract discriminative facial expression features, thereby achieving expression recognition. Compared with manually designed facial features, the features extracted by deep learning technology are generally more abstract, more discriminative, and more applicable.

[0003] However, models trained by deep learning often contain a large number of learnable parameters, and the updates of these parameters require a large amount of data to drive them. Therefore, the features extracted by deep learning techniques generally lack interpretability. On the other hand, conventional deep learning networks only introduce one handcrafted feature—the location of facial key points—and learn and update parameters based on features at a single resolution. This lack of shallow, superficial features provides insufficient guidance for extracting abstract facial expression features. Therefore, learning only single-resolution features cannot encourage the network to pay attention to detailed features at different scales. Furthermore, conventional deep learning networks first downscale the original image and then use super-resolution techniques to construct magnified images of different sizes, extracting features from these images at different scales. However, the magnification process causes loss of information in the original image, and the network's feature extraction ability comes solely from deep learning, making it unable to accurately locate key regions highly correlated with facial expressions, thus affecting the network's final classification ability. Summary of the Invention

[0004] To address the aforementioned problems in the existing technology, this invention provides a facial expression recognition method and apparatus based on multimodal and multi-scale feature fusion. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] In a first aspect, the present invention provides a facial expression recognition method based on multimodal and multi-scale feature fusion, comprising:

[0006] S100: Acquire the image of the expression to be recognized and the trained facial expression recognition network;

[0007] S200, the image of the expression to be recognized is input into a trained facial expression recognition network, so that the facial expression recognition network extracts abstract features, facial key point features, facial edge features, and facial texture features of the image of the expression to be recognized; deep features are extracted using an attention mechanism for abstract features and facial key point features of the same scale, and shallow features are extracted using a self-attention mechanism for facial edge features and facial texture features of the same scale; the deep features and shallow features are fused and classified to obtain the expression category to which the image of the expression to be recognized belongs.

[0008] Secondly, the present invention provides a facial expression recognition device based on multimodal and multi-scale feature fusion, comprising:

[0009] The acquisition module is configured to acquire the image of the expression to be recognized and the trained facial expression recognition network.

[0010] The recognition module is configured to input the image of the expression to be recognized into a trained facial expression recognition network, so that the facial expression recognition network extracts abstract features, facial key point features, facial edge features, and facial texture features of the image of the expression to be recognized; it uses an attention mechanism to extract deep features for abstract features and facial key point features of the same scale, and uses a self-attention mechanism to extract shallow features for facial edge features and facial texture features of the same scale; and it fuses and classifies the deep features and shallow features to obtain the expression category to which the image of the expression to be recognized belongs.

[0011] Beneficial effects:

[0012] Compared with the prior art 1, the present invention constructs a multimodal feature extraction method that integrates four types of facial features: abstract features, key point features, texture features, and edge features. This can enhance the auxiliary guiding role of artificial features in the extraction of abstract facial features in more dimensions, thereby improving the classification accuracy of the network to a greater extent.

[0013] Compared to the prior art 2, this invention performs multi-scale feature fusion on the feature images of various resolutions output by the convolutional neural network when extracting deep and shallow features, which can reduce the loss of information in the original image. At the same time, the attention mechanism enables the network to accurately locate key regions that are highly related to facial expressions, thereby improving the network's classification ability.

[0014] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a facial expression recognition method based on multimodal and multi-scale feature fusion provided by the present invention.

[0016] Figure 2This is a schematic diagram of the process of training the facial expression recognition network provided by the present invention;

[0017] Figure 3 This is a schematic diagram of the facial expression recognition network provided by the present invention;

[0018] Figure 4 This is a schematic diagram of the processing procedure of the deep attention feature extraction network provided by the present invention;

[0019] Figure 5 This is a schematic diagram of the processing procedure of the shallow self-attention feature extraction network provided by the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0021] like Figure 1 As shown, this invention provides a facial expression recognition method based on multimodal and multi-scale feature fusion, comprising:

[0022] S100: Acquire the image of the expression to be recognized and the trained facial expression recognition network;

[0023] S200, the image of the expression to be recognized is input into a trained facial expression recognition network, so that the facial expression recognition network extracts abstract features, facial key point features, facial edge features, and facial texture features of the image of the expression to be recognized; deep features are extracted using an attention mechanism for abstract features and facial key point features of the same scale, and shallow features are extracted using a self-attention mechanism for facial edge features and facial texture features of the same scale; the deep features and shallow features are fused and classified to obtain the expression category to which the image of the expression to be recognized belongs.

[0024] As a specific embodiment of the present invention, refer to Figure 2 The training process of the trained facial expression recognition network of the present invention includes:

[0025] a. Obtain an expression recognition dataset, which includes multiple expression images carrying real category labels;

[0026] b. Select the training set and test set from the facial expression recognition dataset;

[0027] c. Perform data preprocessing on each facial expression image in the training set and the test set to stretch the resolution of each facial expression image to 224×224 to obtain a preprocessed facial expression image;

[0028] d. The preprocessed expression image is input into a preset facial expression recognition network, so that the facial expression recognition network extracts abstract features, facial key point features, facial edge features, and facial texture features from the preprocessed expression image; deep features are extracted from abstract features and facial key point features of the same scale using an attention mechanism, and shallow features are extracted from facial edge features and facial texture features of the same scale using a self-attention mechanism; the deep features and shallow features extracted from the preprocessed expression image are fused and classified to obtain the predicted expression category of the preprocessed expression image;

[0029] e. Compare the predicted expression category of each preprocessed expression image with the carried true category label, and adjust the internal parameters of the facial expression recognition network according to the comparison result;

[0030] f, repeat d to e until the maximum number of training iterations is reached, and a well-trained facial expression recognition network is obtained.

[0031] In this embodiment of the invention, training and test datasets are obtained from an expression recognition dataset. Then, an abstract feature extraction network based on a convolutional neural network, a facial key point feature extraction network based on MobileFaceNet, a facial edge feature extraction network based on edge detection, and a facial texture feature extraction network based on local binary patterns are constructed respectively. Next, a deep attention feature extraction network is used to extract features from the deep features composed of abstract features and facial key point features. A shallow self-attention feature extraction network is used to extract features from the shallow features composed of facial edge features and facial texture features. Finally, a multi-scale feature fusion network is used to classify the fused features to obtain the classification results of the expression images.

[0032] refer to Figure 3 The facial expression recognition network of the present invention includes a deep feature extraction branch, a shallow feature extraction branch, and a multi-scale feature fusion classification network; the deep feature extraction branch comprises an abstract feature extraction network based on a convolutional neural network, a facial key point feature extraction network based on MobileFaceNet, and a deep attention feature extraction network; the shallow feature extraction branch comprises a facial edge feature extraction network based on an edge detection method, a facial texture feature extraction network based on local binary patterns, and a shallow self-attention feature extraction network; the multi-scale feature fusion classification network includes a fusion module and a classification module.

[0033] The deep feature extraction branch runs in parallel with the shallow feature extraction branch, and their outputs are both connected to the multi-scale feature fusion classification network. The output of the fusion module is connected to the input of the classification module. The abstract feature extraction network runs in parallel with the facial key point feature extraction network, and their outputs are both connected to the deep attention feature extraction network. The facial edge feature extraction network runs in parallel with the facial texture feature extraction network, and their outputs are both connected to the shallow self-attention feature extraction network.

[0034] Combination Figure 2 and Figure 3 This invention consists of three parts: a multimodal four-face feature extraction network, an attention feature extraction network, and a multi-scale feature fusion classification network. The abstract feature extraction network, based on a convolutional neural network (CNN) for extracting abstract facial features, utilizes a CNN based on residual and squeeze activation modules. This network is primarily composed of three feature extraction modules, each with a similar structure, differing only in the resolution of their output features. Each attention feature extraction network first extracts features through convolution and batch regularization operations, then performs nonlinear mapping using parameterized linear units. Next, it uses the squeeze activation module to extract attention features across various dimensions. Finally, it uses residual mapping to add the original input features to the calculated features, further improving performance. The multi-scale feature extraction network includes deep and shallow feature extraction branches. Deep features are the outputs of the attention extraction modules, consisting of abstract features and keypoint features. Shallow features are the outputs of the self-attention extraction modules, consisting of facial edge features and facial texture features. The deep feature extraction branch extracts highly saliency features from the abstract features and keypoint features extracted by the previous network at three different scales through an attention feature extraction network. The structure of the attention feature extraction network is similar to the self-attention mechanism in the Transformer. (Refer to...) Figure 4 The difference lies in that its query (Q), key (K), and value (V) are obtained by linearly mapping different features. Specifically, the linear mapping matrix W... Q The key point features are mapped to query Q, through a linear mapping matrix W. K and W V This maps abstract features to keys K and values ​​V. The shallow feature extraction branch uses facial edge features and facial texture features as input. Unlike the deep feature branch, which extracts facial edge and texture features using traditional methods with only a single resolution, the shallow feature extraction branch first uses a convolutional neural network to pre-process these features, obtaining features at three different scales. (Reference) Figure 5The two methods are fused at different scales by averaging, and then a high-saliency feature extraction network is used to extract the feature. For the multi-scale feature fusion classification network, the multi-scale features output by the deep feature extraction branch and the shallow feature extraction branch are first linearly mapped to features at the same scale. Then, the deep features and shallow features at the same scale are fused by multiplying and adding the features with weights. Finally, the features at each scale are concatenated to form the final fused feature for classification prediction, and the classification loss of the network is calculated.

[0035] exist Figure 3 In the network structure diagram, different network structures can be used to obtain image feature maps for abstract feature extraction networks. Although the network structures differ, their essence is the same. Regarding deep attention feature extraction networks, their structure is similar to the self-attention mechanism in Transformers. The difference is that their query (Q), key (K), and value (V) are obtained by linearly mapping different features. However, the mapping relationships may differ, potentially affecting the network's accuracy, but this does not affect the final prediction result. Regarding multi-scale feature fusion methods, this invention uses the multiplication and summation of features and attention weights. Alternative solutions could use convolutional layers or fully connected layers to obtain the final fused features.

[0036] As a specific embodiment of the present invention, combined with Figure 1 , Figure 4 and Figure 5 S200 includes:

[0037] The image of the expression to be recognized is input into a trained facial expression recognition network. The abstract feature extraction network and the facial keypoint feature extraction network extract abstract features and facial keypoint features from the image of the expression to be recognized, respectively, resulting in abstract features and facial keypoint features at multiple scales. Similarly, the facial edge feature extraction network and the facial texture feature extraction network extract facial edge features and facial texture features from the image of the expression to be recognized, respectively, resulting in facial edge features and facial texture features at multiple scales. The deep attention feature extraction network extracts deep features from the abstract features and facial keypoints of the same scale of the image of the expression to be recognized using an attention machine. The shallow self-attention feature extraction network extracts shallow features from the facial edge features and facial texture features of the same scale of the image of the expression to be recognized using a self-attention machine. Finally, the multi-scale feature fusion classification network fuses and classifies the deep and shallow features of the image of the expression to be recognized at multiple scales to obtain the expression category to which the image of the expression to be recognized belongs.

[0038] As a specific embodiment of the present invention, combined with Figure 1 , Figure 4 and Figure 5 The deep attention feature extraction network extracts deep features from the same scale of the expression image to be identified and facial key points using an attention machine, including:

[0039] Abstract features and facial landmark features of the same scale from the image of the expression to be identified are input into the deep attention feature extraction network, so that the deep attention feature extraction network passes through the linear mapping matrix W. Q The key point features are mapped to query Q, through a linear mapping matrix W. K and W V The abstract features are mapped to keys K and values ​​V. Q and K are multiplied and the result is scaled. The scaled result is then normalized using the softmax function to obtain the self-attention weights. The self-attention weights are multiplied with V to obtain deep features at multiple scales.

[0040] The shallow self-attention feature extraction network extracts shallow features from the same scale of the facial edge features and facial texture features of the expression image to be identified using a self-attention machine, including:

[0041] Facial edge features and facial texture features of the same scale are input into the shallow self-attention feature extraction network, so that the shallow self-attention feature extraction network calculates the average value of the facial edge features and facial texture features, and then uses a linear mapping matrix W. Q W K and W V The average value is mapped to query Q, key K, and value V respectively. Q and K are multiplied and the result is scaled. The scaled result is then normalized using the softmax function to obtain the self-attention weight. The self-attention weight is multiplied with V to obtain shallow features at multiple scales.

[0042] In one specific embodiment of the present invention, the multi-scale feature fusion classification network is used to fuse and classify the deep and shallow features of the expression image to be identified at multiple scales, thereby obtaining the expression category to which the expression image to be identified belongs, including:

[0043] The deep and shallow features of the image of the expression to be identified at multiple scales are input into the multi-scale feature fusion classification network. The fusion module uses the method of multiplying and adding features and weights to first fuse the deep and shallow features at the same scale, and then connects the features at each scale to obtain the fused features. The fused features are then input into the classification module to classify the expression to which the image of the expression to be identified belongs.

[0044] As a specific embodiment of the present invention, combined with Figure 1 , Figure 4 and Figure 5 d includes:

[0045] The preprocessed facial expression image is input into a trained facial expression recognition network. The abstract feature extraction network and the facial keypoint feature extraction network extract abstract features and facial keypoint features from the preprocessed facial expression image, respectively, resulting in abstract features and facial keypoint features at multiple scales. Similarly, the facial edge feature extraction network and the facial texture feature extraction network extract facial edge features and facial texture features from the preprocessed facial expression image, respectively, resulting in facial edge features and facial texture features at multiple scales. The deep attention feature extraction network performs deep feature extraction on the same scale of abstract features and facial keypoints of the preprocessed facial expression image using an attention machine. The shallow self-attention feature extraction network performs shallow feature extraction on the same scale of facial edge features and facial texture features of the preprocessed facial expression image using a self-attention machine. Finally, the multi-scale feature fusion classification network fuses and classifies the deep and shallow features of the preprocessed facial expression image at multiple scales to obtain the predicted facial expression category of the preprocessed facial expression image.

[0046] As a specific embodiment of the present invention, combined with Figures 2-5 The step of using the deep attention feature extraction network to perform deep feature extraction on the preprocessed expression image at the same scale and facial key points using an attention machine includes:

[0047] The abstract features and facial landmark features of the preprocessed facial image at the same scale are input into the deep attention feature extraction network, so that the deep attention feature extraction network passes through the linear mapping matrix W. Q The key point features are mapped to query Q, through a linear mapping matrix W. K and W V The abstract features are mapped to keys K and values ​​V. Q and K are multiplied and the result is scaled. The scaled result is then normalized using the softmax function to obtain the self-attention weights. The self-attention weights are multiplied with V to obtain deep features at multiple scales.

[0048] The shallow self-attention feature extraction network extracts shallow features from the preprocessed facial expression image at the same scale using a self-attention machine, including:

[0049] Facial edge features and facial texture features of the same scale are input into the shallow self-attention feature extraction network, so that the shallow self-attention feature extraction network calculates the average value of the facial edge features and facial texture features, and then uses a linear mapping matrix W. Q W K and W V The average value is mapped to query Q, key K, and value V respectively. Q and K are multiplied and the result is scaled. The scaled result is then normalized using the softmax function to obtain the self-attention weight. The self-attention weight is multiplied with V to obtain shallow features at multiple scales.

[0050] As a specific embodiment of the present invention, combined with Figure 2 and Figure 3 The multi-scale feature fusion classification network fuses and classifies deep and shallow features at multiple scales of the preprocessed facial expression image to obtain the predicted facial expression category of the preprocessed facial expression image. This includes:

[0051] The deep and shallow features at multiple scales of the preprocessed facial expression image are input into the multi-scale feature fusion classification network. The fusion module uses the method of multiplying and adding features and weights to first fuse deep and shallow features at the same scale, and then connects the features at each scale to obtain fused features. The fused features are then input into the classification module for classification to obtain the predicted facial expression category of the preprocessed facial expression image.

[0052] Secondly, the present invention provides a facial expression recognition device based on multimodal and multi-scale feature fusion, comprising:

[0053] The acquisition module is configured to acquire the image of the expression to be recognized and the trained facial expression recognition network.

[0054] The recognition module is configured to input the image of the expression to be recognized into a trained facial expression recognition network, so that the facial expression recognition network extracts abstract features, facial key point features, facial edge features, and facial texture features of the image of the expression to be recognized; it uses an attention mechanism to extract deep features for abstract features and facial key point features of the same scale, and uses a self-attention mechanism to extract shallow features for facial edge features and facial texture features of the same scale; and it fuses and classifies the deep features and shallow features to obtain the expression category to which the image of the expression to be recognized belongs.

[0055] This invention provides a facial expression recognition method and apparatus based on multimodal and multi-scale feature fusion. The method involves inputting an image of the expression to be recognized into a trained facial expression recognition network to extract abstract features, facial keypoint features, facial edge features, and facial texture features. For abstract features and facial keypoint features at the same scale, an attention mechanism is used to extract deep features; for facial edge features and facial texture features at the same scale, a self-attention mechanism is used to extract shallow features. The deep and shallow features are then fused and classified. This invention performs multi-scale feature fusion on feature images of various resolutions output by the convolutional neural network during the extraction of deep and shallow features, which reduces the loss of information in the original image. Fusing four types of facial features enhances the auxiliary guidance role of artificial features in the extraction of abstract expression features in more dimensions, thereby significantly improving the network's classification accuracy. Simultaneously, the attention mechanism enables the network to accurately locate key regions highly correlated with the expression, thus improving the network's classification ability.

[0056] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0057] Although this application has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality.

[0058] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A facial expression recognition method based on multimodal and multi-scale feature fusion, characterized in that, include: S100: Acquire the image of the expression to be recognized and the trained facial expression recognition network; S200, the image of the expression to be recognized is input into the trained facial expression recognition network so that the facial expression recognition network can extract the abstract features, facial key point features, facial edge features and facial texture features of the image of the expression to be recognized; The facial expression recognition network extracts deep features from abstract features and facial keypoint features at the same scale using an attention mechanism, and extracts shallow features from facial edge features and facial texture features at the same scale using a self-attention mechanism. The deep and shallow features are then fused and classified to determine the expression category of the expression image to be recognized. The facial expression recognition network includes a deep feature extraction branch, a shallow feature extraction branch, and a multi-scale feature fusion classification network. The deep feature extraction branch comprises an abstract feature extraction network based on a convolutional neural network, a facial keypoint feature extraction network based on MobileFaceNet, and a deep attention feature extraction network. The shallow feature extraction branch comprises a facial edge feature extraction network based on edge detection methods, a facial texture feature extraction network based on local binary patterns, and a shallow self-attention feature extraction network. The multi-scale feature fusion classification network includes a fusion module and a classification module. The deep feature extraction branch runs in parallel with the shallow feature extraction branch, and their outputs are both connected to the multi-scale feature fusion classification network; the output of the fusion module is connected to the input of the classification module; the abstract feature extraction network runs in parallel with the facial key point feature extraction network, and their outputs are both connected to the deep attention feature extraction network; the facial edge feature extraction network runs in parallel with the facial texture feature extraction network, and their outputs are both connected to the shallow self-attention feature extraction network. S200 includes: The image of the expression to be recognized is input into a trained facial expression recognition network. The abstract feature extraction network and the facial keypoint feature extraction network extract abstract features and facial keypoint features from the image of the expression to be recognized, respectively, resulting in abstract features and facial keypoint features at multiple scales. Similarly, the facial edge feature extraction network and the facial texture feature extraction network extract facial edge features and facial texture features from the image of the expression to be recognized, respectively, resulting in facial edge features and facial texture features at multiple scales. The deep attention feature extraction network extracts deep features from the abstract features and facial keypoints of the same scale of the image of the expression to be recognized using an attention machine. The shallow self-attention feature extraction network extracts shallow features from the facial edge features and facial texture features of the same scale of the image of the expression to be recognized using a self-attention machine. Finally, the multi-scale feature fusion classification network fuses and classifies the deep and shallow features of the image of the expression to be recognized at multiple scales to obtain the expression category to which the image of the expression to be recognized belongs.

2. The facial expression recognition method based on multimodal and multi-scale feature fusion according to claim 1, characterized in that, The training process of the trained facial expression recognition network includes: a. Obtain an expression recognition dataset, which includes multiple expression images carrying real category labels; b. Select the training set and test set from the facial expression recognition dataset; c. Perform data preprocessing on each facial expression image in the training set and the test set to stretch the resolution of each facial expression image to 224×224 to obtain a preprocessed facial expression image; d. The preprocessed expression image is input into a preset facial expression recognition network, so that the facial expression recognition network extracts abstract features, facial key point features, facial edge features, and facial texture features from the preprocessed expression image; deep features are extracted from abstract features and facial key point features of the same scale using an attention mechanism, and shallow features are extracted from facial edge features and facial texture features of the same scale using a self-attention mechanism; the deep features and shallow features extracted from the preprocessed expression image are fused and classified to obtain the predicted expression category of the preprocessed expression image; e. Compare the predicted expression category of each preprocessed expression image with the carried true category label, and adjust the internal parameters of the facial expression recognition network according to the comparison result; f, repeat d to e until the maximum number of training iterations is reached, and a well-trained facial expression recognition network is obtained.

3. The facial expression recognition method based on multimodal and multi-scale feature fusion according to claim 2, characterized in that, The step of using the deep attention feature extraction network to perform deep feature extraction on the abstract features and facial key points of the expression image to be identified at the same scale using an attention machine includes: Abstract features and facial landmark features of the same scale from the image of the expression to be identified are input into the deep attention feature extraction network, so that the deep attention feature extraction network passes through the linear mapping matrix W. Q The key point features are mapped to query Q, through a linear mapping matrix W. K and W V The abstract features are mapped to keys K and values ​​V. Q and K are multiplied and the result is scaled. The scaled result is then normalized using the softmax function to obtain the self-attention weights. The self-attention weights are multiplied with V to obtain deep features at multiple scales. The shallow self-attention feature extraction network extracts shallow features from the same scale of the facial edge features and facial texture features of the expression image to be identified using a self-attention machine, including: Facial edge features and facial texture features of the same scale are input into the shallow self-attention feature extraction network, so that the shallow self-attention feature extraction network calculates the average value of the facial edge features and facial texture features, and then uses a linear mapping matrix W. Q W K and W V The average value is mapped to query Q, key K, and value V respectively. Q and K are multiplied and the result is scaled. The scaled result is then normalized using the softmax function to obtain the self-attention weight. The self-attention weight is multiplied with V to obtain shallow features at multiple scales.

4. The facial expression recognition method based on multimodal and multi-scale feature fusion according to claim 2, characterized in that, The multi-scale feature fusion classification network fuses and classifies the deep and shallow features of the expression image to be identified at multiple scales, resulting in the following expression categories: The deep and shallow features of the image of the expression to be identified at multiple scales are input into the multi-scale feature fusion classification network. The fusion module uses the method of multiplying and adding features and weights to first fuse the deep and shallow features at the same scale, and then connects the features at each scale to obtain the fused features. The fused features are then input into the classification module to classify the expression to which the image of the expression to be identified belongs.

5. The facial expression recognition method based on multimodal and multi-scale feature fusion according to claim 2, characterized in that, d includes: The preprocessed expression image is input into the trained facial expression recognition network. The abstract feature extraction network and the facial key point feature extraction network are used to extract the abstract features and facial key point features of the preprocessed expression image, respectively, to obtain the corresponding abstract features and facial key point features at multiple scales. The facial edge feature extraction network and the facial texture feature extraction network are used to extract the facial edge features and facial texture features of the preprocessed expression image, respectively, to obtain the corresponding facial edge features and facial texture features at multiple scales. The deep attention feature extraction network extracts deep features from the preprocessed expression image using an attention machine to extract abstract features and facial key points at the same scale. The shallow self-attention feature extraction network extracts shallow features from the preprocessed expression image using a self-attention machine to extract facial edge features and facial texture features at the same scale. The multi-scale feature fusion classification network fuses and classifies the deep and shallow features of the preprocessed expression image at multiple scales to obtain the predicted expression category of the preprocessed expression image.

6. The facial expression recognition method based on multimodal and multi-scale feature fusion according to claim 5, characterized in that, The deep attention feature extraction network extracts deep features from the preprocessed facial expression image using an attention machine, including abstract features of the same scale and facial landmarks. The abstract features and facial landmark features of the preprocessed facial image at the same scale are input into the deep attention feature extraction network, so that the deep attention feature extraction network passes through the linear mapping matrix W. Q The key point features are mapped to query Q, through a linear mapping matrix W. K and W V The abstract features are mapped to keys K and values ​​V. Q and K are multiplied and the result is scaled. The scaled result is then normalized using the softmax function to obtain the self-attention weights. The self-attention weights are multiplied with V to obtain deep features at multiple scales. The shallow self-attention feature extraction network extracts shallow features from the preprocessed facial expression image at the same scale using a self-attention machine, including: Facial edge features and facial texture features of the same scale are input into the shallow self-attention feature extraction network, so that the shallow self-attention feature extraction network calculates the average value of the facial edge features and facial texture features, and then uses a linear mapping matrix W. Q W K and W V The average value is mapped to query Q, key K, and value V respectively. Q and K are multiplied and the result is scaled. The scaled result is then normalized using the softmax function to obtain the self-attention weight. The self-attention weight is multiplied with V to obtain shallow features at multiple scales.

7. The facial expression recognition method based on multimodal and multi-scale feature fusion according to claim 6, characterized in that, The multi-scale feature fusion classification network fuses and classifies the deep and shallow features at multiple scales of the preprocessed expression image to obtain the predicted expression category of the preprocessed expression image, including: The deep and shallow features at multiple scales of the preprocessed facial expression image are input into the multi-scale feature fusion classification network. The fusion module uses the method of multiplying and adding features and weights to first fuse deep and shallow features at the same scale, and then connects the features at each scale to obtain fused features. The fused features are then input into the classification module for classification to obtain the predicted facial expression category of the preprocessed facial expression image.

8. A facial expression recognition device based on multimodal and multi-scale feature fusion, characterized in that, include: The acquisition module is configured to acquire the image of the expression to be recognized and the trained facial expression recognition network. The recognition module is configured to input the image of the expression to be recognized into a trained facial expression recognition network, so that the facial expression recognition network can extract the abstract features, facial key point features, facial edge features and facial texture features of the image of the expression to be recognized. The facial expression recognition network extracts deep features from abstract features and facial keypoint features at the same scale using an attention mechanism, and extracts shallow features from facial edge features and facial texture features at the same scale using a self-attention mechanism. The deep and shallow features are then fused and classified to determine the expression category of the expression image to be recognized. The facial expression recognition network includes a deep feature extraction branch, a shallow feature extraction branch, and a multi-scale feature fusion classification network. The deep feature extraction branch comprises an abstract feature extraction network based on a convolutional neural network, a facial keypoint feature extraction network based on MobileFaceNet, and a deep attention feature extraction network. The shallow feature extraction branch comprises a facial edge feature extraction network based on edge detection methods, a facial texture feature extraction network based on local binary patterns, and a shallow self-attention feature extraction network. The multi-scale feature fusion classification network includes a fusion module and a classification module. The deep feature extraction branch runs in parallel with the shallow feature extraction branch, and their outputs are both connected to the multi-scale feature fusion classification network; the output of the fusion module is connected to the input of the classification module; the abstract feature extraction network runs in parallel with the facial key point feature extraction network, and their outputs are both connected to the deep attention feature extraction network; the facial edge feature extraction network runs in parallel with the facial texture feature extraction network, and their outputs are both connected to the shallow self-attention feature extraction network. S200 includes: The image of the expression to be recognized is input into a trained facial expression recognition network. The abstract feature extraction network and the facial keypoint feature extraction network extract abstract features and facial keypoint features from the image of the expression to be recognized, respectively, resulting in abstract features and facial keypoint features at multiple scales. Similarly, the facial edge feature extraction network and the facial texture feature extraction network extract facial edge features and facial texture features from the image of the expression to be recognized, respectively, resulting in facial edge features and facial texture features at multiple scales. The deep attention feature extraction network extracts deep features from the abstract features and facial keypoints of the same scale of the image of the expression to be recognized using an attention machine. The shallow self-attention feature extraction network extracts shallow features from the facial edge features and facial texture features of the same scale of the image of the expression to be recognized using a self-attention machine. Finally, the multi-scale feature fusion classification network fuses and classifies the deep and shallow features of the image of the expression to be recognized at multiple scales to obtain the expression category to which the image of the expression to be recognized belongs.

Images