A facial expression recognition method based on cross-feature type interaction and multi-scale fusion
By employing cross-feature type interaction and multi-scale fusion of RGB subnetwork and key point heatmap subnetwork in facial expression recognition, the problem of insufficient cross-feature type interaction in existing technologies is solved, thereby improving the robustness and accuracy of recognition.
Patent Information
- Application Number
- CN202211277059.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-18
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2042-10-18
AI Technical Summary
Existing deep learning-based facial expression recognition methods that rely on cross-feature type interaction and multi-scale fusion suffer from insufficient cross-feature type interaction or only one-way interaction during the feature extraction stage. This leads to inaccurate key point heatmaps, affecting the accuracy of the final recognition results.
Parallel RGB subnetworks and keypoint heatmap subnetworks are employed. Through cross-feature type interaction modules and multi-scale fusion recognition networks, cross-feature type interaction and multi-scale fusion of RGB image and keypoint heatmap feature vectors are achieved. Spatial attention weights and biases are generated using spatial attention mechanisms and cross-feature type cross-self-attention mechanisms to enhance and weight features. Finally, classification and recognition are performed through fully connected layers.
It improves the robustness and accuracy of facial expression recognition by extracting higher-quality features through cross-feature type complementarity, thereby enhancing recognition performance.
Smart Images

Figure CN115620366B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of image processing and computer vision technology, and in particular to a facial expression recognition method based on cross-feature type interaction and multi-scale fusion. Background Technology
[0002] Facial expressions are one of the most natural and direct ways for humans to express their emotions. Therefore, understanding human emotional states is a fundamental prerequisite for many computer vision tasks, including interpersonal interaction, driver fatigue monitoring, and healthcare. Although RGB-based cross-feature type interaction and multi-scale fusion facial expression recognition methods have made significant progress due to the rise of deep learning, single-feature type input methods based on cross-feature type interaction and multi-scale fusion still face many challenges, such as inter-class similarity, intra-class differences, and insufficient extraction of important features.
[0003] Currently, deep learning-based facial expression recognition methods that utilize cross-feature type interaction and multi-scale fusion are mainly based on single-stream network structures. A single-stream network structure refers to first using a backbone network to extract the feature vectors of RGB faces, then designing specific deep network modules based on the key point heatmaps generated from the RGB faces to generate corresponding attention weights, and then using these attention weights to weight and refine the feature vectors of the RGB faces. Finally, the refined RGB face feature vectors are fed into a fully connected layer to obtain the recognition result.
[0004] However, the key point positions reflected in the facial landmark heatmap generated from the RGB image of the face are not necessarily completely accurate. Therefore, in existing deep learning-based facial expression recognition methods that rely on cross-feature type interaction and multi-scale fusion, the attention weights generated from the landmark heatmap are easily affected by the inaccuracy of the key point positions in the landmark heatmap. In other words, current deep learning-based facial expression recognition methods that rely on cross-feature type interaction and multi-scale fusion either lack cross-feature type interaction in the feature extraction stage or only have unidirectional interaction to obtain RGB features from the landmark features. This makes the features finally extracted by the deep network easily affected by the inaccuracy of the landmark heatmap, thus further affecting the accuracy of the final recognition result. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of existing deep learning-based facial expression recognition methods that rely on cross-feature type interaction and multi-scale fusion. These methods either lack cross-feature type interaction during the feature extraction stage or only have a one-way interaction from key point features to RGB features. This makes the features extracted by the deep network susceptible to the inaccuracy of the key point heatmap, which further affects the accuracy of the final recognition result. The invention provides a facial expression recognition method based on cross-feature type interaction and multi-scale fusion.
[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:
[0007] A facial expression recognition method based on cross-feature type interaction and multi-scale fusion includes the following steps:
[0008] S1: Obtain the RGB image of the face whose expression is to be recognized, and generate a key point heatmap from the RGB image;
[0009] S2: Input the RGB image and the key point heatmap into the pre-trained facial expression recognition model to obtain the facial expression recognition result;
[0010] The facial expression recognition model includes: a feature extraction network and a multi-scale fusion recognition network across feature types;
[0011] The feature extraction network includes: a parallel RGB subnetwork and a key point heatmap subnetwork, and multiple cross-feature type interaction modules inserted hierarchically between the RGB subnetwork and the key point heatmap subnetwork;
[0012] The cross-feature type multi-scale fusion recognition network includes: a cross-feature type multi-scale fusion module and a fully connected layer classification and recognition module;
[0013] The cross-feature type interaction module receives the feature vectors output by the i-th layer of the RGB sub-network and the keypoint heatmap sub-network, and performs cross-feature type interaction on the received feature vectors to obtain cross-feature type interaction features. At the same time, it outputs the cross-feature type interaction features to the (i+1)-th layer of the RGB sub-network and the keypoint heatmap sub-network. Then, the multi-layer cross-feature type interaction features obtained by the cross-feature type interaction module from the RGB sub-network and the keypoint heatmap sub-network are output to the cross-feature type multi-scale fusion recognition module. The cross-feature type fusion network and the fully connected layer classification recognition network process the received cross-feature type interaction features in sequence to finally obtain the facial expression recognition result.
[0014] Where i = 0, 1...n; n is the number of network layers of the RGB subnetwork and the key point heatmap subnetwork.
[0015] Using the above technical solution, when extracting feature vectors from RGB images and key point heatmaps, there is cross-feature type interaction and multi-scale fusion, which enables the two feature types to complement each other better. Compared with deep networks that only have one-way interaction or no interaction in the feature extraction stage, the facial expression recognition network of the present invention is more robust, thereby effectively improving the facial expression recognition performance.
[0016] As a preferred embodiment of the present invention, step S2, performing cross-feature type interaction on the received feature vector, includes the following steps:
[0017] S21: The cross-feature type interaction module generates corresponding spatial attention weights and biases using a spatial attention mechanism based on the received feature vector, and generates a spatial attention-enhanced feature vector by combining the spatial attention weights and the biases with the feature vector.
[0018] S22: The spatial attention-enhanced feature vector is weighted by the global spatial dimension through a cross-feature type cross-self-attention mechanism to obtain the next layer of cross-feature type interactive features.
[0019] In a preferred embodiment of the present invention, in step S21, the formulas for generating the spatial attention weights and the bias are as follows:
[0020]
[0021]
[0022]
[0023]
[0024] in, This refers to the feature vector output by the i-th layer of the RGB sub-network. This refers to the feature vector output by the i-th layer of the keypoint heatmap subnetwork. and These represent the convolutional layer used for RGB feature vectors and the convolutional layer used for keypoint feature vectors, respectively. and According to The generated attention weights and biases; and According to The generated spatial attention weights and biases;
[0025] The enhancement formula for generating spatial attention-enhanced feature vectors is as follows:
[0026]
[0027]
[0028] in, For RGB feature vectors The feature vector output from the first convolution; For key point feature vectors The feature vector output from the first convolution; ReLU is the linear correction unit activation function. Feature vectors for spatial attention enhancement.
[0029] As a preferred embodiment of the present invention, the cross-feature type self-attention processing in step S22 is as follows:
[0030]
[0031]
[0032]
[0033]
[0034] in, Convolutional layers For RGB feature vectors The feature vector output after convolution; Convolutional layers Feature vectors of key points The feature vector output after convolution; This is an interactive feature across feature types.
[0035] As a preferred embodiment of the present invention, in step S2, the cross-feature type interaction features are output to the cross-feature type multi-scale fusion recognition module. The cross-feature type fusion network and the fully connected layer classification recognition network process the received cross-feature type interaction features in sequence to finally obtain the facial expression recognition result, including the following steps:
[0036] S201: The cross-feature fusion recognition module performs max pooling on the cross-feature type interaction features output by the cross-feature type interaction module for the 0th, 1st, ..., ith, and i+1th layers of the RGB subnetwork and the key point heatmap subnetwork, respectively. Then, the cross-feature type interaction features after max pooling of the first i layers are concatenated with the cross-feature type interaction features output by the i+1th layer on the channel, and then output multi-scale fusion features after passing through a convolutional layer.
[0037] S202: Perform cross-feature type fusion on the multi-scale fusion features of the received RGB image and the key point heatmap to obtain cross-feature type multi-scale fusion features;
[0038] S203: The multi-scale fusion features across feature types are sequentially passed through an average pooling layer and a fully connected layer to finally output the recognition result of facial expressions.
[0039] In a preferred embodiment of the present invention, in step S201, the formula for outputting the multi-scale fusion feature is:
[0040]
[0041]
[0042] in, For cross-feature type interaction features from multiple scales; MaxPool represents the max pooling operation, used to align feature space resolutions at different scales; [·, ·] represents the channel concatenation operation; conv r6 conv L6 This is a convolutional network used to reduce and fuse multi-scale cross-feature type interactive features concatenated across channels; i = 0, 1...n; n is the number of network layers in the RGB sub-network and the keypoint heatmap sub-network; f r2 f L2 This is a multi-scale fusion feature.
[0043] As a preferred embodiment of the present invention, in step S202, the formula for cross-feature type fusion of the multi-scale fusion features of the received RGB image and key point heatmap is as follows:
[0044] σ r =sigmoid((W fc2 )T((W fc1 ) T [AvgPool(f r2 ), AvgPool(f L2 )]))
[0045] σ L =sigmoid((W fc3 ) T ((W fc1 ) T [AvgPool(f r2 ), AvgPool(f L2 )]))
[0046] f3=σ r ×f r2 +σL ×f L2
[0047] Among them, f r2 f L2 For multi-scale fusion features; AvgPool represents the average pooling operation, used to reduce the dimensionality of multi-scale fusion features; [·, ·] represents the channel concatenation operation; (W fc1 ) T 、(W fc2 ) T 、(W fc3 ) T A fully connected layer for dimensionality reduction and fusion of dimensionality-reduced feature vectors from concatenated channels; σ r σ L These represent the multi-scale fusion features f respectively. r2 f L2 The generated attention weights; f3 is a cross-feature type fusion feature.
[0048] In a preferred embodiment of the present invention, in step S203, the formula for outputting the recognition result of facial expression is:
[0049] Expression = Soft max((W fc ) T AvgPool(f3))
[0050] Where AvgPool represents the average pooling operation, used to reduce the dimensionality of the aforementioned cross-feature type fused feature f3; W fc is a fully connected layer used to classify the extracted feature vectors; Soft max is a normalization exponential function used to map each element of the network output vector to the range [0, 1]; Expression is the network output of the recognition result of facial expression.
[0051] As a preferred embodiment of the present invention, the facial expression recognition model is trained using a multi-class cross-entropy loss function, the formula of which is:
[0052]
[0053] Where N is the total number of training samples, M is the total number of categories in the samples, and y ic Let p be the label value of the i-th sample. ic Let be the network's prediction for the i-th sample, and log is generally the natural logarithm.
[0054] On the other hand, a facial expression recognition system based on cross-feature type interaction and multi-scale fusion is disclosed, including at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the method described above.
[0055] Compared with the prior art, the beneficial effects of the present invention are as follows: when extracting feature vectors from RGB images and key point heatmaps, there is cross-feature type interaction and multi-scale fusion, which enables the two feature types to complement each other better. Compared with deep networks that only have one-way interaction or no interaction in the feature extraction stage, the facial expression recognition network of the present invention is more robust, thereby effectively improving the facial expression recognition performance. Attached Figure Description
[0056] Figure 1 This is a flowchart of a facial expression recognition method based on cross-feature type interaction and multi-scale fusion as described in Embodiment 1 of the present invention;
[0057] Figure 2 This is a schematic diagram of the network structure of a facial expression recognition method based on cross-feature type interaction and multi-scale fusion as described in Embodiments 1 and 2 of the present invention;
[0058] Figure 3 This is a detailed diagram of the cross-feature type interaction module CFIM network structure of a facial expression recognition method based on cross-feature type interaction and multi-scale fusion as described in Embodiment 2 of the present invention.
[0059] Figure 4 This is a detailed diagram of the cross-feature type multi-scale fusion recognition module CFFM network structure of a facial expression recognition method based on cross-feature type interaction and multi-scale fusion as described in Embodiment 2 of the present invention;
[0060] Figure 5 This is a structural block diagram of a facial expression recognition device based on cross-feature type interaction and multi-scale fusion as described in Embodiment 3 of the present invention. Detailed Implementation
[0061] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.
[0062] Example 1
[0063] A facial expression recognition method based on cross-feature type interaction and multi-scale fusion, such as Figure 1 As shown, it includes the following steps:
[0064] S1: Obtain the RGB image of the face whose expression is to be recognized, and generate a key point heatmap from the RGB image. Then, convert the key point heatmap into a three-channel image by channel copying, and scale both the RGB image and the key point heatmap to a size of 224×224.
[0065] S2: Input the RGB image and the key point heatmap into the pre-trained facial expression recognition model to obtain the facial expression recognition result;
[0066] like Figure 2 As shown, the facial expression recognition model includes: a feature extraction network and a multi-scale fusion recognition network across feature types;
[0067] The feature extraction network includes: a parallel RGB subnetwork and a key point heatmap subnetwork, and multiple cross-feature type interaction modules inserted hierarchically between the RGB subnetwork and the key point heatmap subnetwork;
[0068] The cross-feature type multi-scale fusion recognition network includes: a cross-feature type multi-scale fusion module and a fully connected layer classification and recognition module;
[0069] The cross-feature type interaction module receives the feature vectors output by the i-th layer of the RGB sub-network and the keypoint heatmap sub-network, and performs cross-feature type interaction on the received feature vectors to obtain cross-feature type interaction features. At the same time, it outputs the cross-feature type interaction features to the (i+1)-th layer of the RGB sub-network and the keypoint heatmap sub-network. Then, the multi-layer cross-feature type interaction features obtained by the cross-feature type interaction module from the RGB sub-network and the keypoint heatmap sub-network are output to the cross-feature type multi-scale fusion recognition module. The cross-feature type fusion network and the fully connected layer classification recognition network process the received cross-feature type interaction features in sequence to finally obtain the facial expression recognition result.
[0070] Where i = 0, 1...n; n is the number of network layers of the RGB subnetwork and the key point heatmap subnetwork.
[0071] In step S2, the cross-feature type interaction of the received feature vector includes the following steps:
[0072] S21: The cross-feature type interaction module generates corresponding spatial attention weights and biases using a spatial attention mechanism based on the received feature vector. The spatial attention weights and biases are then used with the feature vector to generate a spatial attention-enhanced feature vector, highlighting the more critical parts of the feature vector in spatial location.
[0073] S22: The spatial attention-enhanced feature vector is weighted globally in terms of spatial dimension using a cross-feature type self-attention mechanism to obtain the next hierarchical feature. The cross-self-attention mechanism fully facilitates cross-feature type interaction by exchanging the query values (q) generated by the two features, allowing them to better complement each other.
[0074] In step S21, the formulas for generating the spatial attention weights and the bias are as follows:
[0075]
[0076]
[0077]
[0078]
[0079] in, This refers to the feature vector output by the i-th layer of the RGB sub-network. This refers to the feature vector output by the i-th layer of the keypoint heatmap subnetwork. and These represent the convolutional layer used for RGB feature vectors and the convolutional layer used for keypoint feature vectors, respectively. and According to The generated attention weights and biases; and According to The generated spatial attention weights and biases;
[0080] The enhancement formula for generating spatial attention-enhanced feature vectors is as follows:
[0081]
[0082]
[0083] in, For RGB feature vectors The feature vector output from the first convolution; For key point feature vectors The feature vector output from the first convolution; ReLU is the linear correction unit activation function. Feature vectors for spatial attention enhancement.
[0084] The cross-feature type self-attention processing procedure in step S22 is as follows:
[0085]
[0086]
[0087]
[0088]
[0089] in, Convolutional layers For RGB feature vectors The feature vector output after convolution; Convolutional layers Feature vectors of key points The feature vector output after convolution; This is an interactive feature across feature types.
[0090] In step S2, the cross-feature type interaction features are output to the cross-feature type multi-scale fusion recognition module. The cross-feature type fusion network and the fully connected layer classification recognition network process the received cross-feature type interaction features sequentially to obtain the facial expression recognition result, which includes the following steps:
[0091] S201: The cross-feature fusion recognition module performs max pooling on the cross-feature type interaction features output by the cross-feature type interaction module for the 0th, 1st, ..., ith, and i+1th layers of the RGB subnetwork and the key point heatmap subnetwork, respectively. Then, the cross-feature type interaction features after max pooling of the first i layers are concatenated with the cross-feature type interaction features output by the i+1th layer on the channel, and then output multi-scale fusion features after passing through a convolutional layer.
[0092] S202: Perform cross-feature type fusion on the multi-scale fusion features of the received RGB image and the key point heatmap to obtain cross-feature type multi-scale fusion features, which mainly rely on improved channel attention.
[0093] S203: The multi-scale fusion features across feature types are sequentially passed through an average pooling layer and a fully connected layer to finally output the recognition result of facial expressions.
[0094] In step S201, the formula for outputting the multi-scale fusion feature is:
[0095]
[0096]
[0097] in, For cross-feature type interaction features from multiple scales; MaxPool represents the max pooling operation; [·, ·] represents the channel concatenation operation; conv r6 conv L6 This is a convolutional network used to reduce and fuse multi-scale cross-feature type interactive features concatenated across channels; i = 0, 1...n; n is the number of network layers in the RGB sub-network and the keypoint heatmap sub-network; f r2 f L2 This is a multi-scale fusion feature.
[0098] In step S202, the formula for cross-feature type fusion of the multi-scale fusion features of the received RGB image and the key point heatmap is as follows:
[0099] σ r =sigmoid((W fc2 ) T ((W fc1 ) T [AvgPool(f r2 ), AvgPool(f L2 )]))
[0100] σ L =sigmoid((W fc3 ) T ((W fc1 ) T [AvgPool(f r2 ), AvgPool(f L2 )]))
[0101] f3=σ r ×f r2 +σ L ×f L2
[0102] Among them, f r2 f L2 For multi-scale fusion features; AvgPool represents the average pooling operation, used to reduce the dimensionality of multi-scale fusion features; [·, ·] represents the channel concatenation operation; (W fc1 ) T 、(W fc2 ) T 、(W fc3 ) T A fully connected layer for dimensionality reduction and fusion of dimensionality-reduced feature vectors from concatenated channels; σ r σ L These represent the multi-scale fusion features f respectively. r2 f L2 The generated attention weights; f3 is a cross-feature type fusion feature.
[0103] In step S203, the formula for outputting the facial expression recognition result is:
[0104] Expression = Soft max((W fc ) T AvgPool(f3))
[0105] Where AvgPool represents the average pooling operation, used to reduce the dimensionality of the aforementioned cross-feature type fused feature f3; W fc is a fully connected layer used to classify the extracted feature vectors; Soft max is a normalization exponential function used to map each element of the network output vector to the range [0, 1]; Expression is the network output of the recognition result of facial expression.
[0106] Using the above technical solution, the cross-feature type interaction module enables the features of the RGB image and the features of the key point heatmap to fully utilize the complementarity between different feature types in the feature extraction stage, thereby extracting higher quality robust features. Furthermore, this invention employs a cross-feature type fusion module to subsequently fuse the hierarchical features of multiple scales output by the cross-feature type interaction module and output the final expression recognition result.
[0107] Example 2
[0108] This embodiment is a specific implementation of embodiment 1;
[0109] The RGB subnetwork and keypoint heatmap subnetwork in this embodiment are based on the common ResNet-50 network structure. The RGB subnetwork and the keypoint heatmap subnetwork are composed of two parallel ResNet-50 networks. The ResNet-50 in this embodiment has 0 to n (n=4) layers, namely: Conv-head, Residual-i (i=1,...,4), and 5 cross-feature type interaction modules (CFIM). i (i = 0, ..., 4) are inserted into layers 0-4 of the two sub-networks to achieve cross-feature type interaction.
[0110] Specifically, the outputs of layers 0-4 of the RGB subnetwork and the key point heatmap subnetwork are: The number of channels are c0=64, c1=64, c2=128, c3=256, and c4=512. The input is fed into the corresponding cross-feature type interaction module CFIM. i In the i-th layer of CFIM i according to After performing bidirectional interaction across feature types, the hierarchical features of the i-th level are output to the max pooling layer. Simultaneously, the hierarchical features of the i-th level are used as input to the (i+1)-th layer of the RGB subnetwork and the keypoint heatmap subnetwork, as follows: Figure 4 As shown, a set of hierarchical features across feature types can be obtained through five cross-feature type interaction modules (CFIM).
[0111] The cross-feature type interaction module is mainly implemented using spatial attention and cross-self-attention. For example... Figure 3 As shown, the specific operation of the cross-feature type interaction module is as follows: the output of the i-th layer of the RGB sub-network and the key point heatmap sub-network is... and As input, attention weights are generated using spatial attention mechanism according to equation (1). and bias
[0112]
[0113]
[0114]
[0115]
[0116] in, and This indicates a convolution kernel size of 3, a stride of 1, zero padding of 1, and c output channels. i Convolutional layers; and This indicates a convolution kernel size of 3, a stride of 1, and 2c output channels. i The convolutional layer; W and B have the same scale as sf, and the number of channels is c. i (i = 0, ..., 4).
[0117] Then, calculate the feature vector enhanced by spatial attention according to equation (2).
[0118]
[0119]
[0120] in, For the feature vector of the RGB image The feature vector output from the first convolution; For key point feature vectors The feature vector output from the first convolution; ReLU is the activation function of the linear correction unit.
[0121] Then, calculate the cross-feature type interaction features of the cross-feature type cross self-attention processing output according to equation (3).
[0122]
[0123]
[0124]
[0125]
[0126] in, and Both indicate that the kernel size is 1, the stride is 1, the zero-padding is 0, and the number of output channels is c. i The convolutional layer.
[0127] The RGB image and keypoint heatmap described in step S1 are input into the feature extraction stage to obtain a set of cross-feature type interaction features of the RGB image and keypoint heatmap. Their feature resolutions are: 112×112×64, 56×56×64, 28×28×128, 14×14×256, and 7×7×512, respectively. Figure 4 As shown.
[0128] Multiple hierarchical features across feature types obtained from S2 The input is fed into the cross-feature type multi-scale fusion recognition stage of the design to obtain the recognition result. The specific steps are as follows:
[0129] A1, such as Figure 4 As shown, the cross-feature type interaction feature vectors output by the RGB subnetwork and the keypoint heatmap subnetwork at layers 0, 1, 2, and 3 are... Max pooling is performed according to equation (4) to unify these multi-scale feature vectors to a 7×7 scale. Then, they are concatenated with the cross-feature type interaction feature vectors output from layer 4 on the channel, and then passed through a convolutional layer to output the multi-scale fused feature f. r2 and f L2 .
[0130]
[0131]
[0132] Where [·, ·] denotes channel concatenation; the resolution of the feature vector output after channel concatenation is 7×7×1024, conv r6 and convL6 Both represent convolutional layers with a kernel size of 1, a stride of 1, zero padding of 0, and 512 output channels; respectively processed by conv r6 and conv L6 The multi-scale fused feature vector f output after two convolutions r2 and f L2 The resolution of all of them is 7×7×512.
[0133] A2, such as Figure 4 As shown, the multi-scale fusion features f are calculated according to formula (5). r2 and f L2 The input is the cross-feature type fusion module CFFM, which outputs the cross-feature type fused feature f3.
[0134] σ r =sigmoid((W fc2 ) T ((W fc1 ) T [AvgPool(f r2 ), AvgPool(f L2 )]))
[0135] σ L =sigmoid((W fc3 ) T ((W fc1 ) T [AvgPool(f r2 ), AvgPool(f L2 )]))
[0136] f3=σ r ×f r2 +σ L ×f L2 (5)
[0137] Where AvgPool represents the average pooling operation; [·, ·] represents the channel concatenation operation; (W fc1 ) T 、(W fc2 ) T 、(W fc3 ) T The weight matrices are 1024×256, 256×512, and 256×512 respectively; Sigmoid represents the Sigmoid activation function; attention weights σ r σ L The resolution of each feature is 1×1×512; the resolution of the final cross-feature type fusion feature f3 is 7×7×512.
[0138] A3, then classify the cross-feature type fusion feature f3 according to formula (6) to output the recognition result of facial expression.
[0139] Expression = Soft max((W fc ) T AvgPool(f3)) (6)
[0140] Where AvgPool represents the average pooling operation; W fc The weight matrix is 512×7; Soft max is a normalized exponential function that maps each element of the network output vector to the range [0, 1]; Expression is the network output of the facial expression recognition result, which is a 7-dimensional feature vector.
[0141] A4. Before using the network, the facial expression recognition model is trained, the result is calculated and the multi-class cross-entropy loss function of the true result is compared and the error is backpropagated, and gradient descent optimization is performed, that is, the neural network is trained according to equation (7).
[0142]
[0143] Where N is the total number of training samples, M is the total number of categories in the samples, and y ic Let p be the label value of the i-th sample. ic Let be the network's prediction for the i-th sample, and log is generally the natural logarithm.
[0144] Example 3
[0145] like Figure 5 As shown, a facial expression recognition device based on cross-feature type interaction and multi-scale fusion includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform a method described in the foregoing embodiments. The input / output interface may include a display, keyboard, mouse, and USB interface for inputting and outputting data; a power supply provides electrical power to the electronic device.
[0146] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.
[0147] When the integrated units of this invention are implemented as software functional units and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.
[0148] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A face expression recognition method based on cross-feature type interaction and multi-scale fusion, characterized in that, Includes the following steps: S1: Obtain the RGB image of the face whose expression is to be recognized, and generate a key point heatmap from the RGB image; S2: Input the RGB image and the key point heatmap into the pre-trained facial expression recognition model to obtain the facial expression recognition result; The facial expression recognition model includes: a feature extraction network and a multi-scale fusion recognition network across feature types; The feature extraction network includes: a parallel RGB subnetwork and a key point heatmap subnetwork, and multiple cross-feature type interaction modules inserted hierarchically between the RGB subnetwork and the key point heatmap subnetwork; The cross-feature type multi-scale fusion recognition network includes: a cross-feature type multi-scale fusion module and a fully connected layer classification and recognition module; The cross-feature type interaction module receives the feature vectors output by the i-th layer of the RGB sub-network and the keypoint heatmap sub-network, and performs cross-feature type interaction on the received feature vectors to obtain cross-feature type interaction features. At the same time, it outputs the cross-feature type interaction features to the (i+1)-th layer of the RGB sub-network and the keypoint heatmap sub-network. Then, the multi-layer cross-feature type interaction features obtained by the cross-feature type interaction module from the RGB sub-network and the keypoint heatmap sub-network are output to the cross-feature type multi-scale fusion recognition network. The cross-feature type multi-scale fusion module and the fully connected layer classification and recognition module process the received cross-feature type interaction features in sequence to finally obtain the facial expression recognition result. Where i = 0, 1, ..., n; n is the number of network levels of the RGB subnetwork and the key point heatmap subnetwork; In step S2, the cross-feature type interaction features are output to the cross-feature type multi-scale fusion recognition network. The cross-feature type multi-scale fusion module and the fully connected layer classification and recognition module process the received cross-feature type interaction features in sequence to finally obtain the facial expression recognition result, including the following steps: S201: The cross-feature type multi-scale fusion module performs max pooling on the cross-feature type interaction features output by the cross-feature type interaction module for the RGB sub-network and the key point heatmap sub-network at layers 0, 1, ..., n respectively. Then, the cross-feature type interaction features after max pooling from layers 0 to n-1 are concatenated with the cross-feature type interaction features output from layer n on the channel, and then output the multi-scale fusion feature after passing through a convolutional layer. S202: Perform cross-feature type fusion on the multi-scale fusion features of the received RGB image and the key point heatmap to obtain cross-feature type multi-scale fusion features; S203: The multi-scale fusion features across feature types are sequentially passed through an average pooling layer and a fully connected layer to finally output the recognition result of facial expressions.
2. The facial expression recognition method based on cross-feature type interaction and multi-scale fusion according to claim 1, characterized in that, In step S2, the cross-feature type interaction of the received feature vector includes the following steps: S21: The cross-feature type interaction module generates corresponding spatial attention weights and biases using a spatial attention mechanism based on the received feature vector, and generates a spatial attention-enhanced feature vector by combining the spatial attention weights and the biases with the feature vector. S22: The spatial attention-enhanced feature vector is weighted by the global spatial dimension through a cross-feature type cross self-attention mechanism to obtain cross-feature type interactive features.
3. The facial expression recognition method based on cross-feature type interaction and multi-scale fusion according to claim 2, characterized in that, In step S21, the formulas for generating the spatial attention weights and the bias are as follows: in, This refers to the feature vector output by the i-th layer of the RGB sub-network. This refers to the feature vector output by the i-th layer of the key point heatmap subnetwork. and j=1, 2, representing the convolutional layer used for RGB feature vectors and the convolutional layer used for keypoint feature vectors, respectively; and According to The generated attention weights and biases; and According to The generated spatial attention weights and biases; The enhancement formula for generating spatial attention-enhanced feature vectors is as follows: in, For RGB feature vectors The feature vector output from the first convolution; For key point feature vectors The feature vector output from the first convolution; The activation function is a linear correction unit. , These are the RGB feature vectors and keypoint feature vectors enhanced with spatial attention, respectively.
4. The facial expression recognition method based on cross-feature type interaction and multi-scale fusion according to claim 3, characterized in that, The cross-feature type self-attention processing procedure in step S22 is as follows: in, , , Convolutional layers , , For RGB feature vectors The feature vector output after convolution; , , Convolutional layers , , Feature vectors of key points The feature vector output after convolution; , These are the cross-feature type interaction features of the i-th layer.
5. The facial expression recognition method based on cross-feature type interaction and multi-scale fusion according to claim 4, characterized in that, In step S201, the formula for outputting the multi-scale fusion feature is: in, , These are cross-feature type interaction features from multiple scales; i = 0, 1, 2, 3, 4; This indicates the max pooling operation, used to align feature space resolutions at different scales; Indicates channel serialization operation; , This is a convolutional method for channel reduction and fusion of multi-scale cross-feature type interactive features connected in channels; , This is a multi-scale fusion feature.
6. The facial expression recognition method based on cross-feature type interaction and multi-scale fusion according to claim 4, characterized in that, In step S202, the formula for cross-feature type fusion of the multi-scale fusion features of the received RGB image and key point heatmap is as follows: in, , It is a multi-scale fusion feature; This indicates the average pooling operation, used for dimensionality reduction of multi-scale fused features; Indicates channel serialization operation; , , A fully connected layer for dimensionality reduction and fusion of dimensionality-reduced feature vectors from concatenated channels; , These respectively represent the characteristics based on multi-scale fusion. , The generated attention weights; It is a multi-scale fusion feature across feature types.
7. The facial expression recognition method based on cross-feature type interaction and multi-scale fusion according to claim 4, characterized in that, In step S203, the formula for outputting the facial expression recognition result is: in, This represents the average pooling operation, used for multi-scale fusion of features across the aforementioned feature types. Perform dimensionality reduction; This is a fully connected layer used to classify the extracted feature vectors; It is a normalized exponential function used to map each element of the network output vector to the range [0, 1]. This refers to the facial expression recognition results output by the network.
8. The facial expression recognition method based on cross-feature type interaction and multi-scale fusion according to claim 1, characterized in that, The facial expression recognition model is trained using a multi-class cross-entropy loss function, as shown in the formula: in The total number of training samples, The total number of categories in the sample. For the first t The label value of each sample, For the network to the first t The predicted value for each sample, where log is the natural logarithm.
9. A facial expression recognition system, characterized in that, It includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.
Citation Information
Patent Citations
Pedestrian re-identification method based on attention mechanism
CN113158905A
Object emotion analysis method and device and electronic equipment
CN115171176A
