A facial expression recognition method based on multi-module collaborative optimization

Abstract

The application provides a facial expression recognition method based on multi-module collaborative optimization, has an efficient network structure, and can realize high-precision and strong-robust facial expression recognition in a complex environment. The method comprises the following steps: receiving an RGB face image and performing preprocessing to obtain a preprocessed image; inputting the preprocessed image into a hybrid feature network module to output a first feature map; inputting the first feature map into an efficient local attention mechanism module to output a second feature map; and inputting the second feature map into a classifier module to realize facial expression recognition.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a facial expression recognition method based on multi-module collaborative optimization. Background Technology

[0002] Facial Expression Recognition (FER) technology has been widely applied in various sectors of society, demonstrating significant value, particularly in healthcare, security, commerce, education, and manufacturing. In FER tasks, the first step is facial image preprocessing, including noise removal, edge smoothing, and contrast enhancement. Data augmentation techniques such as rotation, scaling, translation, and flipping can increase dataset diversity and improve model generalization. Pixel normalization, image resolution reduction, and image cropping can reduce computational and memory consumption, accelerating model training and inference. Cropping and filtering operations can improve model efficiency and accuracy by removing task-irrelevant information and extracting key regions. Color space conversion and histogram equalization can enhance image features and contrast, aiding model learning and understanding. The first step in facial expression recognition is image information extraction. This involves extracting image features, a process that retrieves meaningful information from the image. This plays a crucial role in facial expression recognition. Successfully extracting effective features representing classification information is essential for accurate expression recognition. Feature extraction methods can be categorized into five types: texture-based, statistical, motion-based, geometric, and deep learning-based. The final stage of facial expression recognition is classification. Classifiers are used to categorize expressions such as anger, disgust, fear, happiness, sadness, and surprise. Feature classification plays a key role in this process, automatically extracting the most discriminative and representative features from the original image. These features are crucial for distinguishing different facial expressions. By employing appropriate feature classification algorithms, recognition accuracy can be improved, leading to better facial expression recognition. Existing facial expression recognition methods include texture feature-based methods, geometric feature-based methods, and deep learning-based methods, with deep learning-based methods currently being the mainstream approach. A search of existing technical literature revealed a patent application with application number 202411802551.X, titled "Optimization Method, Device, and Electronic Equipment for Facial Expression Recognition Based on Deep Learning." This patent acquires facial expression data, constructs a three-dimensional facial model using 3D facial reconstruction technology based on the facial expression data, extracts facial features from the three-dimensional facial model using the SIFT algorithm, determines the corresponding expression labels for the facial features, constructs a deep learning model, and trains the deep learning model using facial features and expression labels as training samples to obtain a facial expression recognition model. It then uses at least the facial expression recognition model to recognize the facial expression to be identified and generates a recognition result. However, it suffers from insufficient accuracy and robustness in facial expression recognition under complex environments. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing methods, this invention proposes a facial expression recognition method based on multi-module collaborative optimization. This method features an efficient network structure and can achieve high-precision and robust facial expression recognition in complex environments. To address the aforementioned technical problems, this invention provides a facial expression recognition method based on multi-module collaborative optimization, the method comprising the following steps: S1: Receive RGB face images and perform preprocessing to obtain the preprocessed image; S2: The preprocessed image is input into the hybrid feature network module, and the first feature map is output. S3: The first feature map is input into the efficient local attention mechanism module, and the second feature map is output. S4: The second feature map is input into the classifier module to realize facial expression recognition.

[0004] Preferably, the hybrid feature network module includes: the preprocessed image sequentially passing through a first convolutional layer, a depthwise convolutional layer, a second convolutional layer, a channel attention mechanism module, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, and an eighth convolutional layer to obtain a first feature map.

[0005] Preferably, spatial feature extraction and transformation are performed on the image output by the second convolutional layer to obtain the first... Initial feature map of the channel The calculation expression is as follows:

[0006] in, Indicates the first Channel convolution kernel, This represents the image output by the second convolutional layer. Indicates the first Channel convolution kernel, Indicates the first The image output by the second convolutional layer of the channel. This represents the number of channels in the image output by the second convolutional layer. This represents the number of channels in the initial feature map.

[0007] Preferably, the channel attention mechanism module further includes: compression operation, activation operation, and weighted operation.

[0008] Preferably, the first Initial feature map of the channel The local statistical vector is obtained through compression. The calculation expression is as follows:

[0009] in, Indicates the first Initial feature map of the channel height, Indicates the first Initial feature map of the channel width, Indicates the first Initial feature map of the channel The upper position is The pixel value.

[0010] Preferably, based on all local statistical vectors Combining them yields the global statistical vector. The activation operation refers to the global statistical vector Passing through the first fully connected layer and the ReLU activation function in sequence The second fully connected layer and the Sigmoid activation function To obtain the channel weights The calculation expression is as follows:

[0011] in, This represents the weight matrix of the fully connected layer. This represents the weight matrix of the first fully connected layer. This represents the weight matrix of the second fully connected layer. Represents two fully connected layers and the ReLU activation function. Combinatorial operation functions.

[0012] Preferably, the weighting operation refers to the first... Channel weights Applied to the Initial feature map of the channel The recalibrated feature map is obtained. The calculation expression is as follows:

[0013] Among them, the recalibrated feature map Used as input to the third convolutional layer.

[0014] Preferably, the efficient local attention mechanism module uses two heads to extract the first feature map separately. Horizontal eigenvectors and vertical eigenvectors The calculation expression is as follows:

[0015]

[0016] in, Represents the first feature map height, Represents the first feature map width, Represents the first feature map The index values ​​of the horizontal and vertical feature vectors are processed by global normalization and the Sigmoid activation function to obtain the horizontal position attention prediction. Vertical position attention prediction Horizontal position attention prediction Vertical position attention prediction Multiply to obtain the position attention map. The calculation expression is as follows:

[0017] in, This represents the tensor outer product operation. This represents the learnable scaling factor. The transpose matrix is ​​used to apply the positional attention map to the first feature map. The enhanced feature map is obtained. The calculation expression is as follows:

[0018] in, This indicates channel-by-channel spatial attention weighting. This represents the learnable residual coefficient.

[0019] Preferably, the cross-entropy loss function is used throughout the entire method process. and attention loss function By combining the cross-entropy loss function and the attention loss function, the total loss function is obtained. The calculation expression is as follows:

[0020]

[0021]

[0022] in, Represents the total number of samples. Indicates the total number of emoji categories. This indicates that the model outputs the original logits vector. Represents the actual label category index. Indicates the first The first sample The logit vector of the class, , Let represent the output tensors of the first head of the efficient local attention mechanism module and the output tensor of the second head of the efficient local attention mechanism module, respectively. Indicates mean square error. This indicates the weight of attention loss in the overall loss.

[0023] Preferably, the classifier module includes: the second feature map is sequentially processed through a residual network, global average pooling, and a dynamic neuron discarding strategy to achieve facial expression recognition.

[0024] Compared with the prior art, the beneficial effects of the present invention are: This invention introduces a channel attention mechanism (SE module) to adaptively calibrate feature channel weights, suppressing high-dimensional redundant information and enhancing discriminative feature extraction of key facial expression regions (such as eyebrows, eyes, and corners of the mouth). This addresses the overfitting issues caused by the sensitivity of traditional texture features to image translation and rotation, as well as dimensional expansion. An efficient local attention mechanism (ELA) is designed, employing strip pooling operators to fuse long-range contextual features across regions, reducing the dependence on the accuracy of local keypoint localization and thus improving the model's robustness in partially occluded scenarios. During the classification stage, a residual network (ResNet), global average pooling, and a dynamic neuron dropout strategy are combined. A skip connection structure establishes a cross-layer identity mapping path to alleviate gradient decay, overcoming the shortcomings of traditional deep networks in capturing fine-grained features due to homogenized convolution operations and stagnant parameter updates. Ultimately, this achieves high-precision and robust facial expression recognition in complex environments. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of a facial expression recognition method based on multi-module collaborative optimization in one embodiment of the present invention; Figure 2 This is a schematic diagram of the workflow of the Facial Expression Recognition (MFM) method based on multi-module collaborative optimization in one embodiment of the present invention; Figure 3 This is a schematic diagram of the ELA mechanism of the facial expression recognition method based on multi-module collaborative optimization in one embodiment of the present invention; Figure 4 This is a schematic diagram of the ResNet facial expression recognition method based on multi-module collaborative optimization in one embodiment of the present invention; Figure 5This is a schematic diagram showing the recognition accuracy of a facial expression recognition method based on multi-module collaborative optimization in one embodiment of the present invention on four datasets (AffectNet-7 / 8, RAF-DB, FERPlus). Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0027] Example 1 like Figure 1 As shown, this embodiment proposes a facial expression recognition method based on multi-module collaborative optimization, the method comprising: S1: Receive RGB face images and perform preprocessing to obtain the preprocessed image; The specific implementation of this step is as follows: First, the RGB face images in the dataset are preprocessed and uniformly adjusted to 112×112 pixels. Then, the pixels are normalized, that is, the pixel values ​​are linearly normalized from 0,255 to 0,1. Data augmentation (horizontal flipping, random rotation, random erasure, and color jitter, etc.) is performed and then fed into the hybrid feature network module.

[0028] S2: The preprocessed image is input into the hybrid feature network module, and the first feature map is output. The specific implementation of this step is as follows: Initial feature extraction is performed on the preprocessed image in the Hybrid Feature Network (MFN) module. After passing through a convolutional kernel with a stride of 2, padding of 1, and size of 3, along with BatchNorm2d and PReLU functions, the image is transformed from 112×112×3 to 56×56×64. Then, the channels are grouped into 64 channels. Each channel is processed with a convolutional kernel with a stride of 1, padding of 1, and size of 3, along with BatchNorm2d and PReLU, resulting in an output of 56×56×64. The channel dimension is then increased to 128, passing through a convolutional kernel with a stride of 1, padding of 0, and size of 1, along with BatchNorm2d and PReLU. Finally, the channels are divided into three groups (64, 32, 32), and each group is convolved using a different kernel. Group 1 (64): (3×3 kernel, 2 stride, 1 padding); Group 2 (32): (convolution kernel 5×5, stride 2, padding 2); Group 3 (32): (7×7 kernel, stride 2, padding 3).

[0029] The three processed vectors are fused, and their size is transformed to 28×28×128. A CA attention mechanism is applied. First, pooling is applied to the input along the height and width directions, and the features of the height and width vectors are concatenated. Then, convolution is performed using a kernel with a stride of 1, padding of 0, and a size of 1, followed by BatchNorm2d and an activation function. Next, the height and width information are separated, and attention weights are generated for the height and width directions respectively, which are then applied. The dimensionality is reduced to 64 channels using point convolution (1×1 kernel, stride 1, padding 0), and then a channel attention mechanism (SE) module is applied. The channel attention mechanism (SE) module includes: First, spatial feature extraction and transformation are performed to provide initial feature maps for subsequent compression and activation operations. The calculation formula for this stage is as follows:

[0030] in, Indicates the first Channel convolution kernel, This represents the image output by the second convolutional layer. Indicates the first Channel convolution kernel, Indicates the first The image output by the second convolutional layer of the channel. This represents the number of channels in the image output by the second convolutional layer. This represents the number of channels in the initial feature map.

[0031] The compression module uses global average pooling to compress the spatial information (i.e., height and width) of each channel into a single value, namely a local statistical vector. fusion of local statistical vectors Obtain the global statistical vector This yields a feature vector containing global information for all channels. This vector has a dimension of 1×1×C, where C is the number of channels, as shown in the formula below:

[0032] in, Indicates the first Initial feature map of the channel height, Indicates the first Initial feature map of the channel width, Indicates the first Initial feature map of the channel The upper position is The pixels.

[0033] The activation operation will be compressed into the global statistical vector of the input. The channel weights are predicted using two fully connected (FC) layers. The first FC layer reduces the dimension of the feature vector from C to C / r (where r is the dimensionality reduction ratio) and then applies a ReLU activation function. The second FC layer restores the dimension from C / r back to C. Finally, the channel weights are obtained after applying a Sigmoid activation function. This yields the weight of each channel. These weights reflect the importance of each channel. The formula is shown below:

[0034] in, This represents the weight matrix of the fully connected layer. This represents the weight matrix of the first fully connected layer. This represents the weight matrix of the second fully connected layer. , Represents the set of real numbers. It is the dimensionality decay factor. That is the number of hidden nodes in the intermediate layer. Represents two fully connected layers and the ReLU activation function. and Sigmoid activation function Combinatorial operation functions.

[0035] The weighting operation assigns the weights of each channel obtained from the stimulus operation. Applied to the initial feature map, a recalibrated feature map is obtained by multiplying the features of each channel with their corresponding weights. This enhances the network's ability to respond to useful features, as shown in the formula below:

[0036] Among them, the recalibrated feature map Used as input to the third convolutional layer.

[0037] After the above convolution and SE operations, the image feature map is obtained as a 28×28×64 feature map, which is the image output by the channel attention mechanism module. Then it goes through 9 consecutive identical Mix_Depth_Wise modules (each module retains the residual). First, it goes through a single Mix_Depth_Wise module process, which is the third convolutional layer: First, the channels are enlarged to 128 dimensions using a convolutional kernel with a stride of 1, padding of 0, and size of 1. Then, BatchNorm2d and PReLU are applied. Subsequently, the resulting vector channels are divided into two groups (96, 32), and each group is convolved using a different convolutional kernel. Group 1 (96): (3×3 kernel, stride 1, padding 1); Group 2 (32): (convolution kernel 5×5, stride 1, padding 2).

[0038] The two processed vectors are fused for features and their size is transformed to 28×28×128. The CA attention mechanism is applied, and the same process as the previous application is used. The dimensionality is reduced to 64 channels through point convolution (1×1 kernel, stride 1, padding 0). Residual connections are applied, and the final output vector is 28×28×64, which is the image output by the third convolutional layer.

[0039] Then, after processing through the above 9 consecutive identical Mix_Depth_Wise modules, i.e. the fourth convolutional layer, a convolutional kernel with a stride of 1, padding of 0, and size of 1 is first used to increase the channel dimension to 256. Then, BatchNorm2d and PReLU are applied to divide the input channels into 3 groups (128, 64, 64), and convolution is performed on each group using a different convolutional kernel. Group 1 (128): (3×3 kernel, 2 stride, 1 padding); Group 2 (64): (convolution kernel 5×5, stride 2, padding 2); Group 3 (64): (7×7 kernel, stride 2, padding 3).

[0040] The three processed vectors are fused for features and their size is transformed to 14×14×256. The CA attention mechanism is applied, and the same process as the previous application is used. The dimensionality is reduced to 128 through point convolution (1×1 kernel, stride 1, padding 0). Residual connections are applied, and the final output vector is 14×14×128, which is the image output by the fourth convolutional layer.

[0041] After 16 consecutive identical Mix_Depth_Wise modules (each module retains residuals), the process first proceeds through a single Mix_Depth_Wise module, i.e., the fifth convolutional layer: First, the channels are enlarged to 256 dimensions using a convolutional kernel with a stride of 1, padding of 0, and size of 1. Then, BatchNorm2d and PReLU are applied. Subsequently, the resulting vector channels are divided into two groups (192, 64), and each group is convolved using a different convolutional kernel. Group 1 (192): (3×3 kernel, stride 1, padding 1); Group 2 (64): (convolution kernel 5×5, stride 1, padding 2).

[0042] The two processed vectors are fused for features and their size is transformed to 14×14×256. The CA attention mechanism is applied, and the same process as the previous application is used. The dimensionality is reduced to 128 through point convolution (1×1 kernel, stride 1, padding 0). Residual connections are applied, and the final output vector is 14×14×128, which is the image output by the fifth convolutional layer.

[0043] Then, after processing through the above 16 consecutive identical Mix_Depth_Wise modules, i.e. the sixth convolutional layer, the channels are first enlarged to 1024 using a convolutional kernel with a stride of 1, padding of 0, and size of 1. Then, BatchNorm2d and PReLU are applied to divide the input channels into 4 groups (256, 256, 256, 256), and each group is convolved using a different convolutional kernel. Group 1 (256): (3×3 kernel, 2 stride, 1 padding); The second group (256): (convolution kernel 5×5, stride 2, padding 2); Group 3 (256): (7×7 kernel, stride 2, padding 3); Group 4 (256): (9×9 kernel, stride 2, padding 4).

[0044] The four processed vectors are fused for features and their size is transformed to 7×7×1024. The CA attention mechanism is applied, and the same process as the previous application is used. The dimensionality is reduced to 256 through point convolution (1×1 kernel, stride 1, padding 0). Residual connections are applied, and the final output vector is 7×7×256, which is the image output by the sixth convolutional layer.

[0045] After six consecutive identical Mix_Depth_Wise modules (each module retains residuals), the process first proceeds through a single Mix_Depth_Wise module, i.e., the seventh convolutional layer: First, the channels are enlarged to 512 dimensions using a convolutional kernel with a stride of 1, padding of 0, and size of 1. Then, BatchNorm2d and PReLU are applied. Subsequently, the resulting vector channels are divided into 3 groups (172, 170, 170), and each group is convolved using a different convolutional kernel. Group 1 (172): (3×3 kernel, stride 1, padding 1); Group 2 (170): (convolution kernel 5×5, stride 1, padding 2); Group 2 (170): (7×7 kernel, stride 1, padding 2).

[0046] The three processed vectors are fused for features and their size is transformed to 7×7×512. The CA attention mechanism is applied, and the same process as the previous application is used. The dimensionality is reduced to 128 through point convolution (1×1 kernel, stride 1, padding 0). Residual connections are applied, and the final output vector is 7×7×256, which is the image output by the seventh convolutional layer.

[0047] Then, after processing by the above 6 consecutive identical Mix_Depth_Wise modules, i.e. the eighth convolutional layer, the channels are first increased to 512 dimensions using a convolutional kernel with a stride of 1, padding of 0, and size of 1. Then, BatchNorm2d and PReLU are applied, and the final output vector is 7×7×512, i.e. the first feature map. The discriminativeness of the feature channels is enhanced through the above operations.

[0048] S3: The first feature map is input into the efficient local attention mechanism module, and the second feature map is output. The specific implementation of this step is as follows: the feature vector after the above MFN processing is input into the dual-head ELA module, and a weighting operation is further performed on the feature vector. The specific operation is as follows: like Figure 3 As shown, the first feature map is first preserved for subsequent residual connections, and a dual-head ELA mechanism is applied to operate on the first feature map. The single ELA mechanism first separates the first feature map into height and width information, then applies a one-dimensional convolution with a stride of 1, padding of 3, and a size of 7 to each. Group normalization and sigmoid activation are then applied to generate attention weights in the range of 0-1. The weights in the height and width directions are then combined. The other ELA mechanism follows the same process. Finally, the larger feature map from the two heads is selected and combined with the input 7×7×512 first feature map through a residual operation to obtain a second feature map of size 7×7×512.

[0049] The ELA mechanism works as follows: The two heads first extract the feature vectors in the horizontal and vertical directions of the first feature map, respectively, as shown in the following formula:

[0050]

[0051] in, Represents the first feature map height, Represents the first feature map width, Represents the first feature map The index value, the strip pooling operator , By separating the orthogonal dimensions of space through dimensionality reduction projection, horizontal features can effectively capture lateral patterns such as changes in the shape of the corners of the mouth, while vertical features focus on vertical expression features such as raised eyebrows. By using a one-dimensional convolution kernel, ELA can effectively capture long-distance dependencies while reducing the interference of irrelevant regions on label prediction. This design enables ELA to extract rich target location information in all directions. In addition, ELA independently processes the feature vectors in each direction for attention prediction and fuses these features through multiplication operations to ensure that the accurate location information of the region of interest is preserved.

[0052] In the second step, a one-dimensional convolution is applied to perform local interaction between the two feature vectors. The kernel size can be adjusted as needed to control the coverage of the local interaction. The generated feature vectors are then processed by global normalization and a non-linear activation function to obtain positional attention predictions in the horizontal and vertical directions, respectively. Finally, by multiplying the positional attention in these two directions, a comprehensive positional attention map is obtained, as shown in the following formula:

[0053] in, This represents the tensor outer product operation. This represents the learnable scaling factor. ∈R, Represents the transpose matrix. The operation is normalized along the spatial dimension (i,j), and the positional attention map is applied to the first feature map. The enhanced feature map is obtained. The calculation expression is as follows:

[0054] in, , This indicates channel-by-channel spatial attention weighting. Indicates the learnable residual coefficients. ∈R.

[0055] Finally, the two feature vectors obtained from the two heads are combined with the larger feature vector. This feature vector is then combined with the first feature map of size 7×7×512 through a residual operation to obtain a second feature map of size 7×7×512.

[0056] S4: The second feature map is input into the classifier module to realize facial expression recognition.

[0057] The specific implementation of this step is as follows: First, a convolution with a stride of 2, padding of 1, and size of 3 is applied to the second feature map to increase the channel dimension to 1024. Simultaneously, the size of the second feature map is downsampled from 7×7 to 4×4. Batch normalization and ReLU activation are applied. Then, a second convolution with a stride of 1, padding of 1, and size of 3 is executed, keeping the size unchanged. The input is mapped to the same size through a shortcut connection (1×1 convolution). The two branches are summed and ReLU activation is applied. Global average pooling is performed on the 4×4×1024 vector, and the size is changed to 1×1×1024. Then, the vector is flattened, converting the three-dimensional vector to a two-dimensional vector. A fully connected layer reduces the feature dimension from 1024 to 512, and ReLU activation is applied to increase feature expressiveness. Then, half of the neurons are randomly discarded to prevent overfitting and improve the model's generalization ability. Finally, expression classification is performed.

[0058] Example 2 This embodiment further illustrates the facial expression recognition method based on multi-module collaborative optimization proposed in the above embodiments.

[0059] like Figure 2 As shown, the hybrid feature network module includes: the preprocessed image sequentially passes through a first convolutional layer, a depthwise convolutional layer, a second convolutional layer, a channel attention mechanism module, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, and an eighth convolutional layer to obtain a first feature map.

[0060] Spatial feature extraction and transformation are performed on the image output from the second convolutional layer to obtain the first... Initial feature map of the channel The calculation expression is as follows:

[0061] in, Indicates the first Channel convolution kernel, This represents the image output by the second convolutional layer. Indicates the first Channel convolution kernel, Indicates the first The image output by the second convolutional layer of the channel. This represents the number of channels in the image output by the second convolutional layer. This represents the number of channels in the initial feature map.

[0062] The channel attention mechanism module also includes: compression operation, incentive operation, and weighted operation.

[0063] The first Initial feature map of the channel The local statistical vector is obtained through compression. The calculation expression is as follows:

[0064] in, Indicates the first Initial feature map of the channel height, Indicates the first Initial feature map of the channel width, Indicates the first Initial feature map of the channel The upper position is The pixel value.

[0065] Based on all local statistical vectors Combining them yields the global statistical vector. The activation operation refers to the global statistical vector Passing through the first fully connected layer and the ReLU activation function in sequence The second fully connected layer and the Sigmoid activation function To obtain the channel weights The calculation expression is as follows:

[0066] in, This represents the weight matrix of the fully connected layer. This represents the weight matrix of the first fully connected layer. This represents the weight matrix of the second fully connected layer. Represents two fully connected layers and the ReLU activation function. Combinatorial operation functions.

[0067] The weighting operation refers to the first Channel weights Applied to the Initial feature map of the channel The recalibrated feature map is obtained. The calculation expression is as follows:

[0068] Among them, the recalibrated feature map Used as input to the third convolutional layer.

[0069] The efficient local attention mechanism module uses two heads to extract the first feature map separately. Horizontal eigenvectors and vertical eigenvectors The calculation expression is as follows:

[0070]

[0071] in, Represents the first feature map height, Represents the first feature map width, Represents the first feature map The index values ​​of the horizontal and vertical feature vectors are processed by global normalization and the Sigmoid activation function to obtain the horizontal position attention prediction. Vertical position attention prediction Horizontal position attention prediction Vertical position attention prediction Multiply to obtain the position attention map. The calculation expression is as follows:

[0072] in, This represents the tensor outer product operation. This represents the learnable scaling factor. The transpose matrix is ​​used to apply the positional attention map to the first feature map. The enhanced feature map is obtained. The calculation expression is as follows:

[0073] in, This indicates channel-by-channel spatial attention weighting. This represents the learnable residual coefficient.

[0074] The cross-entropy loss function was used throughout the entire methodology. and attention loss function By combining the cross-entropy loss function and the attention loss function, the total loss function is obtained. The calculation expression is as follows:

[0075]

[0076]

[0077] in, Represents the total number of samples. Indicates the total number of emoji categories. This indicates that the model outputs the original logits vector. Represents the actual label category index. Indicates the first The first sample The logit vector of the class, , Let represent the output tensors of the first head of the efficient local attention mechanism module and the output tensor of the second head of the efficient local attention mechanism module, respectively. Indicates mean square error. This indicates the weight of attention loss in the overall loss.

[0078] The classifier module includes: after the second feature map is sequentially processed through a residual network, global average pooling, and a dynamic neuron discarding strategy, facial expression recognition is achieved, such as... Figure 4 As shown, a ResNet-based residual network branch is constructed in the classification stage, and a skip connection structure is introduced to alleviate the gradient vanishing problem in deep networks. At the same time, a global average pooling layer is integrated to compress the feature dimension, and a dynamic neuron drop-off strategy is adopted to drop some neuron activation values ​​with probability during training, thereby improving the model's cross-library generalization ability to interference factors such as illumination and pose.

[0079] Example 3 This embodiment further illustrates the facial expression recognition method based on multi-module collaborative optimization proposed in the above embodiments.

[0080] This invention designs a facial expression recognition method based on multi-module collaborative optimization, the core technical path of which includes the following four parts: First, a channel attention mechanism (SE) is embedded in the hybrid feature network (MFN) backbone architecture. The spatial dimension is compressed by global average pooling (Squeeze operation) to generate channel descriptors. Then, two fully connected layers combined with the sigmoid activation function (Excitation operation) are used to dynamically learn the dependencies between channels and adaptively calibrate the channel weights of the feature map to enhance the feature expression of key facial expression regions (such as eyebrows, eyes, and corners of the mouth). Secondly, to address the issue of small inter-class differences and large intra-class differences in facial expressions, an efficient local attention mechanism (ELA) is designed. Strip pooling operators are used to extract long-range spatial context features in the horizontal and vertical directions, respectively. Bidirectional one-dimensional pooling is used to capture cross-regional semantic associations, and convolutional layers are used to fuse direction-sensitive feature vectors to generate a spatial attention mask. This suppresses background noise interference while enhancing the discriminative representation of subtle local facial changes (such as slightly drooping eyelids and tight lips). Furthermore, a ResNet-based residual network branch is constructed during the classification stage, and a skip connection structure is introduced to alleviate the gradient vanishing problem in deep networks. At the same time, a global average pooling layer is integrated to compress the feature dimension, and a dynamic neuron drop-off strategy is adopted to drop some neuron activation values ​​with probability during training, thereby improving the model's cross-library generalization ability to interference factors such as illumination and pose. Finally, the parameters of multiple modules are optimized through an end-to-end joint training strategy. The cross-entropy loss function is used to supervise the difference between the classifier output and the true label, driving the network to balance local detail perception and global semantic association during multi-level feature fusion. This achieves progressive learning from coarse-grained facial region localization to fine-grained expression attribute parsing. Figure 5 As shown, the method's robustness to complex facial expression patterns was finally verified on four datasets: AffectNet-7 / 8, RAF-DB, and FERPlus.

[0081] After implementing facial expression recognition through the classifier module, cross-entropy loss and attention loss functions are used to assist training. An end-to-end joint training strategy optimizes the parameters of multiple modules. The cross-entropy loss function is used to calculate the difference between the model's predicted class probability distribution and the true labels. For facial expression classification tasks, it outputs a scalar loss value, which guides the model to learn features that distinguish different facial expression categories during training. The formula is as follows:

[0082] in, Represents the total number of samples. Indicates the total number of emoji categories. This indicates that the model outputs the original logits vector. Represents the actual label category index. Indicates the first The first sample The class's logit vector.

[0083] The attention loss function is used to calculate the mean squared error (MSE) between different attention heads in an efficient local attention mechanism. The MSE of all head pairs (two attention heads are used here) is summed, and then the reciprocal is taken. A larger MSE (larger head differences) results in a smaller loss value; a smaller MSE (more similar heads) results in a larger loss value. This design encourages different attention heads to focus on different features, enhancing the diversity of the model. The formula is as follows:

[0084] in, , Let represent the output tensors of the first head of the efficient local attention mechanism module and the output tensor of the second head of the efficient local attention mechanism module, respectively. The mean squared error is represented by the two loss functions used together throughout the process. The total loss function formula is as follows:

[0085] in, This represents the weight of attention loss in the overall loss, set to 0.1.

[0086] Although the present invention has been described in detail through the preferred embodiments above, it should be understood that the above description should not be considered as a limitation of the present invention. Various modifications and substitutions to the present invention will be apparent to those skilled in the art after reading the above description. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A facial expression recognition method based on multi-module collaborative optimization, characterized in that, include: S1: Receive RGB face images and perform preprocessing to obtain the preprocessed image; S2: The preprocessed image is input into the hybrid feature network module, and the first feature map is output. The hybrid feature network module includes: the preprocessed image sequentially passes through a first convolutional layer, a depthwise convolutional layer, a second convolutional layer, a channel attention mechanism module, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, a seventh convolutional layer, and an eighth convolutional layer to obtain a first feature map; S3: The first feature map is input into the efficient local attention mechanism module, and the second feature map is output. The efficient local attention mechanism module includes: the first feature map is processed by the first efficient local attention mechanism module and the second efficient local attention mechanism module to obtain the first enhanced feature map and the second enhanced feature map. After comparing and selecting the first enhanced feature map and the second enhanced feature map, the selected enhanced feature map is combined with the first feature map through a residual operation to obtain the second feature map. The first efficient local attention mechanism module and the second efficient local attention mechanism module have the same structure. S4: The second feature map is input into the classifier module to realize facial expression recognition; The cross-entropy loss function was used throughout the entire methodology. and attention loss function Combined use of the obtained total loss function End-to-end joint training is performed to optimize the parameters of multiple modules, and the total loss function is... The calculation expression is as follows: in, Represents the total number of samples. Indicates the total number of emoji categories. This indicates that the model outputs the original logits vector. Represents the actual label category index. Indicates the first The first sample The logit vector of the class, , Let represent the output tensors after passing through the first efficient local attention mechanism module and the second efficient local attention mechanism module, respectively. Indicates mean square error. This indicates the weight of attention loss in the overall loss.

2. The facial expression recognition method based on multi-module collaborative optimization according to claim 1, characterized in that, The channel attention mechanism module includes: performing spatial feature extraction and transformation on the image output by the second convolutional layer to obtain the first... Initial feature map of the channel The calculation expression is as follows: in, Indicates the first Channel convolution kernel, This represents the image output by the second convolutional layer. Indicates the first Channel convolution kernel, Indicates the first The image output by the second convolutional layer of the channel. This represents the number of channels in the image output by the second convolutional layer. This represents the number of channels in the initial feature map.

3. The facial expression recognition method based on multi-module collaborative optimization according to claim 2, characterized in that, The channel attention mechanism module also includes: compression operation, incentive operation, and weighted operation.

4. The facial expression recognition method based on multi-module collaborative optimization according to claim 3, characterized in that, The first Initial feature map of the channel The local statistical vector is obtained through compression. The calculation expression is as follows: in, Indicates the first Initial feature map of the channel height, Indicates the first Initial feature map of the channel width, Indicates the first Initial feature map of the channel The upper position is The pixel value.

5. The facial expression recognition method based on multi-module collaborative optimization according to claim 4, characterized in that, Based on all local statistical vectors Combining them yields the global statistical vector. The activation operation refers to the global statistical vector Passing through the first fully connected layer and the ReLU activation function in sequence The second fully connected layer and the Sigmoid activation function To obtain the channel weights The calculation expression is as follows: in, This represents the weight matrix of the fully connected layer. This represents the weight matrix of the first fully connected layer. This represents the weight matrix of the second fully connected layer. Represents two fully connected layers and the ReLU activation function. Combinatorial operation functions.

6. The facial expression recognition method based on multi-module collaborative optimization according to claim 5, characterized in that, The weighting operation refers to the first Channel weights Applied to the Initial feature map of the channel The recalibrated feature map is obtained. The calculation expression is as follows: Among them, the recalibrated feature map Used as input to the third convolutional layer.

7. The facial expression recognition method based on multi-module collaborative optimization according to claim 1, characterized in that, The first efficient local attention mechanism module extracts the first feature map respectively. Horizontal eigenvectors and vertical eigenvectors The calculation expression is as follows: in, Represents the first feature map height, Represents the first feature map width, Represents the first feature map The index values ​​of the horizontal and vertical feature vectors are processed by global normalization and the Sigmoid activation function to obtain the horizontal position attention prediction. Vertical position attention prediction Horizontal position attention prediction Vertical position attention prediction Multiply to obtain the position attention map. The calculation expression is as follows: in, This represents the tensor outer product operation. This represents the learnable scaling factor. The transpose matrix is ​​used to apply the positional attention map to the first feature map. The enhanced feature map is obtained. The calculation expression is as follows: in, This indicates channel-by-channel spatial attention weighting. This represents the learnable residual coefficient.

8. The facial expression recognition method based on multi-module collaborative optimization according to claim 1, characterized in that, The classifier module includes: the second feature map is sequentially processed through a residual network, global average pooling, and a dynamic neuron discarding strategy to achieve facial expression recognition.

Images