Face expression recognition method based on cross-layer multi-scale channel mutual attention learning

Abstract

The face expression recognition method based on cross-layer multi-scale channel mutual attention learning belongs to the technical fields of image processing and pattern recognition, and the CMSCMAL-Net model is used to complete the recognition of face expressions in a natural scene, and the method has good robustness in the case that the expression is blurred or there is an occlusion.The multi-scale channel attention mechanism is introduced to improve the backbone network, the fine details in the low-level features are retained and highlighted on the basis of the combination of local features and global features, the backbone network is divided into different stages from shallow to deep, the shallow stage learns low-level detail information, the deep stage learns high-level abstract semantic information, and the progressive multi-step strategy is adopted for training, so that different stages can complement information, and the recognition effect of the model is improved.The present application provides a fine-grained direction model for face expression recognition, which can better capture the subtle changes of face expressions and improve the recognition rate of face expressions in a natural scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition processing and pattern recognition, and particularly relates to a face expression recognition method based on cross-layer multi-scale channel mutual attention learning. Background Art

[0002] Artificial intelligence technology mainly studies how to use machines to simulate humans and realize and extend human thinking modes, which makes the work of understanding and simulating human emotions a hot topic. In the most important human-computer interaction link, if a machine wants to understand and analyze human emotions and psychological activities, it must first be able to accurately recognize human expressions. Facial expression is one of the most natural, powerful and common signals for humans to convey their emotional states and intentions, playing an important role in human daily communication, and face expression recognition is the only way to achieve true artificial intelligence. At the same time, with the rapid improvement of hardware computing power and the support of excellent deep learning models, convolutional neural networks have achieved many very good results in image classification tasks, and applying deep learning methods in face expression recognition has also become the main choice.

[0003] Face expression recognition is usually regarded as an image classification problem, mainly including three steps: data preprocessing, feature extraction, and prediction classification. The main task of data preprocessing is to screen, augment, and normalize the data in the dataset according to the requirements of the model, and crop the face area in the image through face detection, removing the background area irrelevant to expression recognition in the original image. Feature extraction generally uses a convolutional neural network to extract the spatial features of the image, and prediction classification is to use the extracted expression features as the input of the classifier for classification.

[0004] Most of the images in the face expression recognition dataset in natural scenes are obtained from the Internet such as social websites. Therefore, there is a problem of unbalanced sample distribution in the dataset, and it is very difficult or even impossible to collect the same number of data for each category. In addition, due to various factors such as different poses, illuminations, genders, and identities, there are large within-class differences in the dataset. Therefore, face expression recognition in natural scenes still has a large room for development and faces severe challenges. Currently, the relatively effective solutions are roughly as follows: The first method is two important strategies of transfer learning from a face recognition model and semi-supervised learning using unlabeled data; the second method is to suppress factors such as blurred expressions, low-quality images, and the subjectivity of annotators, which can enable the deep network to learn real expression features; the third method is to combine the designed loss function with the cross-entropy loss to achieve within-class compactness and between-class separation. Summary of the Invention

[0005] To address the need for facial expression recognition under natural conditions, this invention proposes a facial expression recognition method based on cross-layer multi-scale channel mutual attention learning.

[0006] The main structure of this method is based on cross-layer mutual learning, primarily comprising a backbone network, a multi-scale channel attention mechanism, a multi-granularity feature region generation mechanism, a CAM attention region generation mechanism, and a multi-step mutual learning mechanism. A progressive multi-step strategy is employed to train each stage, with corresponding network hyperparameter configurations set. During training, the loss function is calculated for the output of each stage, and the gradient obtained from the derivative is backpropagated for gradient descent parameter updates, allowing the model to learn the corresponding functionalities.

[0007] The facial expression recognition method based on cross-layer multi-scale channel mutual attention learning of the present invention includes the following steps:

[0008] 1) Obtain a database of facial expression images and divide it into training and testing sets;

[0009] 2) Preprocess the facial expression images from step 1): Use the MTCNN method to detect, align, and crop the facial expression images, remove redundant background information, and obtain magnified facial expression images;

[0010] 3) Data augmentation: Perform data augmentation on the facial expression images obtained from the preprocessing in step 2). Augmentation methods include random flipping and random cropping.

[0011] 4) Building the CMSCMAL-Net model: The various settings for the CMSCMAL-Net model include the following steps:

[0012] 4.1 Constructing the Backbone Feature Extraction Network: The Backbone Feature Extraction Network is used to extract features from the input image. Its overall structure is ResNet50. The construction of ResNet50 includes the following steps:

[0013] 4.1.1 ResNet50 is a deep convolutional neural network. Its characteristic is that it introduces a residual structure to form a residual network. During the forward and backward propagation of the model, it has a skip connection information propagation path, which can effectively avoid the gradient vanishing and gradient exploding phenomena common in deep neural networks, and increase the depth of the model to perform efficient feature extraction.

[0014] 4.1.2 The most basic structure of a residual network is the residual block, which is divided into the main path and the residual path. The main path consists of a sequential structure of a convolutional layer with a kernel size of 1×1, a batch normalized layer, a ReLU activation function layer, a convolutional layer with a kernel size of 3×3, a batch normalized layer, a ReLU activation function layer, a convolutional layer with a kernel size of 1×1, a batch normalized layer, and a ReLU activation function layer, arranged in a top-down order. The residual path is a summation structure that skips between the input and output of the main path.

[0015] 4.1.3 The residual network processes the input RGB image by stacking residual blocks, and finally obtains a deep feature with 2048 channels and a downsampling rate of 32. Since the spatial size of the feature map decreases from the shallow stage to the deep stage, the ResNet50 layer excluding the fully connected classifier is divided into 5 stages, which are set as: stage1, stage2, stage3, stage4, and stage5.

[0016] 4.2 Constructing a cross-layer multi-scale channel attention structure: Utilizing the different stage features and multi-scale channel information of ResNet50, a cross-layer multi-scale channel mutual attention structure is designed, including the following steps:

[0017] 4.2.1 Multi-scale Channel Attention Module Design: By changing the spatial pool size, channel attention can be implemented at multiple scales. Pointwise convolution is performed using pointwise channel interactions at each spatial location, and this pointwise convolution is used as a local channel information aggregator, added to the global information within the attention module. To preserve parameters, channel attention for local features is calculated using a bottleneck structure. Where: X∈{x1,x2,…,x} n ,…x N} represents the intermediate feature map, and its expression is:

[0018]

[0019] Wherein: the kernel sizes of PWConv1 and PWConv2 are respectively: and

[0020] The input X is first subjected to a global average pooling operation, followed by channel attention with local features. The same pointwise convolution operation can obtain channel attention for global features. Combined with local channel context Normalization using the sigmoid function yields the refined features of the multi-scale channel attention module M:

[0021]

[0022] in: Indicates the attention weights generated by M; Indicates broadcast addition; This represents element-wise multiplication;

[0023] 4.2.2 Multi-granularity feature region generation: refining features Input to In convolutional layers, The kernel size of the convolutional layer is Where: 1×1 is the space size; C n It is the number of input channels. The number of output channels, after batch normalization and ELU operations, yields the intermediate feature map x. n '; Then, feature x n 'Enter to In convolutional layers, The kernel size of the convolutional layer is Where: 3×3 is the size of the space; This refers to the number of input channels; C n This is the output channel; then batch normalization and ELU operations are performed to obtain the intermediate feature map x. n Finally, perform max pooling.

[0024] 4.2.3 Note the region generation: First, it is based on x n '' is category k n Generate Class Activation Maps (CAMs). CAMs for specific classes are used to identify the discriminative image regions for that class. They are defined as follows:

[0025]

[0026] Where: the coordinates (α, β) represent x n '' and Φ n Spatial location; p n Represents category k n Corresponding classification predictions based on fully connected layers; Φ n (α,β) represents the importance of activation at spatial location (α,β), thus leading to the classification of the image into k. n Category;

[0027] By upsampling the CAM to the size of the input image, we can understand the relationship between the image and category k at a certain stage. n The most relevant region; therefore, after obtaining Φ, Φ is compared with a bilinear sampling kernel. n Upsampling is performed to generate attention maps. Wherein: H in Win These are the height and width of the input image, respectively; then, each spatial element of the normalized attention map is computed using min-max normalization:

[0028]

[0029] Using normalized attention mapping As a guide, identify and cut off discriminatory areas, specifically, first by... The elements in the array are set to 1 for values ​​greater than the threshold t (t∈[0,1]) and 0 for other values ​​to generate the array. Right now The formula for calculating each spatial element is:

[0030]

[0031] according to Can locate a coverage The bounding boxes of all positive regions are calculated, and the region is cropped from the input image. The cropped region is upsampled to the size of the input image, and the upsampled attention region A is applied. n The attention region is considered as the predicted attention region at a certain stage. The overall attention region is generated by summarizing the attention information learned at different stages. The calculation formula is as follows:

[0032]

[0033] Subsequently, similar to A n The generation process uses min-max normalization. And the result is expressed as Then, based on the same threshold t, The elements are set to 1 or 0; finally, the regions covering all positive values ​​are located, and the same regions of the input image are cropped to obtain an unsampled overall attention region A with the same size as the input image. global ;

[0034] 4.3 Multi-step Mutual Learning: A progressive multi-step strategy is adopted to train each stage. In the early steps, these stages are trained one by one, which allows for focused learning of attentional information for the corresponding stage. In the final two steps, all stages work together to learn effective information from the region of interest and the original image respectively, including the following steps:

[0035] 4.3.1 Training the deepest feature stage: Since training the deepest feature stage involves stages shallower than that feature level, this step can also generate attention regions proposed by all stages and the overall attention region {A1, A2, ..., A...}. n A globalThe classification judgment is based on marking each stage, and these areas carry attentional information for each stage.

[0036] 4.3.2 Mutual data augmentation, training gradually shifts to the shallow feature stage: When training this stage, according to the principle of mutual data augmentation, an input is randomly selected from the image library, which consists of the original input and attention regions other than this shallow stage;

[0037] 4.3.3 Training all hierarchical stages and their connections with the overall attention region: The overall attention region is composed of all hierarchical stages, containing important attention information from each stage, amplifying and studying the attention information obtained together to extract more fine-grained features;

[0038] 4.4 Model Training: The model was trained using stochastic gradient descent (SGD) with 200 epochs, a momentum of 0.9, a weight_decay of 0.0005, a batch size of 64, and a learning rate of 0.002 using cosine annealing. The image input size was set to 224×224, and the threshold t was set to 0.5.

[0039] 5) Loss function: The progressive multi-step strategy of cross-entropy loss is used for training, which includes output prediction for each stage and output prediction for cascaded features;

[0040] 5.1 For the output of each stage, the loss is calculated using the cross-entropy between the true label y and the predicted probability distribution. The calculation formula is as follows:

[0041]

[0042] 5.2 For the output of cascaded features, the cross-entropy between the true label y and the predicted probability distribution is used to calculate the loss. The calculation formula is as follows:

[0043]

[0044] The beneficial effects of this invention are as follows:

[0045] This invention employs a facial expression recognition method based on cross-layer, multi-scale channel mutual attention learning. The model structure includes a backbone network, a multi-scale channel attention module, a multi-granularity feature region generation module, a CAM attention region generation module, and a multi-step mutual learning module. The facial expression image to be recognized is input into the model. First, features are extracted through the ResNet50 backbone network. Then, the features at each level are input into the multi-scale channel attention module to extract subtle features, resulting in feature maps containing rich information at both deep and shallow layers. These feature maps are input to the multi-granularity feature region generation module and the CAM attention region generation module in stages, focusing on local areas helpful for discrimination, then gradually expanding the surrounding area, and finally to the entire image. During training, each iteration involves multiple steps. In each step, one stage is trained or all stages are fused together. For a given stage, attention regions proposed by other stages are considered as possible data augmentations. Specifically, starting from the deepest stage, the parameters of other stages are updated to obtain the attention regions predicted by all stages, then gradually moving to shallower stages, and finally training the fusion of all stages. By focusing on the features of multi-scale channels and the mutual learning of deep and shallow attention features, and using a progressive multi-step training method with cross-entropy loss, the facial expression recognition task is completed.

[0046] This invention utilizes the CMSCMAL-Net model, combining local and global features to obtain subtle and detailed basic features. It also avoids inter-class similarity and intra-class variation by leveraging effective information from attention regions. Furthermore, treating attention regions as data augmentation not only addresses the problem of limited training data but also avoids introducing undesirable noise (such as occlusion). The progressive multi-step training strategy not only yields complementary information from the prediction scores at each stage and the overall prediction score but also from the original input and the overall attention region, resulting in more accurate final prediction and classification results. This invention provides an effective solution for facial expression recognition under natural conditions and is applicable to other types of image classification tasks. Attached Figure Description

[0047] Figure 1 This is a flowchart of a facial expression recognition method based on cross-layer multi-scale channel mutual attention learning;

[0048] Figure 2 A flowchart of a multi-scale channel attention module;

[0049] Figure 3 Example image of a database of facial expression images required for the experiment;

[0050] Figure 4 The training loss function curve for the CMSCMAL-Net model;

[0051] Figure 5 The accuracy curve for the CMSCMAL-Net model is shown. Detailed Implementation

[0052] The present invention will now be described in conjunction with the accompanying drawings.

[0053] like Figure 1 As shown, the facial expression recognition method based on cross-layer multi-scale channel mutual attention learning of the present invention includes the following steps:

[0054] 1) Obtain a database of facial expression images, such as... Figure 3 As shown, the training set and the test set are divided.

[0055] 2) Preprocess the facial expression images from step 1): Use the MTCNN method to detect, align, and crop the facial expression images, remove redundant background information, and obtain magnified facial expression images;

[0056] 3) Data augmentation: Perform data augmentation on the facial expression images obtained from the preprocessing in step 2). Augmentation methods include random flipping and random cropping.

[0057] 4) Building the CMSCMAL-Net model: The various settings for the CMSCMAL-Net model include the following steps:

[0058] 4.1 Constructing the Backbone Feature Extraction Network: The Backbone Feature Extraction Network is used to extract features from the input image. Its overall structure is ResNet50. The construction of ResNet50 includes the following steps:

[0059] 4.1.1 ResNet50 is a deep convolutional neural network. Its characteristic is that it introduces a residual structure to form a residual network. During the forward and backward propagation of the model, it has a skip connection information propagation path, which can effectively avoid the gradient vanishing and gradient exploding phenomena common in deep neural networks, and increase the depth of the model to perform efficient feature extraction.

[0060] 4.1.2 The most basic structure of a residual network is the residual block, which is divided into the main path and the residual path. The main path consists of a sequential structure of a convolutional layer with a kernel size of 1×1, a batch normalized layer, a ReLU activation function layer, a convolutional layer with a kernel size of 3×3, a batch normalized layer, a ReLU activation function layer, a convolutional layer with a kernel size of 1×1, a batch normalized layer, and a ReLU activation function layer, arranged in a top-down order. The residual path is a summation structure that skips between the input and output of the main path.

[0061] 4.1.3 The residual network processes the input RGB image by stacking residual blocks, and finally obtains a deep feature with 2048 channels and a downsampling rate of 32. Since the spatial size of the feature map decreases from the shallow stage to the deep stage, the ResNet50 layer excluding the fully connected classifier is divided into 5 stages, which are set as: stage1, stage2, stage3, stage4, and stage5.

[0062] 4.2 Constructing a cross-layer multi-scale channel attention structure: Utilizing the different stage features and multi-scale channel information of ResNet50, a cross-layer multi-scale channel mutual attention structure is designed, including the following steps:

[0063] 4.2.1 Multi-scale Channel Attention Module Design: By changing the spatial pool size, channel attention can be implemented at multiple scales. Pointwise convolution is performed using pointwise channel interactions at each spatial location, and this pointwise convolution is used as a local channel information aggregator, added to the global information within the attention module. To preserve parameters, channel attention for local features is calculated using a bottleneck structure. Where: X∈{x1,x2,…,x} n ,…x N} represents the intermediate feature map, and its expression is:

[0064]

[0065] Wherein: the kernel sizes of PWConv1 and PWConv2 are respectively: and

[0066] The input X is first subjected to a global average pooling operation, followed by channel attention with local features. The same pointwise convolution operation can obtain channel attention for global features. Combined with local channel context Normalization using the sigmoid function yields the refined features of the multi-scale channel attention module M:

[0067]

[0068] in: Indicates the attention weights generated by M; Indicates broadcast addition; This represents element-wise multiplication;

[0069] 4.2.2 Multi-granularity feature region generation: refining features Input to In convolutional layers, The kernel size of the convolutional layer is Where: 1×1 is the space size; C n It is the number of input channels. The number of output channels, after batch normalization and ELU operations, yields the intermediate feature map x. n '; Then, feature x n 'Enter to In convolutional layers, The kernel size of the convolutional layer is Where: 3×3 is the size of the space; This refers to the number of input channels; C n This is the output channel; then batch normalization and ELU operations are performed to obtain the intermediate feature map x. n Finally, perform max pooling.

[0070] 4.2.3 Note the region generation: First, it is based on x n '' is category k n Generate Class Activation Maps (CAMs). CAMs for specific classes are used to identify the discriminative image regions for that class. They are defined as follows:

[0071]

[0072] Where: the coordinates (α, β) represent x n '' and Φ n Spatial location; p n Represents category k n Corresponding classification predictions based on fully connected layers; Φ n (α,β) represents the importance of activation at spatial location (α,β), thus leading to the classification of the image into k. n Category;

[0073] By upsampling the CAM to the size of the input image, we can understand the relationship between the image and category k at a certain stage. n The most relevant region; therefore, after obtaining Φ, Φ is compared with a bilinear sampling kernel. n Upsampling is performed to generate attention maps. Wherein: H in W in These are the height and width of the input image, respectively; then, each spatial element of the normalized attention map is computed using min-max normalization:

[0074]

[0075] Using normalized attention mapping As a guide, identify and cut off discriminatory areas, specifically, first by... The elements in the array are set to 1 for values ​​greater than the threshold t (t∈[0,1]) and 0 for other values ​​to generate the array. Right now The formula for calculating each spatial element is:

[0076]

[0077] according to Can locate a coverage The bounding boxes of all positive regions are calculated, and the region is cropped from the input image. The cropped region is upsampled to the size of the input image, and the upsampled attention region A is applied. n The attention region is considered as the predicted attention region at a certain stage. The overall attention region is generated by summarizing the attention information learned at different stages. The calculation formula is as follows:

[0078]

[0079] Subsequently, similar to A n The generation process uses min-max normalization. And the result is expressed as Then, based on the same threshold t, The elements are set to 1 or 0; finally, the regions covering all positive values ​​are located, and the same regions of the input image are cropped to obtain an unsampled overall attention region A with the same size as the input image. global ;

[0080] 4.3 Multi-step Mutual Learning: A progressive multi-step strategy is adopted to train each stage. In the early steps, these stages are trained one by one, which allows for focused learning of attentional information for the corresponding stage. In the final two steps, all stages work together to learn effective information from the region of interest and the original image respectively, including the following steps:

[0081] 4.3.1 Training the deepest feature stage: Since training the deepest feature stage involves stages shallower than that feature level, this step can also generate attention regions proposed by all stages and the overall attention region {A1, A2, ..., A...}. n A global The classification judgment is based on marking each stage, and these areas carry attentional information for each stage.

[0082] 4.3.2 Mutual data augmentation, training gradually shifts to the shallow feature stage: When training this stage, according to the principle of mutual data augmentation, an input is randomly selected from the image library, which consists of the original input and attention regions other than this shallow stage;

[0083] 4.3.3 Training all hierarchical stages and their connections with the overall attention region: The overall attention region is composed of all hierarchical stages, containing important attention information from each stage, amplifying and studying the attention information obtained together to extract more fine-grained features;

[0084] 4.4 Model Training: The model was trained using stochastic gradient descent (SGD) with 200 epochs, a momentum of 0.9, a weight_decay of 0.0005, a batch size of 64, and a learning rate of 0.002 using cosine annealing. The image input size was set to 224×224, and the threshold t was set to 0.5.

[0085] 5) Loss function: The progressive multi-step strategy of cross-entropy loss is used for training, which includes output prediction for each stage and output prediction for cascaded features;

[0086] 5.1 For the output of each stage, the loss is calculated using the cross-entropy between the true label y and the predicted probability distribution. The calculation formula is as follows:

[0087]

[0088] 5.2 For the output of cascaded features, the cross-entropy between the true label y and the predicted probability distribution is used to calculate the loss. The calculation formula is as follows:

[0089]

[0090] Based on the above formulas, the loss function for each stage, the cascaded feature stage, and the final loss function can be calculated, as follows: Figure 4 As shown; after 200 epochs, the average accuracy of the CMSCMAL-Net model for facial expression recognition on the RAF-DB dataset is approximately 90%, specifically as follows: Figure 5 As shown.

Claims

1. A facial expression recognition method based on cross-layer multi-scale channel mutual attention learning, characterized in that... Includes the following steps: 1) Obtain a database of facial expression images and divide it into training and testing sets; 2) Preprocess the facial expression images from step 1): Use the MTCNN method to detect, align, and crop the facial expression images, remove redundant background information, and obtain magnified facial expression images; 3) Data augmentation: Perform data augmentation on the facial expression images obtained from the preprocessing in step 2). Augmentation methods include random flipping and random cropping. 4) Building the CMSCMAL-Net model: The various settings for the CMSCMAL-Net model include the following steps: 4.1 Constructing the Backbone Feature Extraction Network: The Backbone Feature Extraction Network is used to extract features from the input image. Its overall structure is ResNet50. The construction of ResNet50 includes the following steps: 4.1.1 ResNet50 is a deep convolutional neural network. Its characteristic is that it introduces a residual structure to form a residual network. During the forward and backward propagation of the model, it has a skip connection information propagation path, which can effectively avoid the gradient vanishing and gradient exploding phenomena common in deep neural networks, and increase the depth of the model to perform efficient feature extraction. 4.1.2 The most basic structure of a residual network is the residual block, which is divided into the main path and the residual path. The main path consists of a sequential structure of a convolutional layer with a kernel size of 1×1, a batch normalized layer, a ReLU activation function layer, a convolutional layer with a kernel size of 3×3, a batch normalized layer, a ReLU activation function layer, a convolutional layer with a kernel size of 1×1, a batch normalized layer, and a ReLU activation function layer, arranged in a top-down order. The residual path is a summation structure that skips between the input and output of the main path. 4.1.3 The residual network processes the input RGB image by stacking residual blocks, and finally obtains a deep feature with 2048 channels and a downsampling rate of 32. Since the spatial size of the feature map decreases from the shallow stage to the deep stage, the ResNet50 layer excluding the fully connected classifier is divided into 5 stages, which are set as: stage1, stage2, stage3, stage4, and stage5. 4.2 Constructing a cross-layer multi-scale channel attention structure: Utilizing the different stage features and multi-scale channel information of ResNet50, a cross-layer multi-scale channel mutual attention structure is designed, including the following steps: 4.2.1 Multi-scale Channel Attention Module Design: By changing the spatial pool size, channel attention can be implemented at multiple scales. Pointwise convolution is performed using pointwise channel interactions at each spatial location, and this pointwise convolution is used as a local channel information aggregator, added to the global information within the attention module. To preserve parameters, channel attention for local features is calculated using a bottleneck structure. in: X∈{x1,x2,…,x n ,…x N } represents the intermediate feature map, and its expression is: Wherein: the kernel sizes of PWConv1 and PWConv2 are respectively: and The input X is first subjected to a global average pooling operation, followed by channel attention with local features. The same pointwise convolution operation can obtain channel attention for global features. Combined with local channel context Normalization using the sigmoid function yields the refined features of the multi-scale channel attention module M: in: Indicates the attention weights generated by M; Indicates broadcast addition; This represents element-wise multiplication; 4.2.2 Multi-granularity feature region generation: Input the refined feature χ into... In convolutional layers, The kernel size of the convolutional layer is Where: 1×1 is the space size; C n It is the number of input channels. The number of output channels, after batch normalization and ELU operations, yields the intermediate feature map x. n '; Then, feature x n 'Enter to In convolutional layers, The kernel size of the convolutional layer is Where: 3×3 is the size of the space; It is the number of input channels; C n This is the output channel; then batch normalization and ELU operations are performed to obtain the intermediate feature map x. n Finally, perform max pooling. 4.2.3 Note the region generation: First, it is based on x n '' represents category k n Generate Class Activation Maps (CAMs). CAMs for specific classes are used to identify the discriminative image regions for that class. They are defined as follows: Where: the coordinates (α, β) represent x n '' and Φ n Spatial location; p n Represents category k n Corresponding classification predictions based on fully connected layers; Φ n (α,β) represents the importance of activation at spatial location (α,β), thus leading to the classification of the image into k. n Category; By upsampling the CAM to the size of the input image, we can understand the relationship between the image and category k at a certain stage. n The most relevant region; therefore, after obtaining Φ, Φ is compared using a bilinear sampling kernel. n Upsampling is performed to generate attention maps. Wherein: H in W in These are the height and width of the input image, respectively; then, each spatial element of the normalized attention map is computed using min-max normalization: Using normalized attention mapping As a guide, identify and cut off discriminatory areas, specifically, first by... The elements in the array are set to 1 for values ​​greater than the threshold t (t∈[0,1]) and 0 for other values ​​to generate the array. Right now The formula for calculating each spatial element is: according to Can locate a coverage The bounding boxes of all positive regions are calculated, and the region is cropped from the input image. The cropped region is upsampled to the size of the input image, and the upsampled attention region A is applied. n The attention region is considered as the predicted attention region at a certain stage. The overall attention region is generated by summarizing the attention information learned at different stages. The calculation formula is as follows: Subsequently, similar to A n The generation process uses min-max normalization. And the result is expressed as Then, based on the same threshold t, The elements are set to 1 or 0; finally, the regions covering all positive values ​​are located, and the same regions of the input image are cropped to obtain an unsampled overall attention region A with the same size as the input image. global ; 4.3 Multi-step Mutual Learning: A progressive multi-step strategy is adopted to train each stage. In the early steps, these stages are trained one by one, which allows for focused learning of attentional information for the corresponding stage. In the final two steps, all stages work together to learn effective information from the region of interest and the original image respectively, including the following steps: 4.3.1 Training the deepest feature stage: Since training the deepest feature stage involves stages shallower than that feature level, this step can also generate attention regions proposed by all stages and the overall attention region {A1, A2, ..., A...}. n A global The classification judgment is based on marking each stage, and these areas carry attentional information for each stage. 4.3.2 Mutual data augmentation, training gradually shifts to the shallow feature stage: When training this stage, according to the principle of mutual data augmentation, an input is randomly selected from the image library, which consists of the original input and attention regions other than this shallow stage; 4.3.3 Training all hierarchical stages and their connections with the overall attention region: The overall attention region is composed of all hierarchical stages, containing important attention information from each stage, amplifying and studying the attention information obtained together to extract more fine-grained features; 4.4 Model Training: The model was trained using stochastic gradient descent (SGD) with 200 epochs, a momentum of 0.9, a weight_decay of 0.0005, a batch size of 64, and a learning rate of 0.002 using cosine annealing. The image input size was set to 224×224, and the threshold t was set to 0.

5. 5) Loss function: The progressive multi-step strategy of cross-entropy loss is used for training, which includes output prediction for each stage and output prediction for cascaded features; 5.1 For the output of each stage, the loss is calculated using the cross-entropy between the true label y and the predicted probability distribution. The calculation formula is as follows: 5.2 For the output of cascaded features, the cross-entropy between the true label y and the predicted probability distribution is used to calculate the loss. The calculation formula is as follows:

Images