Facial expression recognition method based on attention mechanism and self-distillation

Abstract

A facial expression recognition model is constructed, the model comprising a feature extraction module, an adaptive channel attention module, a remodelling module, and a self-distillation network. The

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of facial expression recognition, and in particular to a facial expression recognition method based on an attention mechanism and self-distillation. BACKGROUND

[0002] Facial expressions play a key role in interpersonal interactions. A person can clearly sense, through a change in a facial expression, emotion expressed by another person. In the past several decades, facial expression recognition has been widely used in fields such as digital entertainment, human-computer interaction, driving detection, and health care. As a result, more researchers start to engage in research in facial expression recognition. Because of poor lighting conditions, occlusion of the facial expression, and a change in a posture, it is still challenging to recognize the facial expression under field conditions.

[0003] In a conventional method, the facial expression is usually recognized based on a manual feature or a shallow feature, a global facial expression image is taken as an input of a model, and a category which the facial expression belongs to is determined based on a final output of the model. However, because categories of facial expressions are highly similar to each other, the model is difficult to pay attention to key regions that are conducive to classification of the facial expression. The key regions play an important role in correct classification of the facial expression. In some advanced methods, a capability of the model to capture an important region by increasing complexity of the model and an attention mechanism, with disadvantages of significantly increasing the complexity of the model and seriously hindering deployment of the model in low-end devices. SUMMARY

[0004] In view of the above problems, an objective of the present disclosure is to provide a facial expression recognition method based on an attention mechanism and self-distillation. In the method, under a condition that complexity of a model is not increased, the model is adaptively guided to pay attention to some important regions in a facial expression, to implement a correct classification of seven basic emotions, namely, hate, happiness, anger, fear, surprise, sadness, and neutral. This effectively improves efficiency and accuracy of facial expression recognition.

[0005] The technical solution adopted by the present disclosure to resolve the technical problem is as follows:

[0006] step 1: dividing three public datasets, namely, Affect-Net, FERPlus, and RAF-DB, into a training set, a test set, and a validation set, where the training set, the test set, and the validation set each include images of seven basic expressions;

[0007] step 2: constructing a facial expression recognition model, where the facial expression recognition model includes a feature extraction module, an adaptive channel attention module, a remodeling module, and a self-distillation network;

[0008] step 3: performing face alignment on the images of seven basic expressions in the training set, the test set, and the validation set to be input into the facial expression recognition model, and cropping the images to a fixed size; and performing image enhancement, for example, random crop, horizontal flip, and random removal, on the images in the training set to be input into the facial expression recognition model, to prevent the facial expression recognition model from overfitting;

[0009] step 4: inputting the images in the training set into the facial expression recognition model, and training and optimizing a learnable parameter in the facial expression recognition model until accuracy of the facial expression recognition model is unable to be improved, where a specific training process includes the following sub-steps:

[0010] S41: inputting the image into the feature extraction module, and taking a feature map Zj output by each basic block in the feature extraction module as a feature output by the feature extraction module at each stage;

[0011] S42: adding the adaptive channel attention module after the feature map Zj output by each basic block, where in the adaptive channel attention module, firstly, channel information of the feature map Zj output by each basic block is aggregated by performing maximum pooling and average pooling, to generate two different channel vectors qmaX'qavgl secondly, weights wmax,wavg are generated respectively for qmax>qavg by an adaptive weight module; thirdly, a feature obtained after qmax and wmax are multiplied and a feature obtained after qavg and wavg are multiplied are fused through addition, to obtain a final channel weight feature fcw; and finally, fcw is multiplied by an input feature map Zj, to obtain a final output feature map Zj, where calculation formulas are as follows:

[0012] qmax = MaxPool(Zj),

[0013] qavg = AvgPool(Zj),

[0014] wmax = Wml(a(Wm0(qmax))),

[0015] wavg = Wal(a(Wa0(qavg))),

[0016] fcw — a(MLP(qmax) * wmax + MLP(qavg) * wavg),

[0017] Zj = fcw * Zj.

[0018] S43: distilling, by the remodeling module, a feature output by a shallow basic block, and mapping a feature output by each basic block to a same dimension, to obtain a remodeled feature map Fj, where a calculation formula is as follows:

[0019] Fj = Gj (BNj(ConVj (Zj

[0020] S44: calculating a predicted score fj corresponding to the remodeled feature map Fj of each basic block through a fully connected layer and a Softmax function, where a calculation formula is as follows:

[0021] fj = Softmax(W(Fj));

[0022] S45: gradually performing, by the self-distillation network, knowledge distillation on the facial expression recognition model from deep to shallow based on a corresponding distillation loss, specifically including the following sub-steps:

[0023] S451: measuring, based on a cross-entropy loss, a difference between a predicted score output by the last basic block and a real label of the training set, where a calculation formula is as follows:

[0024] Lce = CrossEntropyLoss(fc, y);

[0025] S452: measuring, based on a Kullback-Leibler (KL) divergence loss, a difference between a predicted score obtained by a current basic block and a predicted score obtained by a next basic block of the current basic block, where a calculation formula is as follows:

[0026] Lkl = KL(fc_j, fc_j+1);

[0027] S453: measuring, based on an L2 loss, a difference between a feature output by the current basic block and a feature output by the next basic block of the current basic block, where a calculation formula is as follows:

[0028] L2 = I|Fc_j - Fc_j+11 |i; and

[0029] S454: obtaining a total loss based on the cross-entropy loss, the KL divergence loss, and the L2 loss, where a calculation formula is as follows:

[0030] Ltotal = Lce + aLkl + p L2; and

[0031] step 5: performing inference on the images of seven basic expressions in the test set and the validation set, specifically including: in an inference stage, removing the adaptive channel attention module, the remodeling module, and the self-distillation network, performing, by the feature extraction module after the training in a training stage, final prediction on an input, and inputting the images of seven basic expressions in the training set and the test set into the facial expression recognition model, to obtain a corresponding classification result, where the result is one of the seven basic expressions, namely, hate, happiness, anger, fear, surprise, sadness, and neutral.

[0032] In the method, a ResNet-50 is used as the feature extraction module of the facial expression recognition model. The ResNet-50 includes four basic blocks. Each basic block includes a plurality of convolutional layers. When the image is input to the feature extraction module, a spatial scale of the feature map is halved and a quantity of channels is doubled after the image passes through each basic block.

[0033] The adaptive channel attention module is added after the feature map output by each basic block. In the adaptive channel attention module, firstly, channel information of the input feature map is aggregated by performing maximum pooling and average pooling, to generate two different channel vectors qmax’Qavgi secondly, the two features are input into the MLP and a corresponding adaptive weight module, and the adaptive weight module generates the corresponding weights wmax,wavg; thirdly, a feature obtained after qmax and wmax are multiplied and a feature obtained after qavg and wavg are multiplied are fused through addition, to obtain a channel weight feature fcw, finally, fcw is multiplied by an input feature map Zj, to obtain an output feature map Zj . This implements weighting on the input feature map in a channel dimension.

[0034] In the ResNet-50, semantic information and dimensions of features output by different basic blocks may be different. To overcome a defect that semantic information of the shallow basic block is weak and to facilitate the following knowledge distillation, the remodeling module is used to distill the feature output by the shallow basic block and map the feature output by each basic block to the same dimension.

[0035] In the method, the parameter in the facial expression recognition model is optimized based on some distillation losses in a training process. The model is mainly optimized based on the cross-entropy loss, the KL loss, and the L2 loss.

[0036] In the inference stage, the model removes the adaptive channel attention module, the remodeling module, and the self-distillation network, keeps only the feature extraction module, and performs prediction on an input image based on a final output of the feature extraction module.

[0037] The present disclosure has the following advantages: Due to feature extraction performed by the feature extraction module in a plurality of stages, the feature extracted by the model not only includes low-level spatial information in a shallow network, but also includes high-level semantic information in a deep network. This increases robustness of the feature extracted by the model. Due to the adaptive channel attention module, a weight of a maximum pooling feature and an average pooling feature is adaptively adjusted. This pays more attention to a channel, and improves a representation capability of the output feature. Due to the remodeling module, a difference between features output by the feature extraction module at different stages is reduced. The shallow network (a student network) is guided to continuously learn from the deep network (a teacher network) by a new form of self-distillation, so that knowledge from the network is distilled and compressed, performance of the network is improved, and self-learning of the network is implemented. The adaptive channel attention module, the remodeling module, and the self-distillation network are removed in the inference stage, and the feature extraction module trained in the training stage is kept, so that the complexity of the model in the inference stage is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG. 1 is a flowchart of a facial expression recognition method based on an attention mechanism and self-distillation according to the present disclosure;

[0039] FIG. 2 is a schematic diagram of an overall network architecture according to an example of the present disclosure; and

[0040] FIG. 3 is a schematic diagram of an overall network architecture of an adaptive channel attention module according to an example of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] The present disclosure is described in more detail below with reference to the accompanying drawings and specific implementations.

[0042] It has been found that among many facial expression recognition methods, a convolutional neural network using an attention mechanism is a reliable idea with high efficiency. The convolutional neural network using an attention mechanism is a weakly-supervised method, and uses a multi-stage convolutional network as a feature extraction module while using the attention mechanism. Therefore, a model can obtain more comprehensive information. Due to multi-stage feature extraction, the model can extract low-level physical information (for example, a shape, a contour, and an edge) and high-level semantic information. This promotes the model to correctly recognize a facial expression. Although the facial expression recognition method based on an attention mechanism improves accuracy of facial expression recognition to a specific extent, shortcomings are still present. For example, to improve effectiveness and accuracy of the adaptive channel attention module, a depth and a quantity of attention modules need to be increased. This seriously increases a computational volume of the model and hinders deployment of the model on a low-end device. Therefore, an effective model compression method and a knowledge transfer system are required, to enable the model to effectively ensure the accuracy and efficiency of the model without increasing the complexity.

[0043] To resolve the foregoing problems, the present disclosure provides a facial expression recognition method based on an attention mechanism and self-distillation. In the method, knowledge that is distilled from a deep network is gradually distilled to a shallow network by adaptively adjusting a ratio of maximum pooling to average pooling in channel attention and by a self-distillation idea. Therefore, the accuracy and efficiency of the model is improved. In addition, because an adaptive channel attention module, a remodeling module, and a self-distillation network are only used in a training stage, complexity of the model is not increased.

[0044] Specifically, the present disclosure provides a facial expression recognition method based on an attention mechanism and self-distillation. As shown in FIG. 1, the method mainly includes the following steps.

[0045] Step 1: Divide three public datasets, namely, Affect-Net, FERPlus, and RAF-DB, into a training set, a test set, and a validation set, where the training set, the test set, and the validation set each include images of seven basic expressions.

[0046] The RAF-DB is a dataset of a real world, and includes facial expression images of thousands of people of different races, different genders, and different ages. The RAF-DB includes a total of 15,339 images (of which 12,271 images are taken as the training set, and 3,068 images are taken as the test set). The FERPlus dataset is extended from FER2013 dataset. The FER2013 dataset includes real data collected by the Google search engine. In the FER2013 dataset, all the images have been cropped to 48 x 48 after face alignment is performed. However, because there are many incorrectly labeled images in the FER2013 dataset, the FERPlus dataset expands and relabels the FER2013 dataset. The FERPlus dataset includes a total of 35,887 images (of which 28,709 images are taken as the training set, 3,589 images are taken as the test set, and 3,589 images are taken as the validation set). The Affect-Net dataset is the largest of all FER public datasets so far, and includes a total of 287,401 images (of which 283,901 images are taken as the training set, and 3,500 images are taken as the test set).

[0047] Step 2: Construct a facial expression recognition model. As shown in FIG. 2, the facial expression recognition model includes a feature extraction module, an adaptive channel attention module, a remodeling module, and a self-distillation network.

[0048] In the facial expression recognition model in the present disclosure, the feature extraction module is used to extract features output by the facial expression recognition model at different stages. Features extracted by a shallow basic block are mainly some low-dimensional detailed features, such as colors, textures, and contours. Features extracted by a deep basic block have high-level semantic information, and are closer to understanding of information of content of images by a person. The feature extraction module mainly includes a ResNet-50. The ResNet-50 includes four basic blocks. Each basic block includes multiple convolutional layers. When an image is input into the feature extraction module, a spatial scale of a feature map is halved and a quantity of channels is doubled while the image passes through each basic block. A feature map Zj output by each basic block in the feature extraction module is used as a feature output by the feature extraction module at each stage.

[0049] In a conventional technology, the adaptive channel attention module is widely used in a computer vision task, to guide the facial expression recognition model to pay attention to an important region that is favorable to a final prediction of the adaptive channel attention model. A conventional channel attention module directly adds and fuses an average pooling feature to and with a maximum pooling feature only, ignoring importance of the average pooling feature and the maximum pooling feature to a final output feature. As a result, robustness of the final output feature is poor.

[0050] To resolve the foregoing problems, the method provides an adaptive channel attention module. As shown in FIG. 3, specifically, the adaptive channel attention module is added after the feature map output by each basic block. In the adaptive channel attention module, firstly, channel information of the feature map Zj output by each basic block is aggregated by performing maximum pooling and average pooling, to generate two different channel vectors qmax>qavg e rcxixi, anj the twQ chapel vectors respectively represent the maximum pooling feature and the average pooling feature of the feature map. Secondly, the two features are input into an MLP and a corresponding adaptive weight module. The MLP has only one hidden layer, a size of the hidden layer is Rc / rxlxl, and r represents a reduction ratio. The adaptive weight module includes a fully connection layer. The adaptive channel attention module is used to adaptively adjust contribution of the maximum pooling feature and the average pooling feature to a final output channel weight feature, and generate a corresponding weight wmax,wavg for qmax, qavg- Thirdly, a feature obtained after qmax and wmax are multiplied and a feature obtained after qavg and wavg are multiplied are fused through addition, to obtain a final channel weight feature fcw. Finally, fcw is multiplied by the input feature map Zj, to obtain a final output feature map Zy. This implements weighting on the input feature map Zj in a channel dimension. Specific formulas are as follows:

[0051] qmax = MaxPool(Zj),

[0052] qavg = AvgPool(zj),

[0053] wmax = Wml(a(Wm0(qmax))),

[0054] wavg = Wa1(o(Wa0(qavg))).

[0055] fcw = o(MLP(qmax) * wmax + MLP(qavg) * wavg),

[0056] Zj = fcw * Zj, where

[0057] o' represents a ReLU activation function, MaxPool and AvgPool respectively represent a maximum pooling operation and an average pooling operation, MLP(*) represents an output after a specified vector enters the MLP, and W(*) represents a parameter of a fully connected layer.

[0058] The semantic information and sizes of features output by different basic blocks in the ResNet-50 are different. To overcome a defect that the semantic information of the shallow basic block is weak and to facilitate the following self-distillation, a remodeling module is provided. The remodeling module distills a feature of the shallow basic block, maps the feature output by each 7 basic block to a same dimension, to Fj, and then calculates a predicted score fj corresponding to the output Fj of each basic block through the fully connected layer and a Softmax function. The remodeling module is mainly formed by stacking a quantity of convolutions, a normalization layer, and an activation layer. Specifically:

[0059] Fj =Oj{BNj{Convj{Zj-,^,

[0060] fj = Softmax(W(Fy)), where

[0061] ConVj(- ;e) represents a convolution operation, s represents a parameter of a convolution kernel, BN and o' respectively represent the normalization layer and the activation layer, c represents the quantity of basic blocks, Fj represents the output obtained after Zj passes through a convolutional layer, a normalization layer, and a activation layer, fj represents the predicted score obtained after Fj passes through the fully connected layer and a Softmax layer, and W represents a parameter of the fully connected layer.

[0062] Step 3: Perform face alignment on the images of seven basic expressions in the training set, the test set, and the validation set to be input into the facial expression recognition model, and cropping the images to 224 x 224. Selective enhancement (for example, random crop, horizontal flip, random removal) is performed on the images in the training set to be input into the facial expression recognition model, to prevent the facial expression recognition model from overfitting.

[0063] Step 4: Input the images in the training set into the facial expression recognition model, and training and optimizing a learnable parameter in the facial expression recognition model until the accuracy of the facial expression recognition model is unable to be improved.

[0064] In the method, throughout the training process, the facial expression recognition model is optimized in a new distillation form. Knowledge distillation is one of commonly used model compression methods. For conventional knowledge distillation, a lot of time is taken to search and train a suitable teacher network, and knowledge from the trained teacher network is distilled to the student network through knowledge distillation, to implement compression of the facial expression recognition model. This manner has extremely high costs. This problem is resolved through self-distillation. Therefore, the teacher network does not need to be found, and knowledge learned from the deep network is distilled to the shallow network, to implement self-learning of the facial expression recognition model. However, knowledge is learned progressively in diversified manners. Conventional self-distillation only takes an output of a deepest basic block as an output of the teacher network, and takes an output of another basic block as an output of the student network. After training is completed each time, the student network extracts knowledge from the teacher network. This is not conducive to the student network learns knowledge in the diversified manners.

[0065] To resolve the foregoing problems, the method provides a new form of self-distillation. Except for a last basic block, each basic block takes a next basic block as a teacher, and knowledge is gradually distilled from deep to shallow, to enable a current basic block to learn knowledge from all basic blocks after the current basic block. This ensures that the shallow student network learns the knowledge in the diversified manners. Through the form of self-distillation, semantically richer knowledge in the deep network can be effectively transferred to the shallow network gradually. Therefore, a representation capability of the feature output by the shallow network is improved, and the model is promoted to correctly classify facial expressions. A total of self-distillation loss Ltotal is defined below:

[0066] Ltotal = Lce + a Lkl + p L2, where

[0067] Lce represents the cross-entropy loss, Lkl represents the KL divergence loss, and L2 represents the L2 loss, a, p is a hyper-parameter, and the hyper-parameter is used to adjust contribution of the KL loss and the L2 loss to the total loss.

[0068] Because there is no evaluation index for the prediction of the last basic block, in this method, the cross-entropy loss is used to calculate a difference between the predicted score obtained after the last basic block passes through the Softmax layer and the true labels of the training set. In this way, the predicted score of the last basic block may have an evaluation index, to promote the learning of the facial expression recognition model. Specifically:

[0069] Lce = CrossEntropyLoss(fc, y), where

[0070] CrossEntropyLoss represents the cross-entropy loss, fc represents a predicted score obtained by a Cth (that is, deepest) basic block, and y represents a corresponding label.

[0071] The KL divergence is mainly used to measure a difference between two probabilities. In the method, the KL divergence is used to measure a difference between a predicted score obtained by the current basic block (student) and a predicted score obtained by a next basic block (teacher) of the current basic block, to enable the prediction of the student for a sample is close to prediction of the teacher as much as possible. Specifically:

[0072] Lkl = S^KLCfc-j, fc_j+1), where

[0073] KL represents the KL divergence loss, and f represents update of cancellation of gradient when gradient back propagation is performed.

[0074] The L2 loss is used to monitor a feature output by the student network. The feature output by the student network is close to the feature output by the teacher network as much as possible based on the L2 loss, to improve a representation capability of the feature output by the student network. Specifically:

[0075] L2 = S^llFc-j-Fc-j+illi where

[0076] F represents update of cancellation of gradient when gradient back propagation is performed.

[0077] Step 5: Perform inference on the images of seven basic expressions in the test set and the validation set. In an inference stage, remove the adaptive channel attention module, the remodeling module, and the self-distillation network, perform, by the feature extraction module after the training in a training stage, final prediction on an input, and input the images of seven basic expressions in the training set and the test set into the facial expression recognition model, to obtain a corresponding classification result, where the result is one of the seven basic expressions: hate, happiness, anger, fear, surprise, sadness, and neutral.

Claims

13 06 251. A facial expression recognition method based on an attention mechanism and self-distillation, comprising:SI: dividing a public dataset into a training set, a test set, and a validation set, wherein the training set, the test set, and the validation set each comprise images of seven basic expressions;S2: constructing a facial expression recognition model, wherein the facial expression recognition model comprises a feature extraction module, an adaptive channel attention module, a remodeling module, and a self-distillation network;S3: performing face alignment on the images of seven basic expressions in the training set, the test set, and the validation set to be input into the facial expression recognition model, and cropping the images to a fixed size; and performing image enhancement on the images in the training set to be input into the facial expression recognition model, to prevent the facial expression recognition model from overfitting;S4: inputting the images in the training set processed in step 3 into the feature extraction module, and training and optimizing a learnable parameter in the facial expression recognition model until accuracy of the facial expression recognition model is unable to be improved, wherein a specific training process comprises the following sub-steps:S41: inputting the images of seven basic expressions in the training set into the feature extraction module, and taking a feature map Zj output by each basic block in the feature extraction module as a feature output by the feature extraction module at each stage;S42: adding the adaptive channel attention module after the feature map Zj output by each basic block, wherein in the adaptive channel attention module, firstly, channel information of the feature map Zj output by each basic block is aggregated by performing maximum pooling and average pooling, to generate two different channel vectors qmax<qavg; secondly, weights wmax, wavg are generated respectively for qmax> qavg by an adaptive weight module; thirdly, a feature obtained after qmax and wmax are multiplied and a feature obtained after qavg and wavg are multiplied are fused through addition, to obtain a final channel weight feature fcw; and finally, fcw is multiplied by an input feature map Zj, to obtain a final output feature map Zj, wherein calculation formulas are as follows:qmax = MaxP001(Zj),Qavg = AvgP001(Zj),13 06 25Wmax ^ml (^(WmQ (C[max))),Wavg ^al (^(Wao(qaVg))),few ty(MLP(qmax) * Wmax + MLP(cjavg) * ^avg),Zy = few * zj >whereo represents a ReLU activation function, MLP(*) represents an output after a specified vector enters the MLP, Oj and BNj respectively represent the normalization layer and the activation layer, £ represents a parameter of a convolution kernel, W(*) represents a parameter of a fully connected layer;S43: remodelling, by the remodeling module, a feature output by a shallow basic block, and mapping a feature output by each basic block to a same dimension, to obtain a remodeled feature map Fj, wherein a calculation formula is as follows:Fj=aj(BNj^ConVj(Zj;E^y,S44: calculating a predicted score fj corresponding to the remodeled feature map Fj of each basic block through a fully connected layer and a Softmax function, wherein a calculation formula is as follows:fj = Softmax(W(Fj));S45: gradually performing, by the self-distillation network, knowledge distillation on the facial expression recognition model from deep to shallow based on a corresponding distillation loss, wherein a form of self-distillation is that except a last basic block, each basic block distills knowledge from a next basic block, and performs self-distillation from deep to shallow;S5: performing inference on the images of seven basic expressions in the test set and the validation set, specifically comprising: in an inference stage, removing the self-distillation module and the adaptive channel attention module, performing, by the feature extraction module after the training in a training stage, final prediction on an input, and inputting the images of seven basic expressions in the training set and the test set into the facial expression recognition model, to obtain a corresponding classification result, wherein the result is one of the seven basic expressions: hate, happiness, anger, fear, surprise, sadness, and neutral.

2. The facial expression recognition method based on an attention mechanism and selfdistillation according to claim 1, wherein step S45 comprises the following sub-steps:S451: measuring, based on a cross-entropy loss, a difference between a predicted score output by the last basic block and a real label of the training set, wherein a calculation formula is as follows:Lce = CrossEntropyLoss(fc, y);S452: measuring, based on a Kullback-Leibler (KL) divergence loss, a difference between a13 06 25predicted score obtained by a current basic block and a predicted score obtained by a next basic block of the current basic block, wherein a calculation formula is as follows:Lki = Sf=i KL(fc_j, fc_j+1);wherein, KL represents the KL divergence loss, and fc_j+1 represents update of cancellation of gradient when gradient back propagation is performed on the (c — j + 1 )th basic block;S453: measuring, based on an L2 loss, a difference between a feature output by the current basic block and a feature output by the next basic block of the current basic block, wherein a calculation formula is as follows:L2= S^IIFc-j -Fc_j+1||i;wherein, represents update of cancellation of gradient when gradient back propagation is performed on the (c — j + l)th basic block; andS454: obtaining a total loss based on the cross-entropy loss, the KL divergence loss, and the L2 loss, wherein a calculation formula is as follows:Ltotai Lce + a Lkl + P L2.

3. The facial expression recognition method based on an attention mechanism and selfdistillation according to claim 1, wherein the adaptive channel attention module comprises a channel attention module and an adaptive weight module; andthe channel attention module acts on an average pooling feature and a maximum pooling feature, determines importance of each channel by an attention mechanism, and performs weighting on each channel based on the importance of each channel; and the adaptive weight module acts on the average pooling feature and the maximum pooling feature, determines importance of the average pooling feature and the maximum pooling feature by the attention mechanism, and generates a corresponding weight.

Images