Eye contact identification method based on local and global feature mining

Abstract

The invention discloses an eye contact recognition method based on local and global feature mining, and the method comprises the following steps: obtaining full-face image data, and taking the full-face image data and eye state types corresponding to the full-face image data as an initial eye contact image data set; dividing the initial eye contact image data set into a training data set and a test data set; constructing an eye contact recognition network model based on local and global feature mining; training the eye contact recognition network model based on the eye contact image training data set until a set number of training times is reached, and obtaining a trained eye contact recognition model; and carrying out classification identification on the eye contact image test data set by adopting the trained eye contact identification model weight. According to the global learning enhanced feature provided by the invention, more diversified features can be learned, so that finer and deeper information of key parts such as eyes can be obtained, and the accuracy of eye state judgment is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision detection, and more specifically, to an eye contact recognition method based on local and global feature mining. Background Art

[0002] Eye contact is an important foundation for face-to-face social interaction. In the social process, the social behavior of making eye contact with the interaction partner is one of the earliest social skills of human beings. In social interaction, eye contact has many important functions, such as establishing relationships between partners, expressing interests and concerns, and identifying intentions. From the existing research, the research on social behavior exploration relies heavily on the face, which carries the expression of emotional interaction. In social interaction, the content of the face of the social object in an unconstrained natural situation is more dependent on the interaction between the interaction partners than the content of the unresponsive face (such as pictures or videos) described in typical experiments. It is more comprehensive and dynamic, and can more truly reflect the social behavior characteristics of the social interaction process.

[0003] Abnormal eye contact and gaze patterns are often a sign of many medical or psychiatric disorders, including autism spectrum disorder (ASD), ADHD, Williams syndrome, social anxiety / behavioral inhibition, and oppositional defiant disorder. For ASD, the Autism Diagnostic Observation Scale (ADOS) constantly tracks and quantifies eye contact, especially reduced eye contact has been included in the DSM-5 ASD diagnostic criteria and is also the focus of early screening and treatment. In real life, eye contact social behavior recognition in a natural and unconstrained environment can be widely used in the fields of eye contact social behavior analysis and disease detection during social interaction, and it also has the value of promoting human-computer interaction and communication in robots.

[0004] In natural social interactions, the recognition of eye contact social behavior is of great value, but efficient and simple detection and recognition methods are still being explored. At present, eye contact recognition methods can be divided into three types. The first is to use one or more cameras to record social interaction activities, and then manually judge the eye contact in the interaction activity records. Obviously, this method is highly subjective, and because it is a purely manual operation, it is time-consuming and labor-intensive, making it difficult to be widely used in the recognition and detection of large amounts of data; the second is to use wearable eye trackers for eye contact social behavior recognition, but the instrument price of this method is relatively expensive and limits the use of scenarios, and it will also affect the natural movement in social activities; the third is to use deep learning methods for recognition and detection. This method can be applied to the recognition and analysis of eye contact behavior in large-scale social activity data, and it is objective and effective. From the current research, the eye contact social behavior recognition method based on deep learning has shown great potential and broad development prospects in the application of real-life scenarios. This method can more accurately and objectively analyze and understand human social behavior, thereby playing an important role in a variety of practical scenarios and providing support for the intelligent analysis of social communication and interaction.

[0005] Research shows that in complex natural scenes, eye contact recognition detection errors mainly occur when the eyes of social objects are difficult to observe from the observed image. In this case, the extracted features cannot provide enough information for accurate prediction. In fact, under natural, unconstrained conditions, the movement of the head is complex and changeable. Some extreme head positions make it difficult to observe the eyes. When communicating face to face, social people wear glasses and other items that block the eyes to a certain extent, making it impossible to make a good recognition judgment of the eye state.

[0006] It is worth noting that most existing studies focus only on the features of a single recognition area, while ignoring the relative relationship between different areas in the entire facial image. However, in eye contact behavior recognition, the relationship between different facial recognition areas contains rich regional relative information, which can supplement the missing information of facial features in complex environments to a certain extent.

[0007] Eye contact recognition method that learns enhanced features and mines regional relationships. In order to solve the problem of missing eye features or facial information in eye contact recognition with complex postures (such as complex changes in head posture and occlusion of the eyes) in natural environments, this paper studies the construction of an eye contact recognition method that learns enhanced features and mines regional relationships. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide an eye contact recognition method based on local feature and global feature mining in view of the deficiencies of the above-mentioned prior art.

[0009] The technical solution of the present invention is:

[0010] An eye contact recognition method based on local and global feature mining comprises the following steps:

[0011] S1: Acquire full-face image data, and use the full-face image data and its corresponding eye contact state category as an initial eye contact image dataset;

[0012] S2: Divide the initial eye contact image dataset into a training dataset and a test dataset;

[0013] S3: Construct an eye contact recognition network model based on local and global feature mining;

[0014] S4: training the eye contact recognition network model based on the eye contact image training data set obtained in step S2 until a set number of training times is reached to obtain a trained eye contact recognition model;

[0015] S5: Use the trained eye contact recognition model to classify and recognize the eye contact image test dataset;

[0016] The eye contact images in the eye contact image test dataset are input into the trained eye contact recognition model, and a probability vector is output through the softmax classification module to indicate the probability of whether eye contact is recognized. The eye contact state category corresponding to the maximum probability value in the probability vector is selected as the recognition result of the eye contact state in the eye contact image test dataset, and the recognition result of each eye contact image in the test dataset is obtained.

[0017] Preferably, the eye state category corresponding to the full-face image data in S1 is annotated with the eye state category of each face image based on "whether there is eye contact", "yes" is marked as "1", and "no" is marked as "0", and the full-face image and its corresponding eye state category are used as the initial eye contact image data set.

[0018] Preferably, in S2, after dividing the training data set and the test data set, the image size of the eye contact image data set is adjusted to 224×224, and then data enhancement is performed to obtain enhanced eye contact image training data set and test data set.

[0019] Preferably, the implementation process of the eye contact recognition network model of S3 is:

[0020] S31: Global feature extraction: Define the input full face image X∈R H×W×3 , where H and W are the height and width of the image respectively, then for the full face image X∈R H×W×3The first four convolutional blocks of ResNet50 are used as the backbone network for global feature extraction, and the size of the extracted and output global features of the full face image is B×2048×7×7;

[0021] S32: Feature enhancement: Perform feature enhancement on the global feature X of the full face image, enhance the secondary discriminant area to obtain a diverse feature including the main discriminant feature and the confusion feature as the final feature and obtain a confidence score;

[0022] S33: Region Navigation: Based on the global feature X of the full face image, a series of rectangular regions {R 1 ′ ,R 2 ′ ,…R A ′}, the region proposal network reduces the dimension and downsamples the global feature X of the input full face image to form a multi-scale feature map. The features extracted by each layer of the feature map focus on different receptive fields of view, thereby capturing the information of objects of different sizes; each region has a score representing the amount of information in the region, as shown in the following formula, where A is the number of anchor points, I (R i ) is the i-th element in the sorted information list:

[0023] I(R 1 ) ≥ I (R 2 )≥…≥I(R A ) (4)

[0024] S34: Graph construction: Use multi-scale feature maps from different layers to generate information content of regions at different scales and ratios; then use non-maximum suppression (NMS) to filter regions according to their information content and select the top M regions with the richest information {R 1 ,R 2 ,...,R M}; After selecting the number of regional modules M, the local face image X is obtained by resizing M ∈R H×W×3 , regional features are obtained through feature extraction, which are used to represent node features B×M×2048, where B is the batch size, M is the number of nodes in the graph, and 2048 is the feature dimension of each node; using the obtained node features, inference and learning are performed from unstructured data to obtain a graph construction (GCN), which has the ability to reason about regional relationships. The multi-layer GCN of the graph construction processes the graph layer by layer, gradually aggregates and updates the node features, and mines the relative relationship between different regions; the output of each layer of GCN is the updated node feature matrix, and the update formula is generally:

[0025]

[0026] in represents the node feature matrix of the lth layer, A represents the adjacency matrix of the graph, and represents the learnable weights and biases, and σ represents the activation function;

[0027] S35: Eye contact recognition network model construction: Get the relevant information of the last layer of GCN, map the features to the dimension of the target category through the linear layer, combine the confidence score of the enhanced feature learning module, and finally get the eye contact recognition result.

[0028] Preferably, in step S32, feature enhancement of the global feature X of the full face image is achieved by combining the original learning branch and the enhanced learning branch.

[0029] The original learning branch and the enhanced learning branch share the global feature X of the whole face image, but the inputs of the two branches are different;

[0030] The original learning branch generates class activation maps and locates the main identification areas. After convolving the feature F by a 1×1 convolutional layer, the class activation map is obtained. , where C represents the number of categories, W and H represent the width and height of the image respectively; then Perform global average pooling to obtain the identified class activation map , where C represents the number of categories and p is the recognition vector The maximum index of , identifying the vector as follows:

[0031] (1)

[0032] Where g(·) is the global average pooling;

[0033] The reinforcement learning branch uses the class activation map Generate a mask to suppress the top-k discriminant positions of F; since the feature information of the top-k positions is suppressed, the branch focuses on other secondary distinguishable features, achieving the purpose of enhancing learning features;

[0034] Let B be The obtained mask and the input of the reinforcement learning branch are , as follows, where Represents the element-wise multiplication of two tensors:

[0035] (2)

[0036] Then the output of the reinforcement learning branch is obtained After that; let M be the output of the enhanced feature learning module, the obtained M is:

[0037] (3)

[0038] Finally, the confidence score is obtained after the average pooling layer

[0039] Compared with the prior art, the technical effects of the present invention are:

[0040] (1) The global learning enhanced feature module proposed in the present invention can learn more diverse features, thereby obtaining more subtle and deeper information about key parts such as the eyes, and improving the accuracy of eye state discrimination;

[0041] (2) The eye contact recognition method of mining regional relationships proposed in the present invention will focus on and explore the relative relationships between different regions, rather than focusing only on the features of a single recognition region. When facial information is lacking in complex environments, the relationship between regions can be used to supplement information and learn the behavior of attribute objects. This can effectively solve the problem of misjudgment of eye contact recognition caused by complex changes in head posture and occlusion of the eyes by the environment in natural environments, thereby improving the performance of eye contact recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Flowchart of the eye contact recognition method based on local and global feature mining

[0043] Figure 2 Schematic diagram of the eye contact recognition network model based on local and global feature mining.

[0044] Figure 3 Schematic diagram of the enhanced feature learning process. DETAILED DESCRIPTION

[0045] The drawings are only used for illustrative purposes and should not be construed as limiting the present invention. In order to better illustrate the present embodiment, some parts of the drawings may be omitted, enlarged, or reduced, and do not represent the size of the actual product.

[0046] For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the drawings. The technical solution of the present invention is further described below in conjunction with the drawings and embodiments.

[0047] like Figure 1 As shown, this embodiment provides an end-to-end eye contact recognition method based on local and global feature mining. This embodiment is different from the feature method that only focuses on a single recognition area. This embodiment only needs to input a full-face RGB image and use the relative information between the local and global facial discrimination areas to perform eye contact behavior recognition. The eye contact recognition result of this embodiment can achieve an improvement in the comprehensive performance of accuracy and F-measure.

[0048] The method of the present invention specifically comprises the following steps:

[0049] S1: Acquire full face image data and perform preprocessing;

[0050] S11: First, obtain multiple full-face image data, and mark the eye state category of each face image according to "whether there is eye contact", "yes" is marked as "1", and "no" is marked as "0", and the full-face image and its corresponding eye state category are used as the initial eye contact image data set;

[0051] S2: Divide the initial eye contact image dataset into a training dataset and a test dataset;

[0052] In this embodiment, the image sizes in the divided eye contact image dataset are adjusted to 224×224 and then data enhancement is performed to obtain enhanced eye contact image training dataset and test dataset;

[0053] S3: Construct an eye contact recognition network model based on local and global feature mining;

[0054] In this embodiment, the recognition method constructs an eye contact recognition network model that learns enhanced features and mines regional relationships, such as Figure 2 As shown in the figure, the eye contact recognition network model is mainly composed of two modules: global enhanced feature learning module and regional navigation and graph construction.

[0055] The specific steps include:

[0056] S31: Define the input full face image X∈R H×W×3 , where H and W are the height and width of the image respectively, and then the full face image X∈R is input H×W×3 They are input into the global feature extraction network respectively. The global feature extraction network uses the first four convolution blocks of ResNet50 as the backbone network and outputs the global feature X of the whole face image with a size of B×2048×7×7.

[0057] S32: Input the global image feature X into the reinforcement learning network. The purpose of the reinforcement learning network is to enable the network to extract more diverse features of key parts of the eye, such as Figure 2 As shown in the figure, the reinforcement learning network consists of an original learning branch and an reinforcement learning branch. The reinforcement learning network generates an inhibition mask based on the class activation map, inhibits the most distinguishable area, and obtains a diversified feature containing the main discriminative features and confusion features by enhancing the secondary discriminative area, which is used as the final feature.

[0058] The original learning branch and the enhanced learning branch share the global feature X of the whole face image, but the inputs of the two branches are different.

[0059] The original learning branch aims to generate class activation maps and locate the main discriminative areas. The branch convolves the extracted features F through a 1×1 convolutional layer to obtain the class activation map , where C represents the number of categories, W and H represent the width and height of the image respectively. Perform global average pooling to obtain the recognized class activation map , where C represents the number of categories and p is the recognition vector The maximum index of , identifying the vector as follows:

[0060] (1)

[0061] Where g(·) is the global average pooling. For the reinforcement learning branch, we use Generate a mask to suppress the top-k discriminant positions of F. Since the feature information of the top-k positions is suppressed, the branch focuses on other secondary distinguishable features, achieving the purpose of enhancing learning features.

[0062] Let B be The obtained mask and the input of the reinforcement learning branch are , as follows, where Represents the element-wise multiplication of two tensors:

[0063] (2)

[0064] Then the output of the reinforcement learning branch is obtained After that. Let M be the output of the reinforcement learning network, and the obtained M is:

[0065] (3)

[0066] Finally, the confidence score is obtained after the average pooling layer.

[0067] S33: Combination Figure 2 As shown in the figure, the region navigation and graph construction module obtains the global feature X of the whole face image from the global feature network as the shallow feature of the image. The region navigation network generates the proposed region of the recognized object from the image according to the shallow feature expression, constructs the graph structure of the object according to the position of the interaction area in the image, and further extracts features from the region and corresponds these features to the node features of the graph structure to form a complete graph data. The graph data not only contains the feature information of the image, but also contains the relative position relationship information between the image regions, which can effectively identify the social behavior recognition of eye contact in complex posture scenes.

[0068] The region navigation network uses the region proposal network (FPN) to propose a series of rectangular regions {R 1′ ,R 2 ′ ,…R A ′}, the region proposal network reduces the dimension and downsamples the input global feature X to form a multi-scale feature map. The features extracted by each layer of feature map focus on different receptive fields of view, thereby capturing information of objects of different sizes. Each region has a score indicating the amount of information in the region, as shown below, where A is the number of anchor points, I (R i ) is the i-th element in the sorted information list:

[0069] I(R 1 ) ≥ I (R 2 )≥...≥I(R A )(4)

[0070] S34: Using multi-scale feature maps from different layers, the network can generate information about regions at different scales and ratios. In order to reduce regional redundancy, non-maximum suppression (NMS) is used to filter regions according to their information content, and the top M regions with the richest information can be selected. 1 ,R 2 ,...,R M After selecting the number of regional modules M, the local face image X can be obtained by resizing M ∈R H×W×3 , regional features are obtained through the feature extraction network, which are used to represent the node features B×M×2048, where B is the batch size, M is the number of nodes in the graph, and 2048 is the feature dimension of each node. Using the obtained node features, the graph construction module (GCN) can infer and learn from unstructured data, and has the ability to reason about regional relationships. The multi-layer GCN of the graph construction module processes the graph layer by layer, gradually aggregates and updates the node features, and mines the relative relationships between different regions. The output of each layer of GCN is the updated node feature matrix, and the update formula is generally:

[0071]

[0072] in represents the node feature matrix of the lth layer, A represents the adjacency matrix of the graph, and represents the learnable weights and biases, and σ represents the activation function;

[0073] S35: Get the relevant information of the last layer of GCN, map the features to the dimension of the target category through the linear layer, combine with the confidence score of the enhanced feature learning module, and finally obtain the eye contact recognition result;

[0074] S4: training the eye contact recognition model based on the eye contact image training data set obtained in step S2 until a set number of training times is reached to obtain a trained eye contact recognition model;

[0075] S5: Classify and recognize the eye contact image test data set using the trained network model weights; specifically, input the eye contact images in the eye contact image test data set into the trained eye contact recognition model, and output a probability vector through the softmax classification module to represent the probability of whether eye contact is recognized, and select the eye contact state category corresponding to the maximum probability value in the probability vector as the recognition result of the eye contact state in the eye contact image test data set. After obtaining the recognition result of each eye contact image in the test data set, the recognition and classification effect of the trained eye contact recognition model can be evaluated.

[0076] The following is a further description of the technical effects of the present invention in combination with simulation experiments:

[0077] 1. Simulation conditions and contents:

[0078] Using the above technical solution, in order to verify the effectiveness of the eye contact recognition method based on local and global feature mining proposed in the present invention, comparative experiments were conducted. The experiments were conducted on the same experimental platform, that is, in the hardware environment of RTX 2080 GPU and the software environment of Window10. All experiments were developed based on the PyTorch deep learning framework. The experiment uses the public dataset OFDIW-HF. The OFDIW-HF dataset has great diversity in terms of different age levels of individuals, complexity of image scenes, degree of occlusion of faces, emotional distribution of individuals, gender distribution of individuals, and facial orientation of individuals. OFDIW-HF contains 9924 training images and 3309 test images;

[0079] The batch size of the model training is set to 16, the loss function uses SGD, the initial learning rate is 0.001, and the learning rate decay is 5E -5 , the number of training iterations epoch is uniformly set to 100, and the number of region proposals M is 6.

[0080] 2. Description of simulation process:

[0081] The dataset is trained and tested according to the set batch size (batchsize=16). Then, after the image data is input into the network model in the format of [16,3,224,224] (3 represents the number of channels of the image, and 224×224 is the height and width of the image), the specific simulation process is shown in Table 1. The classification result of size 16×2 is obtained through the feature fusion output of the model, corresponding to 2 eye contact status categories;

[0082] Table 1 Model simulation process

[0083]

[0084] 3. Analysis of simulation results:

[0085] The eye contact recognition method based on local and global feature mining proposed in this invention is different from some regional attention or regional information mining network methods in recent years, such as:

[0086] NTS-Net: [Yang Z, Luo T, Wang D, et al. Learning to navigate for fine-grained classification[C] / / Proceedings of the European conference on computervision (ECCV). 2018: 420-435.],

[0087] MMAL-Net: [

[31] Zhang F, Li M, Zhai G, et al. Multi-branch and multi-scale attention learning for fine-grained visual categorization[C] / / MultiMedia Modeling: 27th International Conference, MMM 2021, Prague, CzechRepublic, June 22–24, 2021, Proceedings, Part I 27. Springer InternationalPublishing, 2021: 136-147.]、

[0088] RARN: [

[32] Li

[0089] F-Net: [

[33] Nie X, Chai B, Wang L, et al. Learning enhanced features and inferring twice for fine-grained image classification[J]. MultimediaTools and Applications, 2023, 82(10): 14799-14813.]

[0090] Compared with the above methods, as shown in the table, the eye contact recognition method used in the study that learns enhanced features and mines regional relationships achieves the best results: Accuracy = 0.8550, F1 = 0.9058. The experimental results show that the method proposed in this embodiment can significantly improve the performance of eye contact social behavior recognition by mining the relationship between different facial regions.

[0091] Table 1 Results of comparative experiments on eye contact recognition methods based on local and global feature mining

[0092]

[0093] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. An eye contact recognition method based on local and global feature mining, characterized in that: The following steps are involved: S1: Acquire full-face image data, and use the full-face image data and its corresponding eye contact state category as an initial eye contact image dataset; S2: Divide the initial eye contact image dataset into a training dataset and a test dataset; S3: Construct an eye contact recognition network model based on local and global feature mining; S4: training the eye contact recognition network model based on the eye contact image training data set obtained in step S2 until a set number of training times is reached to obtain a trained eye contact recognition model; S5: Use the trained eye contact recognition model weights to classify and recognize the eye contact image test dataset; The eye contact images in the eye contact image test dataset are input into the trained eye contact recognition model, and a probability vector is output through the softmax classification module to indicate the probability of whether eye contact is recognized. The eye contact state category corresponding to the maximum probability value in the probability vector is selected as the recognition result of the eye contact state in the eye contact image test dataset, and the recognition result of each eye contact image in the test dataset is obtained.

2. The method according to claim 1, characterized in that The eye state category corresponding to the full face image data in S1 is based on "whether there is eye contact" to mark the eye state category of each face image, "yes" is marked as "1" and "no" is marked as "0", and the full face image and its corresponding eye state category are used as the initial eye contact image data set.

3. The method according to claim 2, characterized in that In S2, after dividing the training data set and the test data set, the image size of the eye contact image data set is adjusted to 224×224, and then data enhancement is performed to obtain an enhanced eye contact image training data set and a test data set.

4. The method according to claim 3, characterized in that: The implementation process of the eye contact recognition network model of S3 is as follows: S31: Global feature extraction: Define the input full face image X∈R H×W×3 , where H and W are the height and width of the image respectively, then for the full face image X∈R H×W×3 The first four convolutional blocks of ResNet50 are used as the backbone network for global feature extraction, and the size of the extracted and output global features of the full face image is B×2048×7×7; S32: Feature enhancement: Perform feature enhancement on the global feature X of the full face image, enhance the secondary discriminant area to obtain a diverse feature including the main discriminant feature and the confusion feature as the final feature and obtain a confidence score; S33: Region Navigation: Based on the global feature X of the full face image, a series of rectangular regions {R1 ′ ,R2 ′ ,…R A ′ }, the region proposal network reduces the dimension and downsamples the global feature X of the input full face image to form a multi-scale feature map. The features extracted by each layer of the feature map focus on different receptive fields of view, thereby capturing the information of objects of different sizes; each region has a score representing the amount of information in the region, as shown in the following formula, where A is the number of anchor points, I (R i ) is the i-th element in the sorted information list: I(R1)≥I(R2)≥…≥I(R A ) (4) S34: Graph construction: Use multi-scale feature maps from different layers to generate information content of regions at different scales and ratios; then use non-maximum suppression (NMS) to filter regions according to their information content, and select the top M regions {R1, R2, ..., R M }; After selecting the number of regional modules M, the local face image X is obtained by resizing M ∈R H×W×3 , regional features are obtained through feature extraction, which are used to represent node features B×M×2048, where B is the batch size, M is the number of nodes in the graph, and 2048 is the feature dimension of each node; using the obtained node features, inference and learning are performed from unstructured data to obtain a graph construction (GCN), which has the ability to reason about regional relationships. The multi-layer GCN of the graph construction processes the graph layer by layer, gradually aggregates and updates the node features, and mines the relative relationship between different regions; the output of each layer of GCN is the updated node feature matrix, and the update formula is generally: ； in represents the node feature matrix of the lth layer, A represents the adjacency matrix of the graph, and represents the learnable weights and biases, and σ represents the activation function; S35: Eye contact recognition network model construction: Get the relevant information of the last layer of GCN, map the features to the dimension of the target category through the linear layer, combine the confidence score of the enhanced feature learning module, and finally get the eye contact recognition result.

5. The method according to claim 4, characterized in that In step S32, feature enhancement of the global feature X of the full face image is achieved by combining the original learning branch and the enhanced learning branch. The original learning branch and the enhanced learning branch share the global feature X of the whole face image, but the inputs of the two branches are different; The original learning branch generates class activation maps and locates the main identification areas. After convolving the feature F by a 1×1 convolutional layer, the class activation map is obtained. , where C represents the number of categories, W and H represent the width and height of the image respectively; then Perform global average pooling to obtain the identified class activation map , where C represents the number of categories and p is the recognition vector The maximum index of , identifying the vector as follows: （1） Where g(·) is the global average pooling; The reinforcement learning branch uses the class activation map Generate a mask to suppress the top-k discriminant positions of F; since the feature information of the top-k positions is suppressed, the branch focuses on other secondary distinguishable features, achieving the purpose of enhancing learning features; Let B be The obtained mask and the input of the reinforcement learning branch are , as follows, where Represents the element-wise multiplication of two tensors: （2） Then the output of the reinforcement learning branch is obtained After that; let M be the output of the enhanced feature learning module, the obtained M is: （3） Finally, the confidence score is obtained after the average pooling layer.

Images