Method and device for occluded face recognition based on multi-attention multi-scale feature learning

Abstract

The application discloses a kind of based on multi-attention multi-scale feature learning's occlusion face recognition method and device.The method is based on channel attention and spatial attention mechanism, reduce the influence of occlusion on face recognition, solve the problem of low face recognition accuracy under the condition of occlusion.First, add multi-level attention network on the basis of traditional convolutional neural network, extract the channel attention map and spatial attention map of face image;Second, construct a multi-scale feature fusioner, integrate local and global information of face image;Next, locate the occlusion area and generate the occlusion mask through the occlusion mask generator, reduce the influence of occlusion area;Finally, through a multi-task learning network, the classification of occlusion category and the classification of face identity are carried out, and the final face recognition result is obtained.The application is simple and easy to implement, the model depth is relatively shallow, and the overhead is low, realizes accurate recognition of occlusion face while ensuring recognition efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence face recognition, and particularly relates to a blocked face recognition method and device based on multi-attention multi-scale feature learning. BACKGROUND

[0002] As a non-invasive identification verification method, face recognition is more popular and acceptable than other biometric recognition technologies. With the development and progress of recognition technology, face recognition technology has been widely deployed in various scenarios such as monitoring systems, security systems, industrial production, home care, and other aspects of people's life.

[0003] The accuracy of face recognition technology depends largely on the extraction of key features of the face by the model, and whether the face region is complete has a great influence on the feature extraction process. As an external interference factor, masks cause face pictures to be blocked, and thus cause some features to be damaged. In this case, the commonly used face recognition algorithm will lose its high accuracy, and ultimately cannot complete the mask face recognition task. Therefore, there is an urgent need to propose a new research algorithm for the recognition of blocked faces. SUMMARY

[0004] The purpose of the present application is to provide a blocked face recognition method and device based on multi-attention multi-scale feature learning. The present application extracts and fuses multi-level features of face images, while using channel attention and spatial attention mechanisms to eliminate the influence of blocked areas on face recognition, providing a method for effectively improving the accuracy of blocked face recognition. The present application has simple logic and significant effect, and can effectively shield the adverse effects of partial blocking on face recognition, while also supporting face recognition tasks in unblocked scenarios.

[0005] The present application is implemented by adopting the following technical solutions:

[0006] The blocked face recognition method based on multi-attention multi-scale feature learning first adds a multi-level attention network to the convolutional neural network to extract channel attention maps and spatial attention maps of face images. Second, a multi-scale feature fusioner is constructed to integrate local and global information of face images to obtain face features with stronger robustness. Third, a blocked mask generator is used to locate the blocked area and generate a blocked mask to reduce the influence of the blocked area. Finally, a multi-task learning network is used to simultaneously classify the blocked category and the face identity to achieve the best face recognition generalization effect.

[0007] The present application adopts a blocked face recognition method based on multi-attention multi-scale feature learning, which specifically includes the following steps:

[0008] Extracting face image features: a residual neural network is used as a face image feature extraction module to extract face image features;

[0009] Obtaining a channel attention map of a face image: after obtaining the face image features, the attention weights are calculated for the feature maps of each channel to obtain the relevance of each channel to the key information;

[0010] Obtaining a spatial attention map of a face image: the feature map after refining the channel attention map is taken as input to calculate the relevance of different pixel points to the key information, and a multi-level feature of the face image is obtained;

[0011] Multi-scale feature fusion of a face image: for the multi-level feature of the face image, a multi-scale feature fusioner is constructed using a three-layer deconvolution structure, and the multi-scale feature information of the face image containing different resolutions and semantic strengths is obtained by element-wise addition of feature maps of different scales;

[0012] Obtaining a channel attention map of a face image: after obtaining the face image features, the attention weights are calculated for the feature maps of each channel to obtain the relevance of each channel to the key information;

[0013] Obtaining a channel attention map of a face image: after obtaining the face image features, the attention weights are calculated for the feature maps of each channel to obtain the relevance of each channel to the key information;

[0014] Face class classification: the region weight obtained by the occlusion mask generator is multiplied by the multi-scale feature information of the face image to obtain the cleaned face feature, and the cleaned face feature is taken as the input of the face class classifier to obtain the classification result of the face class.

[0015] The further improvement of the present application is that the channel attention map of the face image is obtained, specifically comprising:

[0016] The extracted face features are respectively subjected to average pooling and maximum pooling operations to aggregate spatial information, and are input into two shared fully connected layers to fit the correlation between the channel features, and two channel feature maps are obtained;

[0017] The corresponding elements in the two channel feature maps are added, and are processed using a Sigmoid activation function to obtain the channel attention map of the face image, wherein the weight in the map reflects the relevance of the channel to the key information.

[0018] The further improvement of the present application is that the spatial attention map of the face image is obtained, comprising:

[0019] The extracted face features are subjected to maximum pooling and average pooling operations along the channel direction to obtain two spatial feature maps;

[0020] The two spatial feature maps are spliced, and the feature correlation in the spatial dimension is fitted through convolution operation to obtain a spatial attention map, and the weight in the map reflects the correlation degree of different pixel points and key information.

[0021] The further improvement of the present application is that the multi-scale feature fusion of the face image is specifically as follows:

[0022] The face image feature extraction module is taken as the main body of the multi-scale feature fusioner, and a pyramid structure model is constructed using a top-down horizontal connection architecture.

[0023] The input of the pyramid structure model is a preprocessed face image, and the multi-scale feature information of the face image containing different resolutions and semantic strengths is obtained through convolution operation and up-sampling operation.

[0024] The further improvement of the present application is that the occlusion mask generation is specifically as follows:

[0025] The face features containing different scales and global information are input, and the final occlusion mask is obtained through a convolution network combined with a PReLu activation function, a batch regularization layer and a Sigmoid function, which is used to clean the original face features damaged by partial occlusion.

[0026] The further improvement of the present application is that the occlusion category classification is specifically as follows:

[0027] The face picture is divided into a plurality of rectangular grids, the occlusion area is simulated through rectangular combination, and a new occlusion category is constructed, and based on this, an occlusion dictionary of all occlusion categories is obtained, wherein the no-occlusion condition is still included.

[0028] Different categories of mask pictures are selected as the occlusion objects, and the occlusion object pictures are integrated on the face picture by randomly selecting the center of the occlusion object.

[0029] According to whether each grid is occluded, the corresponding occlusion matrix is calculated, and the corresponding occlusion category is found in the generated occlusion dictionary as the label of the occluded face picture.

[0030] The labeled occluded face picture is sent to the occlusion mask generator to learn the mask related to the occlusion category.

[0031] The learned mask is sent to the occlusion category classifier for classification, and cross entropy is used as a loss function to supervise the learning process of the occlusion mask generator to obtain a more accurate occlusion mask.

[0032] The further improvement of the present application is that the face category classification is specifically as follows:

[0033] The face features processed by the occlusion mask are input, and an edge-based loss function LMCL is used to supervise the model learning of the face features related to the identity.

[0034] Finally, the loss function of the face recognition task and the loss function of the occlusion category recognition task are added as the final loss function, so that the model converges faster and completes face category classification.

[0035] The occluded face recognition device based on multi-attention multi-scale feature learning comprises:

[0036] The face image feature extraction module adopts a residual neural network as the face image feature extraction module to extract face image features.

[0037] The channel attention map construction module calculates the attention weight of each channel feature map after obtaining the face image features, and obtains the correlation degree of each channel and key information.

[0038] The spatial attention map construction module takes the feature map refined by the channel attention map as input, calculates the correlation degree of different pixel points and key information, and obtains the multi-level features of the face image.

[0039] The multi-scale feature fusion module uses a three-layer deconvolution structure to construct a multi-scale feature fusion device for the multi-level features of the face image, and obtains face image multi-scale feature information containing different resolutions and semantic strengths by element-wise addition of feature maps of different scales.

[0040] The occlusion mask generation module learns a feature mask highly sensitive to the occlusion position of the input image, calculates the weight corresponding to the occlusion area, and eliminates the influence of damaged features on face recognition by assigning different weights to the features.

[0041] The occlusion category classification module takes the feature mask learned by the occlusion mask generator as input, classifies it into an occlusion category, and supervises the learning of the occlusion mask generator.

[0042] The face category classification module multiplies the region weight obtained by the occlusion mask generator and the face image multi-scale feature information to obtain the cleaned face features, and takes the cleaned face features as the input of the face category classifier to obtain the classification result of the face category.

[0043] The present application has at least the following beneficial technical effects:

[0044] The application provides a method for occluded face recognition based on multi-attention multi-scale feature learning. The method first adds a multi-level attention mechanism to a face feature extraction network to extract a channel attention map and a spatial attention map of a face image; secondly, a multi-scale feature fusioner is constructed to integrate local and global information of the image; next, an occlusion mask generator is used to locate an occlusion area and generate an occlusion mask to reduce the influence of the occlusion area; finally, a multi-task learning network is used for classification of the occlusion category and the face identity to obtain a final face recognition result. Compared with ordinary face recognition algorithms, the method can reduce the influence of masks and other occlusions on the face recognition accuracy. Experiments show that the method achieves an accuracy of 97.76% on the occluded face task, which is better than existing occluded face recognition algorithms. On the non-occluded face recognition task, the accuracy of the method is comparable to existing methods.

[0045] The application also provides an occluded face recognition device based on multi-attention multi-scale feature learning, which includes seven modules: a face feature extraction module, a channel attention map construction module, a spatial attention map construction module, a multi-scale feature fusion module, an occlusion mask generation module, an occlusion classification module, and a face classification module. The face feature extraction module provides deep face features for subsequent modules; the channel attention and spatial attention modules provide extraction methods for channel attention maps and spatial attention maps; the multi-scale feature fusion module can fuse local and global information of the image to obtain multi-scale feature information of the face image that is more conducive to recognition; the occlusion mask generation module can locate the occlusion area and generate an occlusion mask to reduce the influence of the occlusion area and improve classification accuracy; the occlusion classification module is used for classifying the occlusion mask to effectively supervise the process of generating the occlusion mask; and the final face classification module can realize effective classification of the face category by inputting the cleaned face features to improve classification efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 is the overall processing flowchart of the occluded face recognition algorithm based on the attention mechanism;

[0047] Figure 2 is a channel attention map extraction process diagram;

[0048] Figure 3 is a spatial attention map extraction process diagram;

[0049] Figure 4 is a multi-level feature fusion module diagram;

[0050] Figure 5 is a confusion matrix of different algorithms on LFW and Occ-LFW data sets;

[0051] Figure 6The device function module diagram of the occluded face recognition based on multi-attention scale feature learning provided by the application is shown in the figure.

[0052] Figure 7 The structural schematic diagram of the electronic device for implementing the occluded face recognition algorithm based on multi-attention scale feature learning is shown in the figure. DETAILED DESCRIPTION

[0053] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0054] Referring to Figure 1 , the present application provides an occluded face recognition based on multi-attention multi-scale feature learning. The method first extracts features from an input face image, and calculates a channel attention map and a spatial attention map of the face image according to a multi-level attention mechanism. Then, multi-scale feature fusion is performed to obtain feature information containing different resolutions and semantic strengths. An occlusion mask generator is used to learn the occlusion mask of the face image in the occlusion category recognition task branch, and the mask is sent to an occlusion category classifier to supervise the learning process of the mask generator. The face recognition task branch superimposes the occlusion mask learned by the occlusion category recognition task branch on the original face features to eliminate the adverse effects of occlusion on recognition, and completes the occluded face recognition task. The method specifically includes the following modules:

[0055] 1. Multi-level feature extraction of face image: including face image feature extraction, channel attention map acquisition and spatial attention map acquisition, which specifically includes the following steps:

[0056] Step 1: Extract face image features: a residual neural network is used as a face image feature extraction network to extract face image features.

[0057] Step 2: Obtain the channel attention map of the face image: referring to Figure 2 , after obtaining the face image features, the attention weights are calculated for the feature maps of each channel to obtain the correlation degree of each channel and the key information. The specific steps are as follows:

[0058] First, the input is respectively subjected to average pooling and maximum value pooling to aggregate spatial information, and feature maps and are obtained. Secondly, the feature maps are combined to obtain the channel attention map of the face image. and Two shared fully connected layers are inputted to fit the correlation between channel features:

[0059]

[0060]

[0061] where (W0, b0, W1, b1) represent the weights and biases of the two fully connected layers respectively. where r is the compression ratio to reduce the number of parameters.

[0062] Finally, the corresponding elements of formula (1) and (2) are added, and processed using the Sigmoid activation function to obtain the final channel attention map M c (F). The process is shown in the following formula.

[0063]

[0064] where, i.e. the Sigmoid activation function.

[0065] Step 3: Obtain the spatial attention map of the face image: refer to Figure 3 The feature map refined by the channel attention map is inputted to calculate the correlation between different pixel points and key information. The specific steps are as follows:

[0066] The input of this module is the feature map refined by the channel attention map The output is the spatial weight map M s First, maximum and average pooling operations are performed on F along the channel direction to obtain feature maps and Then, the average pooling result and the maximum pooling result are spliced to obtain data with a new channel number of 2, which is sent to a layer of convolutional neural network to fit the feature correlation in the spatial dimension. The process is shown in the following formula.

[0067]

[0068] where, represents a convolution operation with a convolution kernel of k1xk2.

[0069] 2. Multi-scale feature fusion of face image: refer to Figure 4, the face image feature extraction module in step 1 is taken as the main body of the multi-scale feature fusioner, and a pyramid structure is constructed using a top-down horizontal connection architecture. The model takes the processed face image as input and outputs face features x1, x2 and x3 of different scales. Wherein x1 is the bottom layer face recognition feature that needs to be cleaned up, and x2 and x3 contain local and global information of different scales. The process can be formally expressed as follows

[0070] x2 = conv (upsample (conv (x1)) + conv (C2)) (5)

[0071] x3 = conv (upsample (conv (x2)) + conv (C3)) (6)

[0072] Wherein conv is a convolution operation, and upsample is an upsample operation.

[0073] 3. Occlusion mask generation: learn the feature mask that is highly sensitive to the occlusion position of the input image, calculate the weight corresponding to the occlusion area, and eliminate the influence of the damaged feature on face recognition by giving different weights to the feature.

[0074] 4. Multi-task occlusion face classification model: including two sub-tasks, face class classification and occlusion class classification. The specific steps are as follows:

[0075] Step 1: Occlusion class classifier: take the feature mask learned by the occlusion mask generator as input, classify it into an occlusion class, and supervise the learning of the occlusion mask generator.

[0076] Step 2: Face class classification: multiply the region weight calculated by the occlusion mask generator with the multi-scale feature information of the face image to obtain the cleaned face feature, which is taken as the input of the face class classifier to obtain the classification result of the face class.

[0077] Referring to Table 1, compared with other algorithms in the field of face recognition, the algorithm proposed in the present application is basically the same as Arcface in the comparison of unoccluded face, and is improved by 0.4% compared with the FROM algorithm in the field of occluded face recognition. In the comparison of occluded face, the accuracy of the algorithm in this paper is improved by 1.2% compared with Arcface, and is improved by 1% compared with the occluded face recognition algorithm FROM. This shows the effectiveness of the occluded face recognition algorithm based on attention mechanism in face comparison.

[0078] Table 1: Comparison of face recognition accuracy of the present application and other face recognition algorithms ArcFace and FROM on LFW and Occ-LFW data sets.

[0079]

[0080]

[0081] To further evaluate the performance of the algorithm compared with other algorithms, the confusion matrix of the three algorithms on the LFW dataset and the Occ-LFW dataset is further obtained, referring to Figure 5 On the LFW dataset, FROM is most likely to confuse the same face pair with the different face pair, and Arcface is least likely to confuse the two. In the same face pair comparison, the performance of the application is better than FROM, and the performance is similar to Arcface; in the different face pair comparison, the performance of the application is obviously better than FROM, but slightly inferior to Arcface. On the Occ-LFW dataset, Arcface is most likely to confuse the same face pair with the different face pair, and the application is least likely to confuse the two. In the same face pair comparison and the different face pair comparison, the performance of the application is better than FROM and Arcface, which fully illustrates the effectiveness of the attention mechanism based on the occluded face recognition algorithm.

[0082] The occluded face recognition device based on multi-attention multi-scale feature learning provided by the application comprises a face feature extraction module, a channel attention map construction module, a spatial attention map construction module, a multi-scale feature fusion module, an occlusion mask generation module, an occlusion classification module and a face classification module.

[0083] 1. The face image feature extraction module adopts a residual neural network as the face image feature extraction module to extract face image features.

[0084] 2. The channel attention map construction module calculates the attention weight for each channel feature map after obtaining the face image features, and obtains the correlation degree of each channel and key information.

[0085] 3. The spatial attention map construction module takes the feature map after the channel attention map is refined as input, calculates the correlation degree of different pixel points and key information, and obtains multi-level features of the face image.

[0086] 4. The multi-scale feature fusion module uses a three-layer deconvolution structure to construct a multi-scale feature fusioner for the multi-level features of the face image, and obtains multi-scale feature information of the face image containing different resolutions and semantic strengths by element-wise addition of feature maps of different scales.

[0087] 5. The occlusion mask generation module learns a feature mask highly sensitive to the occlusion position of the input image, calculates the weight corresponding to the occlusion area, and eliminates the influence of the damaged features on face recognition by giving different weights to the features.

[0088] 6. An occlusion category classification module, taking the feature mask learned by the occlusion mask generator as input, classifying it into an occlusion category, to supervise the learning of the occlusion mask generator;

[0089] 7. A face category classification module, multiplying the region weight obtained by the occlusion mask generator with the multi-scale feature information of the face image, obtaining the cleaned face feature, and taking it as the input of the face category classifier, to obtain the classification result of the face category.

[0090] Although the present application has been described in detail with general description and specific embodiments above, some modifications or improvements can be made on the basis of the present application, which is obvious to those skilled in the art. Therefore, these modifications or improvements made on the basis of not deviating from the spirit of the present application, all belong to the scope of protection claimed by the present application.

Claims

1. A method for occluded face recognition based on multi-attention multi-scale feature learning, characterized in that, The method first adds a multi-level attention network on the basis of a convolutional neural network to extract a channel attention map and a spatial attention map of the face image; secondly, a multi-scale feature fusioner is constructed to integrate local and global information of the face image to obtain a face feature with stronger robustness; then, a mask generator is used to locate a blocked area and generate a mask to reduce the influence of the blocked area; finally, a multi-task learning network is used to simultaneously classify the blocked area and the face identity to obtain the best face recognition generalization effect; the method specifically comprises the following steps: extracting a face image feature: a residual neural network is used as a face image feature extraction module to extract a face image feature; obtaining a channel attention map of the face image: after obtaining the face image feature, attention weights are calculated for feature maps of each channel to obtain a correlation degree of each channel and key information; obtaining a spatial attention map of the face image: the feature map after channel attention map refinement is taken as input to calculate a correlation degree of different pixel points and key information to obtain a multi-level feature of the face image; multi-scale feature fusion of the face image: a three-layer deconvolution structure is used to construct a multi-scale feature fusioner for the multi-level feature of the face image, and the feature maps of different scales are added element by element to obtain multi-scale feature information of the face image containing different resolutions and semantic strengths; mask generation: a feature mask highly sensitive to the blocked position of the input image is learned, a weight corresponding to the blocked area is calculated, and the influence of the damaged feature on face recognition is eliminated by assigning different weights to the feature; blocked area classification: the feature mask learned by the mask generator is taken as input and classified into a blocked area to supervise the learning of the mask generator; face classification: the region weight obtained by the mask generator is multiplied by the multi-scale feature information of the face image to obtain a cleaned face feature, and the face feature is taken as input of a face classification classifier to obtain a classification result of the face; the multi-scale feature fusion of the face image specifically comprises: the face image feature extraction module is taken as the main body of the multi-scale feature fusioner, and a pyramid structure model is constructed using a top-down horizontal connection architecture; the input of the pyramid structure model is a preprocessed face image, and multi-scale feature information of the face image containing different resolutions and semantic strengths is obtained through convolution operation and up-sampling operation; the mask generation specifically comprises: the face feature containing different scales and global information is input into a convolutional network combined with a PReLu activation function, a batch normalization layer and a Sigmoid function to obtain a final mask for cleaning the original face feature damaged by partial blocking.

2. The method of claim 1, wherein the method of occluded face recognition based on multi-attention multi-scale feature learning is characterized in that, the channel attention map of the face image is obtained specifically by: the extracted face feature is subjected to average pooling and maximum pooling operations to aggregate spatial information, and the spatial information is input into two shared fully connected layers to fit the correlation between the channel features to obtain two channel feature maps. The corresponding elements in the two channel feature maps are added, and a channel attention map of the face image is obtained by using a Sigmoid activation function, wherein the weight in the map reflects the correlation between the channel and the key information.

3. The method of claim 1, wherein the method of occluded face recognition based on multi-attention multi-scale feature learning is characterized by, The spatial attention map of the face image is obtained, including: The extracted face features are subjected to maximum pooling and average pooling operations in the channel direction to obtain two spatial feature maps; The two spatial feature maps are spliced, and the feature correlation in the spatial dimension is fitted through a convolution operation to obtain a spatial attention map, wherein the weight in the map reflects the correlation between different pixel points and the key information.

4. The method of claim 1, wherein the method of occluded face recognition based on multi-attention multi-scale feature learning is characterized by, The occlusion category classification specifically includes: The face picture is divided into a plurality of rectangular grids, the rectangular combination is used to simulate the occlusion area and construct a new occlusion category, and based on this, an occlusion dictionary of all occlusion categories is obtained, which still contains the case of no occlusion; Different categories of mask pictures are selected as the occlusion object, and the occlusion object picture is integrated on the face picture by randomly selecting the center of the occlusion object; According to whether each grid is occluded, the corresponding occlusion matrix is calculated, and the corresponding occlusion category is found in the generated occlusion dictionary as the label of the occluded face picture; The occluded face picture with the label is sent to the occlusion mask generator to learn the mask related to the occlusion category; The learned mask is sent to the occlusion category classifier for classification, and the cross-entropy is used as the loss function to supervise the learning process of the occlusion mask generator to obtain a more accurate occlusion mask.

5. The method of claim 1, wherein the method of occluded face recognition based on multi-attention multi-scale feature learning is characterized by, The face category classification specifically includes: The face features processed by the occlusion mask are input, and an edge-based loss function LMCL is used to supervise the model to learn the face features related to the identity; Finally, the loss function of the face recognition task and the loss function of the occlusion category recognition task are added as the final loss function to supervise the model to make it converge faster and complete the face category classification.

6. The device for occluded face recognition based on multi-attention multi-scale feature learning, characterized in that, The device is based on the occluded face recognition method based on multi-attention multi-scale feature learning of claim 1, comprising: A face image feature extraction module uses a residual neural network as the face image feature extraction module to extract face image features; A channel attention map construction module calculates the attention weight for each channel feature map after obtaining the face image features to obtain the correlation between each channel and the key information; A spatial attention map construction module takes the feature map after the channel attention map is refined as input to calculate the correlation between different pixel points and the key information to obtain multi-level features of the face image; A multi-scale feature fusion module uses a three-layer deconvolution structure to construct a multi-scale feature fusion device for the multi-level features of the face image, and adds the feature maps of different scales element by element to obtain multi-scale feature information of the face image containing different resolutions and semantic strengths; An occlusion mask generation module learns a feature mask that is highly sensitive to the occlusion position of the input image, calculates the weight corresponding to the occlusion area, and eliminates the influence of the damaged features on face recognition by giving different weights to the features; An occlusion category classification module takes the feature mask learned by the occlusion mask generator as input, classifies it into an occlusion category, and supervises the learning of the occlusion mask generator. The face class classification module multiplies the region weight obtained by the occlusion mask generator and the multi-scale feature information of the face image to obtain the cleaned face feature, and takes the cleaned face feature as an input of a face class classifier to obtain a classification result of the face class.

Images