Face recognition method based on cross attention feature fusion convolutional neural network

Abstract

The invention discloses a face recognition method based on a cross attention feature fusion convolutional neural network, and the method comprises the steps: carrying out the face recognition through employing the constructed cross attention feature fusion convolutional neural network, and extracting the global features of a face image through employing a VGG16 model and a ResNet model of the convolutional neural network, a local binary pattern (LBP) is adopted to extract local features in a face image, the face features are completely described from different angles, key information can be better focused, the robustness and recognition efficiency of a cross attention feature fusion convolutional neural network are improved, a cross attention mechanism is adopted to dynamically adjust the contribution of local and global features, and the recognition efficiency is improved. Therefore, the expression ability of the features is enhanced, and the recognition rate and the understanding ability of the cross attention feature fusion convolutional neural network are improved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of image data processing, and particularly relates to a face recognition method based on a cross-attention feature fusion convolutional neural network. BACKGROUND

[0002] Face recognition technology has been widely concerned due to its application value in the field of image analysis and recognition. Face recognition is an important application of image processing technology in the field of biological characteristics, and has the characteristics of non-contact, uniqueness, high precision and high convenience, and has been deeply integrated into social governance, commercial service machine technology innovation, such as city security, community management identity verification, automatic early warning of suspicious persons, intelligent park and the like. With the rapid development of deep learning technology, the convolutional neural network (CNN) has become the dominant algorithm in the field of face recognition due to its excellent feature extraction and pattern recognition capability. Face feature extraction is the most important step of face recognition, and effective extraction of face features is the key to improving the accuracy of face recognition.

[0003] The convolutional neural network has achieved remarkable results by automatically extracting multi-level features, but there are still some deficiencies and challenges in face recognition feature extraction. Due to the spatial invariance, the face detail information is lost, the convolutional neural network realizes the translational invariance through the pooling layer and the down-sampling operation, but destroys the geometric relationship between the local fine features and the global features, and cannot provide multi-resolution analysis like wavelet transform. In low-quality video monitoring, the local texture of the face is blurred due to compression distortion, and the CNN may misjudge it as different people. At the same time, in the twin recognition task, the CNN which only depends on the global features is difficult to distinguish the subtle facial features. The recognition ability of the convolutional neural network for small sample data is insufficient, the generalization ability of the network may be limited, the recognition ability for complex scenes is weak, the low-level detail features are lost, and the performance is significantly reduced in the low-quality scene. SUMMARY

[0004] The technical problem to be solved by the application is to provide a face recognition method based on a cross-attention feature fusion convolutional neural network, which fuses the extracted global and local features through a cross-attention mechanism, significantly improves the face recognition performance of the cross-attention feature fusion convolutional neural network in a complex scene, and makes the face recognition technology have high application value in a complex scene.

[0005] The technical scheme of the application is as follows:

[0006] The face recognition method based on the cross-attention feature fusion convolutional neural network specifically comprises the following steps:

[0007] (1) a plurality of face images are collected, and then image enhancement and image preprocessing are performed to construct a face image dataset;

[0008] (2), the cross attention feature fusion convolutional neural network comprises a global feature extraction module, a local feature extraction module, a cross attention mechanism, a weighted feature fusion module and a Softmax classifier,

[0009] The global feature extraction module extracts global features f1 and f2 of the face image by using a convolutional neural network VGG16 model and a ResNet model respectively.

[0010] The local feature extraction module extracts local texture features in the face image as local features f3 by using a local binary pattern (LBP).

[0011] The cross attention mechanism calculates three attention weights based on the global features f1, f2 and local features f3.

[0012] The weighted feature fusion module performs weighted feature fusion on the global features f1, f2 and local features f3 based on the three attention weights to obtain weighted fusion features.

[0013] The Softmax classifier extracts face feature vectors based on the weighted fusion features and performs feature classification to output face recognition results.

[0014] (3), the loss function is used to train the cross attention feature fusion convolutional neural network, and the face image to be recognized is input into the trained cross attention feature fusion convolutional neural network for detection and recognition to obtain the face recognition result.

[0015] The image enhancement is a random rotation transformation of the collected multiple face images to obtain multiple times of face images; the image preprocessing is data cleaning of the face images, balancing the class distribution, and then performing size standardization processing to unify the size of all collected face images.

[0016] The cross attention mechanism calculates three attention weights based on the global features f1, f2 and local features f3, which comprises the following steps:

[0017] S11, after linear mapping of the global feature f1, a value vector V f1 is obtained; after linear mapping of the global feature f2, a query vector Q f2 is obtained; after linear mapping of the local feature f3, a key vector K f3 is obtained; then the first attention score A f1 is calculated according to the following formula (1):

[0018]

[0019] In formula (1), W V1 , W Q1 , W K1 , G1 are weight matrices; D represents vector dimension;

[0020] S12, after linear mapping of the global feature f1, a query vector Q f1 is obtained; after linear mapping of the global feature f2, a value vector V f2 is obtained; after linear mapping of the local feature f3, a key vector K' is obtained; then the second attention score A f3 is calculated according to the following formula (2): f2 ;

[0021]

[0022] In formula (2), Q V2 , W Q2 , W K2 , G2 are weight matrices; D represents vector dimension;

[0023] S13, after processing of the local feature f3 by the multi-layer perception MLP, a third attention score A f3 is obtained, and the specific formula (3) is as follows:

[0024] A f3 = MLP (f3) (3);

[0025] S14, the first attention score A f1 , the second attention score A f2 and the third attention score A f3 are respectively normalized by the softmax function, and the attention weights w1, w2 and w3 are obtained, and the specific formula (4) is as follows:

[0026]

[0027] In formula (4), ∑ i w i = 1, i = 1, 2, 3.

[0028] The weighted feature fusion module performs weighted feature fusion on the global feature f1, the global feature f2 and the local feature f3 based on the three attention weights, and the specific formula (5) is as follows:

[0029]

[0030] In formula (5), F fused represents the weighted fusion feature.

[0031] The Softmax classifier comprises two full connection layers and a softmax classification function, the two full connection layers are used for dimension reduction processing on the weighted fusion features, and the softmax classification function is used for integrating and classifying the input features.

[0032] The cross-attention feature fusion convolutional neural network is trained by using a cross-entropy loss function to obtain the trained cross-attention feature fusion convolutional neural network.

[0033] Advantages of the present application:

[0034] (1), the present application respectively adopts convolutional neural network VGG16 model and ResNet model to extract global features of face image, adopts local binary pattern LBP to extract local features in face image, and fully describes face features from different angles, can better focus on key information, and improves the robustness and recognition efficiency of cross-attention feature fusion convolutional neural network.

[0035] (2), the present application is further improved in order to further improve the fusion degree, improve the recognition rate and understanding ability of cross-attention feature fusion convolutional neural network, cross-attention mechanism is proposed, the attention of input feature vector is learned, the contribution of local and global features is dynamically adjusted, so as to enhance the expression ability of features, improve the performance of image classification, and effectively fuse the information from different features through the weighted feature fusion module, improve the recognition rate and understanding ability of cross-attention feature fusion convolutional neural network. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flow chart of the present application.

[0037] Figure 2 is a network block diagram of the cross-attention mechanism of the present application. DETAILED DESCRIPTION

[0038] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0039] See Figure 1 , the face recognition method based on cross-attention feature fusion convolutional neural network, specifically comprising the following steps:

[0040] (1) Collect multiple face images, then perform image enhancement and image preprocessing to construct a face image dataset; image enhancement is to use torchvision.transforms of PyTorch to transform the collected multiple face images by random rotation (-15°-15°) to obtain twice the number of face images; image preprocessing is to clean the face image data, balance the class distribution, and then perform size standardization processing to unify the size of all face images to 224x224; 20% of the face image dataset is used as the test set, and the remaining 80% is used as the training set;

[0041] (2) Construct a cross-attention feature fusion convolutional neural network, which includes a global feature extraction module, a local feature extraction module, a cross-attention mechanism, a weighted feature fusion module, and a Softmax classifier.

[0042] The global feature extraction module uses the convolutional neural network VGG16 model and the ResNet model to extract the global features of the face image, obtaining global features f1 and global features f2.

[0043] The local feature extraction module uses the local binary pattern (LBP) to extract the local texture features of the key regions such as eyes, nose, and mouth in the face image as local features f3.

[0044] The cross-attention mechanism calculates three attention weights based on the global features f1, global features f2, and local features f3.

[0045] The weighted feature fusion module performs weighted feature fusion on the global features f1, global features f2, and local features f3 based on the three attention weights to obtain weighted fusion features.

[0046] The Softmax classifier extracts face feature vectors based on the weighted fusion features and performs feature classification to output face recognition results; the Softmax classifier includes two fully connected layers and one softmax classification function, the two fully connected layers are used for dimension reduction processing of the weighted fusion features, and the softmax classification function is used for integration and classification of the input features.

[0047] (3) Use the cross-entropy loss function to train the cross-attention feature fusion convolutional neural network, input the face image to be recognized into the trained cross-attention feature fusion convolutional neural network for detection and recognition to obtain the face recognition result.

[0048] See Figure 2 The cross-attention mechanism calculates three attention weights based on the global features f1, global features f2, and local features f3.

[0049] S11, the global feature f1 is linearly mapped to obtain a value vector V f1 ; the global feature f2 is linearly mapped to obtain a query vector Q f2 ; the local feature f3 is linearly mapped to obtain a key vector K f3 ; then the first attention score A f1 is calculated according to the following formula (1)

[0050]

[0051] In formula (1), W V1 , W Q1 , W K1 , and G1 are weight matrices; and D represents the vector dimension.

[0052] S12, the global feature f1 is linearly mapped to obtain a query vector Q f1 ; the global feature f2 is linearly mapped to obtain a value vector V f2 ; the local feature f3 is linearly mapped to obtain a key vector K' f3 ; then the second attention score A f2 is calculated according to the following formula (2)

[0053]

[0054] In formula (2), W V2 , W Q2 , W K2 , and G2 are weight matrices; and D represents the vector dimension.

[0055] S13, the local feature f3 is processed by a multi-layer perception (MLP) to obtain a third attention score A f3 , which is specifically shown in the following formula (3)

[0056] A f3 = MLP (f3) (3)

[0057] S14, the first attention score A f1 , the second attention score A f2 , and the third attention score A f3 are respectively normalized by a softmax function to obtain attention weights w1, w2, and w3, which are specifically shown in the following formula (4)

[0058]

[0059] In formula (4), ∑ i w i = 1, i = 1, 2, 3.

[0060] The weighted feature fusion module performs weighted feature fusion on the global feature f1, the global feature f2 and the local feature f3 based on three attention weights, and the specific formula (5) is as follows:

[0061]

[0062] In formula (5), F fused represents the weighted fusion feature.

[0063] Performance analysis:

[0064] The recognition results of the face image recognition using the single volumetric neural network model VGG16, ResNet18 and VGG16-ResNet18 feature fusion and the convolutional neural network (Ours) of the present application under the same data set (CASIA-FACE data set) and the same experimental environment are shown in Table 1 below. The recognition rate of the present application Ours is as high as 98.6%, which is 5.4% higher than the recognition rate of VGG16, 3.2% higher than the recognition rate of ResNet18, and 5.2% higher than the recognition rate of VGG16-ResNet18 feature fusion. The cross-attention mechanism of the present application guides the convolutional feature fusion and dynamically adjusts the contribution of the local and global features, thereby enhancing the expression ability of the features and improving the performance of the image classification, which greatly improves the recognition performance of the face.

[0065] Table 1

[0066] Network model VGG16 ResNet18 VGG16-ResNet18 Ours Recognition rate (%) 93.2 95.4 93.4 98.6

[0067] Although the embodiments of the present application have been shown and described, it can be understood by those of ordinary skill in the art that various changes, modifications, replacements and variations can be made to the embodiments without departing from the principles and spirits of the present application, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A face recognition method based on cross-attention feature fusion convolutional neural network, characterized by: The specific steps include: (1) Collect multiple face images, then perform image enhancement and image preprocessing to construct a face image dataset; (2) Construct a cross-attention feature fusion convolutional neural network. The cross-attention feature fusion convolutional neural network includes a global feature extraction module, a local feature extraction module, a cross-attention mechanism, a weighted feature fusion module and a Softmax classifier. The global feature extraction module uses the convolutional neural network VGG16 model and the ResNet model to extract the global features of the face image and obtain the global features f1 and f2; The local feature extraction module uses local binary pattern LBP to extract local texture features in the face image as local features f3; The cross attention mechanism calculates three attention weights based on the global feature f1, global feature f2, and local feature f3; The weighted feature fusion module performs weighted feature fusion on the global feature f1, global feature f2, and local feature f3 based on the three attention weights to obtain the weighted fusion feature; The Softmax classifier extracts facial feature vectors based on weighted fusion features, performs feature classification, and outputs face recognition results; (3) Use the loss function to train the cross-attention feature fusion convolutional neural network, input the face image to be recognized into the trained cross-attention feature fusion convolutional neural network for detection and recognition, and obtain the face recognition result.

2. The face recognition method based on cross-attention feature fusion convolutional neural network according to claim 1 is characterized in that: The image enhancement is to transform the collected multiple face images by random rotation to obtain multiple face images; the image preprocessing is to clean the face images, balance the category distribution, and then perform size standardization to unify the sizes of all collected face images.

3. The face recognition method based on cross-attention feature fusion convolutional neural network according to claim 1 is characterized in that: The cross attention mechanism calculates three attention weights based on the global feature f1, the global feature f2, and the local feature f3, and specifically includes the following steps: S11, after linear mapping of the global feature f1, the value vector V is obtained f1 ; After linear mapping of the global feature f2, we get the query vector Q f2 ; After linear mapping of the local feature f3, the key vector K is obtained f3 ; Then calculate the first attention score A according to the following formula (1): f1 ; In formula (1), W V1 、W Q1 、W K1 , G1 are weight matrices; D represents the vector dimension; S12, after linear mapping of the global feature f1, obtain the query vector Q f1 ; After linear mapping of the global feature f2, we get the value vector V f2 ; After linear mapping of the local feature f3, the key vector K′ is obtained f3 ; Then calculate the second attention score A according to the following formula (2): f2 ; In formula (2), Q V2 、W Q2 、W K2 , G2 are weight matrices; D represents the vector dimension; S13, after the local feature f3 is processed by the multi-layer perceptron MLP, the third attention score A is obtained f3 , see the following formula (3) for details: A f3 =MLP(f3) (3); S14, the first attention score A is calculated by the softmax function f1 , second attention score A f2 and the third attention score A f3 Normalize them separately to get the attention weights w1, w2 and w3, as shown in the following formula (4): In formula (4), ∑ i w i =1, i=1,2,3.

4. The face recognition method based on cross-attention feature fusion convolutional neural network according to claim 3 is characterized in that: The weighted feature fusion module performs weighted feature fusion on the global feature f1, the global feature f2, and the local feature f3 based on the three attention weights, as shown in the following formula (5): In formula (5), F fused Represents weighted fusion features.

5. The face recognition method based on cross-attention feature fusion convolutional neural network according to claim 4 is characterized in that: The Softmax classifier includes two fully connected layers and a softmax classification function. The two fully connected layers are used to perform dimensionality reduction processing on the weighted fusion features, and the softmax classification function is used to integrate and classify the input features.

6. The face recognition method based on cross-attention feature fusion convolutional neural network according to claim 1 is characterized in that: The cross-attention feature fusion convolutional neural network is trained using a cross-entropy loss function to obtain a trained cross-attention feature fusion convolutional neural network.

Images