A face recognition method

Abstract

The present invention discloses a face retrieval method. The method first performs data preprocessing, performs data augmentation and image enhancement on training data respectively; establishes a feature extraction convolutional neural network; adds an angular distance soft interval module on the basis of the softmax loss function to perform category normalization output to train the convolutional neural network; calculates the feature vector of the face through the trained convolutional neural network, and uses the feature vector as the representation of the face; finally, proposes a secondary retrieval method of coarse retrieval plus refined retrieval to complete the face recognition process from the detection set to the bottom library set. The method is based on the face feature calculation structure, introduces artificial understanding of data, and makes timely adjustments through secondary iterations during the retrieval process, thereby improving the generalization and accuracy of retrieval in different scenarios.

Description

Technical Field

[0001] The invention discloses a face retrieval method and relates to the technical field of face recognition. Background Art

[0002] Face recognition is a technology that can identify or authenticate a person from a set of images or videos. Face recognition was first proposed in the 1970s. After nearly half a century of development, face recognition technology has made substantial improvements in recognition accuracy. As one of the most challenging biometric technologies, face recognition is more susceptible to being affected in constrained environments, mainly due to the large number of angle changes, age changes, occlusions, blurs, and changes in facial expressions that appear in real scenes, all of which bring challenges to face recognition. Figure 1 shown.

[0003] Traditional face recognition algorithms are based on hand-designed feature extractors, such as image edge and texture feature extraction, which are combined with machine learning algorithms, such as principal component analysis (PCA), linear discriminant analysis (LDA), or support vector machines (SVM). The limitation of traditional algorithms is that they are not robust enough for unconstrained scenarios, so researchers have to design different algorithms for different scenarios, such as algorithms for age changes, algorithms for posture changes, algorithms for blur, etc. In recent years, traditional face recognition algorithms have been completely surpassed by deep learning methods represented by convolutional neural networks (CNN). The advantage of deep learning methods is that they can train models in massive data to learn the best expression of features. The large number of available face images on the Internet facilitates the collection of massive data, which greatly supports the development of face recognition technology.

[0004] Face recognition systems usually include the following modules: Figure 2 As shown:

[0005] 1) Face detection: Face detection aims to find the position of a face in an image and return the coordinate position of the face frame where the face is located. Figure 3 shown.

[0006] 2) Face alignment. Face alignment first locates the key points of the face, calculates the correspondence between these key points and the key points of the template, and then performs an affine transformation to fit the key points of the face to the corresponding positions of the template. In this way, all the key points of the face will remain consistent under the same template, which is a very useful step for recognition.

[0007] 3) Face representation,In the face representation stage, the image composed of pixels is converted into a dense and discriminative feature vector, also known as a template. Ideally, all faces belonging to the same category will be mapped to similar feature vectors.

[0008] 4) Face matching: In the face matching stage, a similarity score is calculated between the two templates. The similarity score can show the similarity between the two templates, and then determine whether the two belong to the same person.

[0009] However, the following defects generally exist in the current face recognition methods:

[0010] 1. Insufficient generalization ability. Although face recognition technology has achieved recognition capabilities similar to those of humans in many indicators, it still has insufficient generalization ability in industrial applications. In other words, when the model trained with dataset A is applied to dataset B, the recognition accuracy will be greatly reduced when dataset A and dataset B are from different sources. This is a common challenge for face recognition in practical applications, such as ID photos, surveillance photos, network photos and other pictures of different scenes, as well as the challenges brought by unconstrained scenes such as posture changes, lighting changes, blur and occlusion mentioned above. Although there are many studies on domain transfer, domain adaptation and transfer learning, the effect of these improvements in actual engineering applications is very limited.

[0011] 2. Artificial intelligence failed to improve the model's effect on face retrieval. The current face learning method completely relies on the guidance of data labels to fit the model, but in the process of iterative model improvement, artificial understanding of data is not introduced. If the recognition results are manually verified and feedback is given in a timely manner during the model verification process, this will correct the errors that the model cannot learn in a timely manner and improve the model's expressiveness. In the actual deployment process, once the model is trained, it is used directly, and the subsequent verification and recognition work is completely dependent on the model, which is bound to bring instability to the system and is not conducive to the subsequent iteration and upgrade of the model. Summary of the invention

[0012] The technical problem to be solved by the present invention is: in view of the defects of the prior art, a face retrieval method is provided, and a new face retrieval framework is designed. In the process of face model verification, the framework adds a manual verification entry point, introduces manual understanding of the data and timely feeds back to the model and data, forms a closed loop through the process of data-model-verification-manual confirmation-data, and continuously iterates and upgrades the verification accuracy of the system.

[0013] The present invention adopts the following technical solutions to solve the above technical problems:

[0014] A face retrieval method, the method is based on a computing structure, the computing structure includes a feature extraction module, an angular distance interval classification loss module; the method includes:

[0015] Step 1: Perform data preprocessing, perform data augmentation and image enhancement on training samples respectively;

[0016] Step 2: Establish a feature extraction convolutional neural network;

[0017] Step 3: Based on the softmax loss function, the angular distance soft interval is added to perform normalized output of the category;

[0018] Step 4: Incorporate the understanding of the data into model verification. Set two different source data sets, S and T. The purpose of face retrieval is to use S as the base database and T as the detection set, and use the detection set to search in the given base database to accurately find faces with the same identity information.

[0019] As a further preferred embodiment of the present invention, the step 4 comprises:

[0020] 401. Data collection: obtaining a set number of detection set photos and base photos in advance. The base photos contain identity information of faces in the detection set photos. Perform face cropping and face alignment on all pictures.

[0021] 402. Use the computing structure to perform feature extraction and normalization processing;

[0022] 403. Calculate similarity between the feature vector obtained from each detection set photo and the feature corresponding to the base photo, and select n base photos with the highest similarity as the search results;

[0023] 404. According to each detection set photo and the n search results, confirm whether the identity information of the detection set photo is in the n pictures, filter out all the hit base database pictures, and filter out the missed pictures;

[0024] 405. Based on the processing of steps 403 and 404, each detection set photo is replaced with the S set photo with the highest search similarity and hit as a new detection set photo, and this new detection set photo is used to perform a fine search in the base database again, and the n base database photos with the highest similarity are obtained again as new search results.

[0025] As a further preferred embodiment of the present invention, the data augmentation and image enhancement in step 1 are respectively:

[0026] Data augmentation: Data augmentation is performed on training samples of faces with large-angle poses, blur and masks. The cropping method is to select 1.4 times the longest side of the face frame and expand it to a square face image; the mask generation network is used to fit the mask to the mouth area; the blurring methods include Gaussian blur, motion blur or using the GAN network for image generation;

[0027] Image enhancement: The training data is randomly enhanced, and the enhancement method includes horizontal flipping or blurring.

[0028] As a further preferred embodiment of the present invention, the establishment of a feature extraction network specifically includes:

[0029] In the feature extraction module, four blocks with specific structures are connected in series, processing feature maps with 64, 128, 256 and 512 channels respectively, and outputting 128, 256, 512 and 1024 channels respectively to prepare for the next block;

[0030] Each block contains several bottlenecks with the same structure. Except for the last bottleneck, the results of other bottlenecks keep the same number of channels. The last bottleneck increases the number of channels.

[0031] The result of the last bottleneck is called a block residual structure.

[0032] The network structure of the feature extraction network includes a ResNet network, an Inception network or a NasNet network.

[0033] As a further preferred embodiment of the present invention, the step three specifically includes: adding an angular distance soft interval module on the basis of the softmax loss function, wherein the softmax loss function is expressed by the formula:

[0034]

[0035] Among them, x i represents the i-th sample, yi represents the i-th class label, represents the transpose of the weight of yi, represents the bias number of yi, e is the exponential function, N is the number of samples, and log is the logarithmic function;

[0036] Make j =0, the logarithmic term is transformed into:

[0037]

[0038] Among them, ||W j || is the parameter W j The modulus of ||x i || is the parameter x i The modulus, cosθ j is the vector W j and x i The cosine of the angle;

[0039] In order to constrain the modulus length of the classifier and feature vector, we use

[0040] ||W j ||=1,||x i || = s;

[0041] In this way, L2 normalization is performed on the features and weights, and the feature vectors are constrained to a sphere with a radius of s in the high-order space:

[0042]

[0043] Among them, s is a constant term, For sample x i The cosine value of the angle with yi; e is the exponential function, N is the number of samples, and log is the logarithmic function;

[0044] Since the modulus length is constrained, the above loss function controls the size by the angle between the weight and the feature, which is an angular distance metric function;

[0045] On this basis, the interval m is added to improve the compactness within the class and the distinction between classes as an angular distance penalty, thus obtaining the Arcface loss function:

[0046]

[0047] Among them, s and m are constant terms, For sample x i The cosine of the angle with yi, e is the exponential function, and N is the number of samples.

[0048] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0049] 1) A face verification module was added to the system. The entire process first forms preliminary search results through coarse search, and then the screening results are manually confirmed to form a new detection set, and then fine search is performed. It is proved theoretically and experimentally that the results of fine search are better than coarse search. Previous systems did not introduce manual judgment in the verification stage, so there was only coarse search, and they were susceptible to systematic misidentification due to different distributions of different source data sets (such as ID photos and surveillance photos). Our system solves this problem, greatly enhancing the generalization ability of the model and improving the accuracy of face retrieval.

[0050] 2) In response to the needs of current practical application scenarios, data enhancement for large-angle, blurred, and masked face images has been enhanced to improve the robustness of the model.

[0051] 3) The face recognition model is trained on different data sets such as surveillance photos, ID photos, and online photos. Self-supervision methods are used for data labeling and data cleaning, thus forming a data set of millions of IDs with better fitting accuracy than previous models. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Schematic diagram of unconstrained face differences. (a), (b), (c), (d), and (e) show the changes in angle, age, blur, expression, and occlusion, respectively.

[0053] Figure 2 It is a schematic diagram of the processing flow of a face recognition system in the prior art.

[0054] Figure 3 It is a schematic diagram of face frame detection and face alignment in the prior art.

[0055] Figure 4 This is a schematic diagram of the FH-FACE-RECOGNITION-NET network structure in the present invention.

[0056] Figure 5 This is a schematic diagram of the residual network structure in the present invention.

[0057] Figure 6 This is a diagram showing the distribution effects of Softmax and Arcface on the MNIST dataset.

[0058] Figure 7 It is a flowchart of the steps of the verification module in the present invention.

[0059] Figure 8 It is a schematic diagram of the rough search test results in a specific embodiment of the present invention.

[0060] Fig. 9 It is a schematic diagram of the results of a precise search test in a specific embodiment of the present invention.

[0061] Fig.10 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION

[0062] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0063] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0064] The schematic diagram of the method of the present invention is as follows Fig.10 As shown, the method is based on a computing structure, which includes a feature extraction module and an angular distance interval classification loss module; the method includes:

[0065] Step 1: Perform data preprocessing, perform data augmentation and image enhancement on training samples respectively;

[0066] Step 2: Establish a feature extraction convolutional neural network;

[0067] Step 3: Based on the softmax loss function, the angular distance soft interval is added to perform normalized output of the category;

[0068] Step 4: Incorporate the understanding of the data into model verification. Set two different source data sets, S and T. The purpose of face verification is to use S as the base database and T as the detection set, and use the detection set to search in the given base database to accurately find faces with the same identity information.

[0069] In a specific embodiment of the present invention, the computing architecture is named as FH-FACE-RECOGNITION-NET network structure. Figure 4 As shown, it is mainly divided into the following parts: 1. Feature extraction module; 2. Angular distance interval classification loss module.

[0070] 1. Feature extraction module

[0071] Different from the general CNN network, the network of this application adopts the ResnetFace structure specially designed for human face. In each bottleneck of CNN, the front and back layers use the same number of channels to reduce the information loss caused by the change of the number of channels. On the other hand, in order to better maintain the cohesion between the same type of facial features and the distinction between different types, the technical solution adds the angular distance soft interval to the feature space, and calculates the cross entropy classification loss function on this basis, so as to fit the model through back propagation.

[0072] 101. Data preprocessing,In the data preprocessing part, data augmentation and image enhancement are performed on the training samples respectively.

[0073] Data augmentation: In order to meet the needs of current application scenarios, in addition to the training samples of the public data set, data augmentation is performed on the training samples of large-angle posture, blur and masked face images. The cropping method is to select 1.4 times the longest side of the face frame and expand it to a square face image. The mask is fitted to the mouth area using a mask generation network. In addition to using linear blur strategies such as Gaussian blur and motion blur, the blur method also includes using the GAN network for image generation, which makes the training set have richer scenes and distribution.

[0074] Image enhancement: The training samples of face poses and angles are randomly cropped to 112*112 and then input into the network. In order to improve the robustness of the model in different scenarios, the training data is randomly enhanced, such as horizontal flipping, blurring and other image enhancement methods.

[0075] 102. Establish feature extraction network

[0076] Improvements are made based on ResNet50 to better meet the needs of face recognition scenarios, collectively referred to as ResnetFace. Before ResnetFace outputs the residual structure, it needs to pass through the ReLU layer for feature filtering, retaining only useful feature information and filtering out information in negative areas.

[0077] Learn the features of face representation and reduce the complexity of the model gradient map, using Figure 5 The Pre-ReLU residual structure shown in the figure is used for feature extraction. First, in the feature extraction module, four blocks are connected in series to process feature maps with 64, 128, 256 and 512 channels respectively, and output 128, 256, 512 and 1024 channels respectively to prepare for the next block. Each block contains several bottlenecks with the same structure. Except for the last bottleneck, the results of other bottlenecks keep the same number of channels. The last bottleneck performs dimensionality increase on the number of channels. We call the result of the last bottleneck a block residual structure. For the specific structure, see Figure 5 .

[0078] In the feature extraction network part, in addition to using the ResNet basic network, other network structures such as Inception or NasNet can also be used in other specific implementations.

[0079] 103. Angular distance soft interval module

[0080] Traditional image classification tasks generally use softmax to normalize the output of the category and use cross entropy to calculate the loss function. However, in face recognition training, the number of IDs is often in the millions, which means that softmax is used to classify millions of categories, which is not very effective in face recognition. In order to obtain better classification and recognition results, so that the faces within the class are more cohesive and the faces between classes are more distinguishable, this application adds an angular distance soft interval module based on softmax.

[0081] Softmax representation formula:

[0082]

[0083] x i represents the i-th sample, yi represents the i-th class label, represents the transpose of the weight of yi, Represents the bias number of yi, e is the exponential function, N is the number of samples, and log is the logarithmic function.

[0084] For simplicity, let b j = 0, and then transform the logarithmic term into

[0085]

[0086] Among them, ||W j || is the parameter W j The modulus of ||x i || is the parameter x i The modulus, cosθ j is the vector W j and x i The cosine of the angle .

[0087] In order to constrain the modulus length of the classifier and feature vector, we use

[0088] ||W j ||=1,||x i || = s;

[0089] In this way, L2 normalization is performed on the features and weights. The feature vector is constrained to a sphere with a radius of s in the high-order space:

[0090]

[0091] Among them, s is a constant term, For sample x i The cosine of the angle with yi. e is the exponential function, N is the number of samples, and log is the logarithmic function.

[0092] Since the modulus is constrained, the above loss function controls the size by the angle between the weight and the feature, so it is an angular distance metric function. On this basis, the interval m is added to improve the compactness within the class and the distinction between classes as an angular distance penalty, thus obtaining the Arcface loss function formula:

[0093]

[0094] Among them, s is a constant term, and this patent sets s=64, and m is a constant term, and this patent sets m=0.35. For sample x i The cosine value of the angle with yi. e is an exponential function, and N is the number of samples.

[0095] The experimental results intuitively show that using Arcface is much better than directly using softmax classification. Figure 6 shown.

[0096] 104. Manual Assisted Verification Module

[0097] In order to further improve the model's ability to express itself on different source data sets, this application considers incorporating manual understanding of data into model verification. First, two different source data sets are set, S and T. The purpose of face verification is to use S as the base database and T as the detection set, and search for a given detection set from the base database. The manual assisted verification module is divided into the following steps, refer to the following schematic diagram Figure 7 shown.

[0098] 4.1 Data Collection

[0099] For simplicity, closed-set face recognition is used as an explanation and demonstration. Several detection set photos and some base photos are obtained in advance, and it is ensured that the base photos contain the IDs of the faces in the detection set. Without loss of generality, in the schematic diagram, ID photos are used as the detection set and surveillance photos are used as the base. Since the two are from different sources, they meet the aforementioned prior conditions, and all images have been processed with face cropping and face alignment.

[0100] 4.2 Feature Extraction

[0101] In the feature extraction stage, we use the self-developed face model FH-FACE-RECOGNITION-NET-V3.1, which has the same structure as Figure 4 The backbone network remains consistent, the feature dimension extracted is 256, and L2 normalization is maintained.

[0102] 4.3 Rough Image Retrieval

[0103] Based on the feature vector obtained from each detection set photo, the similarity is calculated with the features corresponding to the base photo, and the top n base photos with the highest similarity are selected as the search results. Here n=5, that is, the top-5 search results are retained as the rough search.

[0104] 4.4 Manual screening

[0105] This step simulates the entry point of human-machine collaboration from the perspective of engineering application. According to each detection set photo and n search results, it is manually confirmed whether the detection set photo ID is in these n pictures (top-n search results hit more than one), and all top-n hit detection set photo IDs are screened out, and top-n missed IDs are filtered out (top-n search results all hit).

[0106] The diagram of search results and manual screening is shown below. Figure 8 As shown:

[0107] 4.5 Image Retrieval

[0108] On the basis of rough search and manual screening, each detection set photo id is replaced with the S set photo with the highest similarity and hit as the new detection set photo, and this new detection set photo is used to do a fine search in the base database again, and the n base database photos with the highest similarity are obtained as the new search results, such as Figure 8 As shown, we Figure 8 The third picture from the left in the figure is used as the new detection set photo, and a new refined search is performed. The top-n results and similarities of the refined search are shown in Fig. 9 As shown. Fig. 9 It can be seen that not only the hit rate of fine search is higher than that of rough search, but also the similarity is greatly improved.

[0109] In other implementation scenarios of the technical solution disclosed in this application, in addition to classifying and calculating the image features as a whole, the feature maps can also be aggregated, and learnable weight factors can be applied to different parts of the face. Using the attention method, the model can learn to focus on more important feature parts in different faces.

[0110] This application is specially optimized based on a large amount of face data, including ID photos, surveillance photos, and online photos. Tests show that FH-FACE-RECOGNITION-NET has faster inference speed, less computing resources, better model performance, and more accurate inference results than the original model. Change the original residual network structure and extract the output features of the network in 4 layer groups. Improve the face recognition effect. Add an angular distance soft interval module, and adjust the relevant parameters of the soft interval module according to the experimental results and understanding of the business to achieve the best model evaluation performance. Use different parameter optimization methods and learning rates for the network, and the model converges better during the training process.

[0111] In the process of system verification, a manual verification module is added. By introducing manual understanding of the data, the model can select the best photos from the base library as new detection set photos in the rough search, and perform a secondary fine search in the base library. This not only improves the credibility of the model in face retrieval, but also increases the amount of data in the standing detection library, and improves the system's expressive ability in both quality and quantity. The verification effect of this module on data sets from different sources is more obvious, which solves the problem of insufficient generalization ability of the model on data sets from different sources.

[0112] In addition to the public training set, a semi-supervised learning method is used to perform proxy annotation on large-angle, blurred and masked samples, and the data is cleaned through face denoising and deduplication algorithms to enhance the robustness of the model in complex scenarios.

[0113] The above is a detailed description of the implementation mode of the present invention in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation mode. Various changes can be made within the knowledge scope of ordinary technicians in the field without departing from the purpose of the present invention. The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical content disclosed above to an equivalent embodiment of equivalent changes without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principle of the present invention, still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A face retrieval method, the method is based on a computing structure, characterized in that: The computing structure includes a feature extraction module and an angular distance interval classification loss module; the method includes: Step 1: perform data preprocessing, perform data augmentation and image enhancement on the training samples, the data augmentation and image enhancement are respectively: Data augmentation: Data augmentation is performed on training samples of faces with large-angle poses, blur and masks. The cropping method is to select 1.4 times the longest side of the face frame and expand it to a square face image; the mask generation network is used to fit the mask to the mouth area; the blurring methods include Gaussian blur, motion blur or using the GAN network for image generation; Image enhancement: randomly enhance the training data, and the enhancement method includes horizontal flipping or blurring; Step 2: Establish a feature extraction convolutional neural network, including: In the feature extraction module, four blocks with specific structures are connected in series, processing feature maps with 64, 128, 256 and 512 channels respectively, and outputting 128, 256, 512 and 1024 channels respectively to prepare for the next block; Each block contains several bottlenecks with the same structure. Except for the last bottleneck, the results of other bottlenecks keep the same number of channels. The last bottleneck increases the number of channels. The result of the last bottleneck is called a block residual structure; Step 3: Based on the softmax loss function, the angular distance soft interval is added to perform normalized output of the category; Step 4: Incorporate the understanding of data into model verification. Set two different source data sets, S and T. The purpose of face retrieval is to use S as the base database and T as the detection set to search in the given base database and accurately find the face with the same identity information. The specific steps include:

401. Data collection: obtaining a set number of detection set photos and base photos in advance. The base photos contain identity information of faces in the detection set photos. Perform face cropping and face alignment on all pictures.

402. Use the computing structure to perform feature extraction and normalization processing; 403. Calculate similarity between the feature vector obtained from each detection set photo and the feature corresponding to the base photo, and select n base photos with the highest similarity as the search results; 404. According to each detection set photo and the n search results, confirm whether the identity information of the detection set photo is in the n pictures, filter out all the hit base database pictures, and filter out the missed pictures; 405. Based on the processing of steps 403 and 404, each detection set photo is replaced with the S set photo with the highest search similarity and hit as a new detection set photo, and this new detection set photo is used to perform a fine search in the base database again, and the n base database photos with the highest similarity are obtained again as new search results.

2. A face retrieval method as claimed in claim 1, characterized in that: The network structure of the feature extraction network includes a ResNet network, an Inception network or a NasNet network.

3. A face retrieval method as claimed in claim 1, characterized in that: The step three specifically includes: adding an angular distance soft interval module on the basis of the softmax loss function, wherein the softmax loss function is expressed by the formula: Among them, x i represents the i-th sample, yi represents the i-th class label, represents the transpose of the weight of yi, represents the bias number of yi, e is the exponential function, N is the number of samples, and log is the logarithmic function; Make j =0, the logarithmic term is transformed into: Among them, ||W j || is the parameter W j The modulus of ||x i || is the parameter x i The modulus, cosθ j is the vector W j and x i The cosine of the angle; In order to constrain the modulus length of the classifier and feature vector, we use ||W j ||=1,||x i ||=s; In this way, L2 normalization is performed on the features and weights, and the feature vectors are constrained to a sphere with a radius of s in the high-order space: Among them, s is a constant term, For sample x i The cosine value of the angle with yi, e, is an exponential function, N is the number of samples, and log is a logarithmic function; Since the modulus length is constrained, the above loss function controls the size by the angle between the weight and the feature, which is an angular distance metric function; On this basis, the interval m is added to improve the compactness within the class and the distinction between classes as an angular distance penalty, thus obtaining the Arcface loss function: Among them, s and m are constant terms, For sample x i The cosine value e of the angle with yi is an exponential function, and N is the number of samples.

Images