A face recognition method and device based on multi-task learning
By first training and freezing the face recognition branch in multi-task learning, and then training on the face attribute analysis data set, the problem of unbalanced performance of face recognition and face attribute analysis tasks is solved, and the multi-task face is improved Accuracy of identification.
Patent Information
- Application Number
- CN202111427335.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-25
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-11-25
AI Technical Summary
When the data is unbalanced, the performance of face recognition and face attribute analysis tasks is unbalanced, resulting in better performance of some tasks while poor performance of others, and it is difficult to achieve balance in the loss function design. Influence the learning process of network sharing layer.
By first using the face recognition branch to train and freeze the network layer of the basic network and face recognition branch, and then training for the face attribute analysis data set, avoiding the face attribute analysis affecting the weight of the face recognition branch, and during the training process Adjust the branch weight of face attributes to ensure optimal performance of both.
The performance balance between face recognition and face attribute analysis is achieved, and the problem of poor task performance caused by data imbalance and loss function adjustment is avoided, and the accuracy of face recognition for multi-tasks is improved.
Smart Images

Figure CN114005166B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision, and in particular to a face recognition method and device based on multi-task learning. Background Art
[0002] At present, the existing multi-task learning methods mainly learn multiple related tasks together and learn multiple tasks at the same time, such as multi-task convolutional neural networks and multi-task learning of face detection and face key points. However, when the data of each task is unbalanced, these existing multi-task learning methods will tend to favor tasks with more data during learning, and the performance of the learned model will be better than that of the model with less data; at the same time, the convergence speed of each model during the training process is also different, and the loss function design of each task is difficult to achieve a balance, resulting in better performance of some tasks and poor performance of some tasks. For example: face recognition and face attribute analysis belong to different task categories, and each has a corresponding loss function. During the training process, the existing technology simply adds the losses of the two, resulting in a large difference in the scale of task losses. The overall loss of the technical solution is dominated by one of the tasks, and the other task cannot affect the learning process of the network sharing layer and its performance will be affected. Summary of the invention
[0003] The present invention provides a face recognition method and device based on multi-task learning to solve the technical problem of performance imbalance between face recognition tasks and face attribute analysis tasks.
[0004] In order to solve the above technical problems, an embodiment of the present invention provides a face recognition method based on multi-task learning, comprising:
[0005] Obtaining a first training data set, a second training data set, and a face data set to be recognized, sending the first training data set to a preset first network, training the first training data set, and obtaining a face recognition model;
[0006] Initialize the network weight parameters corresponding to the preset second network through the weight corresponding to the face recognition branch, and freeze the network layer corresponding to the face recognition model;
[0007] Sending the second training data set to a second network with initialized network weight parameters, training the second training data set to obtain a face attribute analysis model;
[0008] Extracting facial features from the face dataset to be identified using the facial attribute analysis model to obtain a feature image;
[0009] Perform facial feature matching on the feature image to obtain a face recognition result.
[0010] Furthermore, the step of sending the first training data set to a preset first network, training the first training data set, and obtaining a face recognition model is specifically as follows:
[0011] The face recognition model is obtained by performing network weight training on the first training data set through a basic network and a face recognition branch until the loss function corresponding to the face recognition branch converges; the basic network is a shared network part for the face recognition task and the face attribute analysis task.
[0012] Furthermore, before initializing the network weight parameters corresponding to the preset second network by the weight corresponding to the face recognition branch, the method further includes:
[0013] A face attribute branch is added after the basic network to form a second network including corresponding weights of the face attribute branch.
[0014] The loss function corresponding to the face attribute branch is:
[0015] ;
[0016] in,
[0017] ;
[0018] N represents the total number of images for training, M represents the number of categories of face attributes, and y ij Logits represents the true label of the face attribute ij Represents the predicted value of the face attribute output, represents the sigmoid activation function, r j Indicates the ratio of the number of images with the jth attribute in the second training dataset to the total training images.
[0019] Furthermore, the second training data set is fed into a second network whose network weight parameters are initialized, and the second training data set is trained to obtain a face attribute analysis model, specifically:
[0020] The second training data set for facial attribute analysis is sent to a preset second network. The network weight parameters of the second network are initialized, and the network weight training is performed on the second training data set until the loss function corresponding to the facial attribute branch converges to obtain the facial attribute analysis model.
[0021] Furthermore, the facial feature matching is performed on the feature image to obtain a face recognition result, specifically: COS similarity is calculated on the feature image, if the COS similarity is greater than a first preset value, the face recognition is successful; if the COS similarity is less than or equal to the first preset value, the face recognition fails.
[0022] Accordingly, an embodiment of the present invention provides a face recognition device based on multi-task learning, comprising a first training module, a freezing module, a second training module, a feature extraction module and a feature matching module; wherein:
[0023] The first training module is used to obtain a first training data set, a second training data set and a face data set to be recognized, send the first training data set to a preset first network, train the first training data set, and obtain a face recognition model;
[0024] The freezing module is used to initialize the network weight parameters corresponding to the preset second network through the weight corresponding to the face recognition branch, and freeze the network layer corresponding to the face recognition model;
[0025] The second training module is used to send the second training data set to the second network with initialized network weight parameters, train the second training data set, and obtain a face attribute analysis model;
[0026] The feature extraction module is used to extract facial features from the face dataset to be identified through the facial attribute analysis model to obtain a feature image;
[0027] The feature matching module is used to perform facial feature matching on the feature image to obtain a face recognition result.
[0028] Furthermore, the first training module sends the first training data set to a preset first network, trains the first training data set, and obtains a face recognition model, specifically:
[0029] The first training module performs network weight training on the first training data set through a basic network and a face recognition branch until the loss function corresponding to the face recognition branch converges, thereby obtaining the face recognition model; the basic network is a shared network part of the face recognition task and the face attribute analysis task.
[0030] Furthermore, the face recognition device also includes a branch adding module, which is used to add a face attribute branch behind the basic network before the freezing module initializes the network weight parameters corresponding to the preset second network through the weights corresponding to the face recognition branch, so as to form a second network including the corresponding weights of the face attribute branch.
[0031] The loss function corresponding to the face attribute branch is:
[0032] ;
[0033] in,
[0034] ;
[0035] N represents the total number of images for training, M represents the number of categories of face attributes, and y ij Logits represents the true label of the face attribute ij Represents the predicted value of the face attribute output, represents the sigmoid activation function, r j Indicates the ratio of the number of images with the jth attribute in the second training dataset to the total training images.
[0036] Furthermore, the second training module sends the second training data set to the second network whose network weight parameters are initialized, trains the second training data set, and obtains a face attribute analysis model, specifically:
[0037] The second training module sends a second training data set for facial attribute analysis into a preset second network, and performs network weight training on the second training data set through the second network initialized with network weight parameters until the loss function corresponding to the facial attribute branch converges, thereby obtaining the facial attribute analysis model.
[0038] Furthermore, the feature matching module performs facial feature matching on the feature image to obtain a face recognition result, specifically:
[0039] The feature matching module calculates the cos similarity of the feature image. If the cos similarity is greater than a first preset value, face recognition is successful; if the cos similarity is less than or equal to the first preset value, face recognition fails.
[0040] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0041] The present invention provides a face recognition method and device based on multi-task learning, the method comprising: sending a first training data set into a first network for training to obtain a face recognition model; initializing a network weight parameter corresponding to a preset second network with a weight corresponding to a face recognition branch, and freezing the network layer corresponding to the face recognition model; sending a second training data set into a second network for training to obtain a face attribute analysis model, and using the second face attribute analysis model for feature extraction and feature matching to obtain a face recognition result. Compared with the prior art, the present invention first uses a face recognition branch for training, freezes the network layers of a basic network and a face recognition branch, and then trains the data set for face attribute analysis. The basic network layer and the face recognition branch network layer will not be gradient updated, ensuring that the training of face attribute analysis will not affect the weight of the trained face recognition branch. At the same time, the weight of the face attribute branch can be adjusted during the network training process, so that the performance of face attribute analysis and face recognition is optimized, avoiding the problem of partial task quantization and poor performance of partial tasks caused by data imbalance and loss function adjustment, and improving the accuracy of multi-task face recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 : A flow chart of an embodiment of a face recognition method based on multi-task learning provided by the present invention.
[0043] Figure 2 :An embodiment of the face recognition method based on multi-task learning provided by the present invention is an example of a ResNet shallow network.
[0044] Figure 3 :An embodiment of the face recognition method based on multi-task learning provided by the present invention is an example of a ResNet deep network.
[0045] Figure 4 : A schematic diagram of the overall structure of a preset network of an embodiment of the face recognition method based on multi-task learning provided by the present invention.
[0046] Figure 5 : A structural diagram of an embodiment of a face recognition device based on multi-task learning provided by the present invention. DETAILED DESCRIPTION
[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0048] Embodiment one:
[0049] Please refer to Figure 1 , a face recognition method based on multi-task learning provided by an embodiment of the present invention, comprising steps S1 to S5; wherein,
[0050] Step S1, obtaining a first training data set, a second training data set and a face data set to be recognized, sending the first training data set to a preset first network, training the first training data set to obtain a face recognition model.
[0051] In this embodiment, a face is detected from a captured image, and the face detection process includes five key point detections; a mapping matrix is calculated based on the preset five face key point standard coordinates and the five key point coordinates detected, and the detected face is corrected and transformed using the mapping matrix to obtain a face alignment image. The five face key points are the two eyes, nose, and mouth of a person (the mouth contains two key points).
[0052] The face alignment image resolution is 112×112, and the standard coordinates of the five preset face key points used are:
[0053] [[30.2946+8.000,51.6963];
[0054] [65.5318+8.000,51.6963];
[0055] [48.0252+8.000,71.7366];
[0056] [33.5493+8.000,92.3655];
[0057] [62.7299+8.000,92.3655]];
[0058] The result of face alignment is the face recognition dataset to be identified. The same ID is used to mark the pictures of the same person, and different IDs are given to different people. At the same time, face detection is performed on the face attribute data in the picture, and the face is aligned and annotated using 5 key points. Each attribute is annotated once. When the face contains a certain attribute, the attribute value is 1, and if not, it is marked as 0.
[0059] The face recognition model is obtained by performing network weight training on the first training data set through a basic network and a face recognition branch until the loss function corresponding to the face recognition branch converges; the basic network is a shared network part for the face recognition task and the face attribute analysis task.
[0060] In this embodiment, the network used is resent100. The ResNet network refers to the VGG19 network and is improved on its basis by adding residual units through a short-circuit mechanism. The changes are mainly reflected in that ResNet directly uses stride=2 convolution for downsampling and replaces the fully connected layer with a global average pool layer. An important design principle of ResNet is that when the feature map size is reduced by half, the number of feature maps is doubled, which maintains the complexity of the network layer.
[0061] ResNet uses two types of residual units, namely Figure 2 and Figure 3 shown. Figure 2 For shallow networks, Figure 3 Corresponding to deep networks. For short-circuit connections, when the input and output dimensions are consistent, the input can be directly added to the output. However, when the dimensions are inconsistent (corresponding to doubling the dimension), they cannot be added directly. There are two strategies: one is to use zero-padding to increase the dimension. At this time, a downsamp is generally required first. Pooling with strde=2 can be used, which will not increase the parameters; the second is to use a new mapping, generally using 1x1 convolution, which will increase the parameters and the amount of calculation.
[0062] In resnet100, we use Figure 3 Several network modules shown are stacked in series to form a basic network resnet100.
[0063] In this embodiment, the loss function used for face recognition uses Angular Margin Loss, specifically:
[0064] ;
[0065] Among them, L3 is the loss function for face recognition. N represents the number of images in a training batch. It represents the angle between the actual label value and the network output weight. Represents the angle between the network predicted value and the network output weight, y i It represents the real label value of the training sample, n represents the number of face categories, m is the additional edge angle, which is 0.35 by default. Its function is to enhance the compactness within the class and increase the difference between classes, so that the final network model can better distinguish faces of different categories. s is a scaling factor, which is a hyperparameter during training. This parameter scales the feature value by s, and the default value of s is 64. In the face recognition training process, when the loss function is minimized, the network converges and the training stops.
[0066] Before implementing step S2, this embodiment also includes: adding a face attribute branch after the basic network to form a second network including the corresponding weights of the face attribute branch.
[0067] Step S2, initializing the network weight parameters corresponding to the preset second network through the weight corresponding to the face recognition branch, and freezing the network layer corresponding to the face recognition model.
[0068] In this embodiment, a face attribute branch is added after the basic network for analyzing face attributes, but the weight of the face recognition branch task continues to be used to initialize the network weight parameters. Therefore, when the face recognition branch is training face attributes, the basic network layer and the face attribute recognition branch network layer (i.e., the network layer corresponding to the face recognition model) do not perform gradient updates, ensuring that face attribute training will not affect the trained face recognition branch weights. It is only necessary to adjust the weight of the face attribute branch during the network training process so that the face attribute performance can also reach the optimal level and the face attribute branch loss no longer decreases. This not only ensures that the performance of the face recognition branch is not affected, but also achieves the optimal face attribute branch. It avoids the problems of quantization of some tasks and poor performance of some tasks caused by data imbalance and loss function adjustment.
[0069] For face attribute analysis, this embodiment regards face attributes as multiple binary classification problems, so the sigmoid activation function is used, and the loss function is expressed as:
[0070] ;
[0071] in,
[0072] ;
[0073] N represents the total number of images for training, M represents the number of categories of face attributes, and y ij Indicates the true label of the face attribute, with a value of 0 or 1. ij Represents the predicted value of the face attribute output, represents the sigmoid activation function ( ), W j r j It represents the ratio of the number of images with the jth attribute in the face attribute analysis training set (the second training data set) to the total training images.
[0074] Step S3, sending the second training data set to the second network whose network weight parameters have been initialized, training the second training data set to obtain a facial attribute analysis model.
[0075] In this embodiment, the second training data set for facial attribute analysis is sent to the preset second network, and the network weight parameters are initialized through the second network, and the network weight training is performed on the second training data set until the loss function corresponding to the facial attribute branch converges, thereby obtaining the facial attribute analysis model. The final output after training is the confidence of 11 facial attributes and 512-dimensional facial features. The overall network structure is as follows: Figure 4 shown.
[0076] Step S4, extracting facial features from the face data set to be identified using the facial attribute analysis model to obtain a feature image.
[0077] Step S5, performing facial feature matching on the feature image to obtain a face recognition result.
[0078] In this embodiment, specifically, the cos similarity (value between 0 and 1) is calculated for the feature image. The higher the similarity between the two faces, the closer the cos similarity is to 1, and the lower the cos similarity is, the closer it is to 0. If the cos similarity is greater than a first preset value, face recognition is successful; if the cos similarity is less than or equal to the first preset value, face recognition fails.
[0079] Accordingly, refer to Figure 5 The embodiment of the present invention provides a face recognition device based on multi-task learning, including a first training module 101, a freezing module 102, a second training module 103, a feature extraction module 104 and a feature matching module 105; wherein,
[0080] The first training module 101 is used to obtain a first training data set, a second training data set and a face data set to be recognized, send the first training data set to a preset first network, train the first training data set, and obtain a face recognition model;
[0081] The freezing module 102 is used to initialize the network weight parameters corresponding to the preset second network through the weight corresponding to the face recognition branch, and freeze the network layer corresponding to the face recognition model;
[0082] The second training module 103 is used to send the second training data set to the second network with initialized network weight parameters, train the second training data set, and obtain a face attribute analysis model;
[0083] The feature extraction module 104 is used to extract facial features from the face dataset to be identified by using the facial attribute analysis model to obtain a feature image;
[0084] The feature matching module 105 is used to perform facial feature matching on the feature image to obtain a face recognition result.
[0085] In this embodiment, the first training module 101 sends the first training data set to a preset first network, trains the first training data set, and obtains a face recognition model, specifically:
[0086] The first training module 101 performs network weight training on the first training data set through a basic network and a face recognition branch until the loss function corresponding to the face recognition branch converges, thereby obtaining the face recognition model; the basic network is a shared network part of the face recognition task and the face attribute analysis task.
[0087] In this embodiment, the face recognition device also includes a branch adding module, which is used to add a face attribute branch behind the basic network before the freezing module initializes the network weight parameters corresponding to the preset second network through the weights corresponding to the face recognition branch, so as to form a second network including the corresponding weights of the face attribute branch.
[0088] In this embodiment, the second training module 103 sends the second training data set to the second network with initialized network weight parameters, trains the second training data set, and obtains a face attribute analysis model, specifically:
[0089] The second training module 103 sends a second training data set for facial attribute analysis into a preset second network, and performs network weight training on the second training data set through the second network initialized with network weight parameters until the loss function corresponding to the facial attribute branch converges, thereby obtaining the facial attribute analysis model.
[0090] In this embodiment, the feature matching module 105 performs facial feature matching on the feature image to obtain a face recognition result, which is specifically:
[0091] The feature matching module 105 calculates the cos similarity of the feature image. If the cos similarity is greater than a first preset value, face recognition is successful; if the cos similarity is less than or equal to the first preset value, face recognition fails.
[0092] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:
[0093] The present invention provides a face recognition method and device based on multi-task learning, the method comprising: sending a first training data set into a first network for training to obtain a face recognition model; initializing a network weight parameter corresponding to a preset second network with a weight corresponding to a face recognition branch, and freezing the network layer corresponding to the face recognition model; sending a second training data set into a second network for training to obtain a face attribute analysis model, and using the second face attribute analysis model for feature extraction and feature matching to obtain a face recognition result. Compared with the prior art, the present invention first uses a face recognition branch for training, freezes the network layers of a basic network and a face recognition branch, and then trains the data set for face attribute analysis. The basic network layer and the face recognition branch network layer will not be gradient updated, ensuring that the training of face attribute analysis will not affect the weight of the trained face recognition branch. At the same time, the weight of the face attribute branch can be adjusted during the network training process, so that the performance of face attribute analysis and face recognition is optimized, avoiding the problem of partial task quantization and poor performance of partial tasks caused by data imbalance and loss function adjustment, and improving the accuracy of multi-task face recognition.
[0094] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.
Claims
1. A face recognition method based on multi-task learning, characterized in that: include: Obtaining a first training data set, a second training data set, and a face data set to be recognized, sending the first training data set to a preset first network, training the first training data set, and obtaining a face recognition model; Initialize the network weight parameters corresponding to the preset second network through the weight corresponding to the face recognition branch, and freeze the network layer corresponding to the face recognition model; Sending the second training data set to a second network with initialized network weight parameters, training the second training data set to obtain a face attribute analysis model; Extracting facial features from the face dataset to be identified using the facial attribute analysis model to obtain a feature image; Performing facial feature matching on the feature image to obtain a face recognition result; The step of sending the first training data set to a preset first network, training the first training data set, and obtaining a face recognition model is specifically as follows: The face recognition model is obtained by performing network weight training on the first training data set through a basic network and a face recognition branch until the loss function corresponding to the face recognition branch converges; the basic network is a shared network part for the face recognition task and the face attribute analysis task; Before initializing the network weight parameters corresponding to the preset second network by the weight corresponding to the face recognition branch, the method further includes: Adding a face attribute branch after the basic network to form a second network including corresponding weights of the face attribute branch; The loss function corresponding to the face attribute branch is: ; in, ; N represents the total number of images for training, M represents the number of categories of face attributes, and y ij Logits represents the true label of the face attribute ij Represents the predicted value of the face attribute output, represents the sigmoid activation function, r j Indicates the ratio of the number of images with the jth attribute in the second training dataset to the total training images.
2. A face recognition method based on multi-task learning as claimed in claim 1, characterized in that: The step of sending the second training data set to the second network with initialized network weight parameters, training the second training data set, and obtaining the face attribute analysis model is specifically as follows: The second training data set for facial attribute analysis is sent to a preset second network. The network weight parameters of the second network are initialized, and the network weight training is performed on the second training data set until the loss function corresponding to the facial attribute branch converges to obtain the facial attribute analysis model.
3. A face recognition method based on multi-task learning as described in any one of claims 1 to 2, characterized in that: The facial feature matching of the feature image is performed to obtain the face recognition result, specifically: COS similarity calculation is performed on the feature image, if the COS similarity is greater than a first preset value, the face recognition is successful; if the COS similarity is less than or equal to the first preset value, the face recognition fails.
4. A face recognition device based on multi-task learning, characterized in that: It includes a first training module, a freezing module, a second training module, a feature extraction module and a feature matching module; wherein, The first training module is used to obtain a first training data set, a second training data set and a face data set to be recognized, send the first training data set to a preset first network, train the first training data set, and obtain a face recognition model; The freezing module is used to initialize the network weight parameters corresponding to the preset second network through the weight corresponding to the face recognition branch, and freeze the network layer corresponding to the face recognition model; The second training module is used to send the second training data set to the second network with initialized network weight parameters, train the second training data set, and obtain a face attribute analysis model; The feature extraction module is used to extract facial features from the face dataset to be identified through the facial attribute analysis model to obtain a feature image; The feature matching module is used to perform facial feature matching on the feature image to obtain a face recognition result; The first training module sends the first training data set to a preset first network, trains the first training data set, and obtains a face recognition model, specifically: The first training module performs network weight training on the first training data set through a basic network and a face recognition branch until the loss function corresponding to the face recognition branch converges, thereby obtaining the face recognition model; the basic network is a shared network part for the face recognition task and the face attribute analysis task; The face recognition device further includes a branch adding module, which is used to add a face attribute branch after the basic network to form a second network including the corresponding weights of the face attribute branch before the freezing module initializes the network weight parameters corresponding to the preset second network through the weights corresponding to the face recognition branch, and forms a second network including the corresponding weights of the face attribute branch; The loss function corresponding to the face attribute branch is: ; in, ; N represents the total number of images for training, M represents the number of categories of face attributes, and y ij Logits represents the true label of the face attribute ij Represents the predicted value of the face attribute output, represents the sigmoid activation function, r j Indicates the ratio of the number of images with the jth attribute in the second training dataset to the total training images.
5. A face recognition device based on multi-task learning as claimed in claim 4, characterized in that: The second training module sends the second training data set to the second network with initialized network weight parameters, trains the second training data set, and obtains a face attribute analysis model, specifically: The second training module sends a second training data set for facial attribute analysis into a preset second network, and performs network weight training on the second training data set through the second network initialized with network weight parameters until the loss function corresponding to the facial attribute branch converges, thereby obtaining the facial attribute analysis model.
6. A face recognition device based on multi-task learning as claimed in any one of claims 4 to 5, characterized in that: The feature matching module performs facial feature matching on the feature image to obtain a face recognition result, which is specifically: The feature matching module calculates the cos similarity of the feature image. If the cos similarity is greater than a first preset value, face recognition is successful; if the cos similarity is less than or equal to the first preset value, face recognition fails.
Citation Information
Patent Citations
Multi-task deep learning network-based training method, system, multi-task deep learning network-based identification method and system
CN106503669A
Multi-task learning convolutional neural network-based face attribute analysis method
CN106529402A
Face-image recognition method, device and apparatus and storage medium
CN109543547A
A convolutional neural network face recognition method and system based on parameter sharing
CN109726619A
A face recognition method and device based on deep learning
CN109815801A
