Face key point detection method and system based on knowledge distillation and multi-task learning
By employing knowledge distillation and multi-task learning methods, students are guided in network training, reducing the number of parameters and improving detection accuracy. This solves the problem of deploying large backbone networks on mobile devices, achieving lightweight and efficient facial landmark detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QILU UNIVERSITY OF TECHNOLOGY (SHANDONG ACADEMY OF SCIENCES)
- Filing Date
- 2022-08-24
- Publication Date
- 2026-04-10
AI Technical Summary
Existing high-precision facial landmark detection algorithms have excessively large backbone network parameters, making them difficult to deploy on mobile devices.
We employ knowledge distillation and multi-task learning methods, using a teacher network to generate soft labels to guide student network training. By combining multi-task learning and sharing underlying representations, we reduce the number of parameters while improving detection accuracy.
While reducing the number of parameters, the detection accuracy is comparable to that of large-scale networks, solving the deployment problem on mobile devices and achieving lightweight and efficient facial landmark detection.
Smart Images
Figure CN115393934B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of image processing, and particularly relates to a face key point detection method and system based on knowledge distillation and multi-task learning. BACKGROUND
[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.
[0003] Face key point detection is an algorithm that, given a face image, detects and locates key positions such as eyebrows, eyes, nose, and mouth on the face, and is a basic link of face recognition, widely used in three-dimensional reconstruction, video communication, face payment, and other fields, so the related research on face key point detection has important practical significance.
[0004] In recent years, with the rapid development of deep learning, the accuracy of face key point detection has been greatly improved, but most of the high-precision detection algorithms currently use large backbone networks such as Hourglass network and ResNet network. Although these methods have excellent performance, they also introduce a large number of parameters, resulting in high memory consumption, which is not conducive to deployment on mobile devices. SUMMARY
[0005] In order to solve at least one of the technical problems in the background art, the present application provides a face key point detection method and system based on knowledge distillation and multi-task learning, which innovatively proposes a detection method based on knowledge distillation and multi-task learning in the process of constructing a lightweight face key point detection network. Compared with the model without introducing knowledge distillation and multi-task learning, this method greatly reduces the network model parameter quantity relative to the complex network, and the detection accuracy can still rival large networks, effectively avoiding the problem of excessive network model parameter quantity in existing methods, which is not conducive to deployment on mobile devices.
[0006] In order to achieve the above purpose, the present application adopts the following technical solutions:
[0007] The first aspect of the present application provides a face key point detection method based on knowledge distillation and multi-task learning, comprising the following steps:
[0008] Obtain a training data set, which includes face picture data and corresponding real label data;
[0009] Generate corresponding soft label data based on the training data set and the trained teacher network model;
[0010] The student network is trained based on the real label data and the soft label data, knowledge learned by the teacher network is transferred to the student network, meanwhile, multiple related tasks are learned together according to a multi-task learning method, learned information is shared and supplemented by shallow shared representation during the learning process, and finally, the student network is well trained.
[0011] The face key points and the head posture are obtained by combining the recognized face picture and the finally well trained student network.
[0012] The second aspect of the present application provides a face key point detection system based on knowledge distillation and multi-task learning, comprising:
[0013] A data acquisition module is configured to acquire a training data set, wherein the data set comprises face picture data and corresponding real label data;
[0014] A soft label data generation module is configured to generate corresponding soft label data based on the training data set and the well trained teacher network model;
[0015] A student network training module is configured to train the student network based on the real label data and the soft label data, transfer the knowledge learned by the teacher network to the student network, meanwhile, multiple related tasks are learned together according to a multi-task learning method, learned information is shared and supplemented by shallow shared representation during the learning process, and finally, the student network is well trained.
[0016] A multi-task recognition module is configured to obtain the face key points and the head posture by combining the recognized face picture and the finally well trained student network.
[0017] The third aspect of the present application provides a computer readable storage medium.
[0018] A computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and the program is executed by a processor to implement the steps of the face key point detection method based on knowledge distillation and multi-task learning.
[0019] The fourth aspect of the present application provides a computer device.
[0020] A computer device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the face key point detection method based on knowledge distillation and multi-task learning when executing the program.
[0021] Compared with the prior art, the present application has the following beneficial effects:
[0022] The application innovatively proposes to use knowledge distillation and multi-task learning method in the process of detecting face key points, knowledge distillation can not only guarantee that the parameter quantity of the student network remains unchanged, but also improve the precision of the student network to rival the teacher network, multi-task learning helps each other to learn through the shared representation of the bottom layer, improves the generalization effect, and effectively avoids the problem that the parameter quantity of the network model in the existing method is too large and is not conducive to deployment on mobile devices.
[0023] Advantages of the additional aspects of the application will be partially given in the following description, partially will become apparent from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS
[0024] The accompanying drawings, which form a part of this specification, are included to provide a further understanding of the application, and are incorporated by reference herein.
[0025] Figure 1 Fig. 1 is a schematic diagram of the overall process of the face key point detection method based on knowledge distillation and multi-task learning of the embodiment of the application;
[0026] Figure 2 Fig. 2 is a schematic diagram of the soft label generated by the student network learning the teacher network and the real label of the photo based on knowledge distillation of the embodiment of the application;
[0027] Figure 3 Fig. 3 is a schematic diagram of the multi-task learning process of the embodiment of the application. DETAILED DESCRIPTION
[0028] The application will be further described below in conjunction with the drawings and embodiments.
[0029] It should be noted that the following detailed description is all exemplary, and is intended to provide further explanation of the application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the application belongs.
[0030] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the application. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form, and in addition, it should be understood that when the terms "comprise" and / or "include" are used in the specification, they mean that the features, steps, operations, devices, components and / or combinations thereof are present.
[0031] As mentioned in the background art, the current detection network existing in the prior art introduces a large number of parameters, resulting in high memory consumption, which is not conducive to deployment on mobile devices. The present application provides a face key point detection processing method based on knowledge distillation and multi-task learning. Knowledge distillation can improve the accuracy of the student network by guiding the student network to train through a high-precision and high-complexity teacher network. Multi-task learning helps each other learn through shared representation at the bottom. The main task uses the domain-related information possessed by the training signal of the related task to complement each other's learned domain-related information and promote learning to improve the generalization effect. Therefore, knowledge distillation and multi-task learning are introduced. The method provides an efficient lightweight face key point detection network. After the guidance of the teacher network and the simultaneous learning of multi-task, the parameter quantity of the student network is greatly reduced compared with the cascade network and large backbone network, and the detection accuracy is still comparable to large networks.
[0032] Embodiment one
[0033] As Figure 1 shown, the present embodiment provides a face key point detection method based on knowledge distillation and multi-task learning, comprising the following steps:
[0034] Step 1: Select a training data set, which includes face picture data and corresponding real label data;
[0035] Step 2: Data augmentation preprocessing is performed on the face picture data, which mainly includes color jitter, noise enhancement, brightness adjustment, angle rotation, horizontal mirror, random grayscale and random erasing.
[0036] Specifically, the color jitter reduces the sensitivity of the network to the color of the image by randomly adjusting the color channel of the image.
[0037] Noise enhancement increases the robustness of the network by randomly adding salt and pepper noise to the image.
[0038] Brightness adjustment randomly adjusts the brightness value of the face image, and the random adjustment factor is 0.5-1.5.
[0039] Angle rotation rotates the image at a random angle of-30°-30°.
[0040] Horizontal mirror horizontally rotates the image with a probability of 50%.
[0041] Random grayscale converts the image to a grayscale image with a probability of 20%.
[0042] Random erasing improves the generalization ability of the network to the occlusion situation by randomly erasing the face image with a random size. The erasing area is 2%-33% of the entire image.
[0043] After the above data enhancement part is executed, the processed data is learned by the teacher network.
[0044] Step 3: generating corresponding soft label data based on the preprocessed training data set and the trained teacher network model;
[0045] In this embodiment, ConvNeXt is selected as the teacher network.
[0046] By taking the output of the teacher network with complex structure and superior performance as the soft label, the student network with simple structure is supervised to learn, so as to transfer the "knowledge" learned by the teacher network to the student network.
[0047] It should be noted that the soft label is relative to the hard label, and the hard label is the corresponding true label. The soft label is the result predicted by the teacher network, which is not necessarily 100% correct.
[0048] Step 4: training the student network based on the true label data and the soft label data, transferring the knowledge learned by the teacher network to the student network, and simultaneously learning multiple related tasks according to the multi-task learning method. During the learning process, the shallow shared representation is used to share and supplement the learned information, so as to obtain the finally trained student network.
[0049] In this embodiment, in step 4, training the student network based on the true label data and the soft label data and transferring the knowledge learned by the teacher network to the student network includes:
[0050] By calculating the accuracy of the coordinate points generated by the student network and the coordinate points generated by the teacher network, if the accuracy of the coordinate points generated by the student network is lower than that of the coordinate points generated by the teacher network, the student network continues to learn from the teacher network until the accuracy of the student network exceeds that of the teacher network, and an initially trained student network is obtained.
[0051] In this embodiment, the block of MobileNetV2 is selected as the block of the student network.
[0052] As shown in Figure 2 the input image is X, the trained complex teacher network model is T_model, the student network model being trained is S_model, the soft label generated by the teacher network is soft_label, the true label of the image is hard_label, the number of key points is N, and the key point coordinates are {(x 1, y1),(x 2, y2),(x 3, y3),…,(x N, y N), the loss of the result S_landmark of the student network and the soft label soft_label is Limit loss, the loss of the result S_landmark of the student network and the real label hard_label of the image is Wing loss, and the overall loss is total loss.
[0053] Specifically, the following steps are included:
[0054] 1) First, the teacher network generates a corresponding soft label soft_label according to an input image X, and the corresponding key point coordinates in the soft_label are {(x1, y1), (x2, y2), (x3, y3), …, (xN, yN)}.
[0055] 2) The student network generates a corresponding coordinate set S_landmark according to the input image X, and the corresponding key point coordinates in S_landmark are {(x1, y1), (x2, y2), (x3, y3), …, (xN, yN)}.
[0056] 3) The Wing loss is used to calculate the loss of S_landmark and hard_label.
[0057] 4) Then, the Limit loss is used to calculate the loss of S_landmark and soft_label, where the Wing loss is represented by f(x), and the Limit loss is represented as follows:
[0058]
[0059] When the loss of S_landmark and soft_label is greater than the loss of S_landmark and hard_label, the Limit loss is f(S_landmark-soft_label), and otherwise, the Limit loss is 0.
[0060] 5) Finally, the overall loss total loss = λ*Limit loss + (1-λ)*Wing loss is used to update the weight parameters of the student network through backpropagation.
[0061] In step 4, multiple related tasks are learned together according to the multi-task learning method, and the learned information is shared and supplemented through shallow shared representation during the learning process to obtain a finally trained student network.
[0062] The purpose of using multi-task learning is to further improve the accuracy of the student network model. Multi-task learning learns multiple related tasks together, and during the learning process, the learned domain-related information is shared and supplemented by a shallow shared representation to promote learning and improve generalization.
[0063] As shown in the figure, Figure 3 In this embodiment, task 1 is face key point detection, task 2 is head pose estimation, Layer 1 and Layer 2 are the bottom shared representation, so that a single model can complete multiple tasks. Since the head pose estimation task and the face key point detection task have strong correlation, the head pose estimation is selected as the auxiliary task and is learned together with the face key point detection task.
[0064] Step 5: Combine the to-be-recognized face picture and the finally trained student network to detect the face key points and the head pose.
[0065] Head pose estimation is to obtain the pose angle of the head through a face image. In 3D space, the rotation of an object can be represented by three Euler angles:
[0066] Pitch (rotation around the X axis), yaw (rotation around the Y axis) and roll (rotation around the Z axis) are calculated respectively, and the names are pitch angle, yaw angle and roll angle.
[0067] Where the head pose data label is pose_label, the predicted head pose data result is pose_predict, and the loss function is pose loss.
[0068] The input image X is preprocessed and input to the student network for learning. The student network predicts the result pose_predict, and then pose_predict and pose_label are input to the pose loss for loss calculation. Then the student network parameters are updated through back propagation. The training of the student network based on knowledge distillation and multi-task learning is synchronous, and the training process is completed until the student network converges.
[0069] The above scheme has the advantages that compared with the traditional deep learning network, the network of the present application can detect face key points and head pose at the same time, and the network of the present application can detect face key points and head pose at the same time. While ensuring that the parameter amount is greatly reduced compared with the cascade network and large backbone network, the detection accuracy can still rival large networks, solving the problem of deploying traditional neural networks on mobile devices.
[0070] Table 1 shows the test results of each model on the 300W dataset. It can be seen that the model of the present application achieves the best result compared with other deep learning models. Table 2 shows the comparison of the parameter amount of the teacher model, the student model, the student model after using knowledge distillation and the LAB model used in the present application. It can be seen that the parameter amount of the model of the present application is greatly reduced compared with the traditional neural network model.
[0071] Table 1 Comparison of each model on the 300W dataset
[0072] Method Common Challenging Full RCPR 6.18 17.26 8.35 CFAN 5.50 16.78 7.69 ESR 5.28 17.00 7.58 SDM 5.57 15.40 7.50 LBF 4.95 11.98 6.32 CFSS 4.73 9.98 5.76 3DDFA 6.15 10.59 7.01 TCDCN 4.80 8.60 5.54 MDM 4.83 10.14 5.88 SeqMT 4.84 9.93 5.74 RAR 4.12 8.35 4.94 DVLN 3.94 7.62 4.66 CPM 3.39 8.14 4.36 DCFE 3.83 7.54 4.55 TSR 4.36 7.56 4.99 LAB 3.42 6.98 4.12 Ours 3.22 6.04 3.69
[0073] Table 2 Comparison of model parameter amount
[0074] Method Param(M) LAB 25.1 Teacher 27.95 Student 1.6 Student+KD 1.6
[0075] Embodiment Two
[0076] The embodiment provides a face key point detection system based on knowledge distillation and multi-task learning, comprising:
[0077] A data acquisition module is configured to acquire a training data set, wherein the data set comprises face picture data and corresponding real label data.
[0078] A soft label data generation module is configured to generate corresponding soft label data based on the training data set and the trained teacher network model.
[0079] A student network training module is configured to train a student network based on the real label data and the soft label data, to transfer the knowledge learned by the teacher network to the student network, and to simultaneously learn multiple related tasks together according to a multi-task learning method, to share and supplement the learned information through a shallow shared representation during the learning process, and to obtain a finally trained student network.
[0080] A multi-task recognition module is configured to obtain face key points and head posture by combining the face picture and the finally trained student network.
[0081] Embodiment Three
[0082] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the steps of the face key point detection method based on knowledge distillation and multi-task learning.
[0083] Embodiment Four
[0084] The embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps in the face key point detection method based on knowledge distillation and multi-task learning when executing the program.
[0085] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage, etc.) containing computer-usable program code.
[0086] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowchart
[0087] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowchart
[0088] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a process for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowchart
[0089] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.
[0090] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. The present application can be variously changed and modified by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A face key point detection method based on knowledge distillation and multi-task learning, characterized in that, The method comprises the following steps: obtain a training data set, which comprises face picture data and corresponding real label data; generate corresponding soft label data based on the training data set and a trained teacher network model; train a student network based on the real label data and the soft label data, transfer the knowledge learned by the teacher network to the student network, and simultaneously learn multiple related tasks together according to a multi-task learning method, share and supplement the learned information through shallow shared representation during the learning process, and obtain a finally trained student network; during the training of the student network, perform loss calculation on a coordinate point set generated by the student network and real labels by using a Wing loss; then perform loss calculation on the coordinate point set generated by the student network and a coordinate point set generated by the teacher network by using a Limit loss; finally, update the weight parameters of the student network through back propagation of an overall loss; when the loss of the coordinate point set generated by the student network and the coordinate point set generated by the teacher network is greater than the loss of the coordinate point set generated by the student network and the real labels, the Limit loss loss is the loss of the coordinate point set generated by the student network and the coordinate point set generated by the teacher network, and is 0 in other cases; combine the recognition of the face picture and the finally trained student network to obtain face key points and head poses.
2. The face key point detection method based on knowledge distillation and multi-task learning according to claim 1, wherein, The process of transferring the knowledge learned by the teacher network to the student network comprises: comparing the accuracy of the coordinate point set generated by the student network and the coordinate point set generated by the teacher network, if the accuracy of the coordinate point set generated by the student network is lower than the accuracy of the coordinate point set generated by the teacher network, the student network continues to learn from the teacher network until the accuracy of the student network exceeds that of the teacher network. 3.The face key point detection method based on knowledge distillation and multi-task learning according to claim 1, wherein, After obtaining the training data set, perform enhancement preprocessing on the data, including color jittering, noise enhancement, brightness adjustment, angle rotation, horizontal mirroring, random grayscale and random erasing. 4.The face key point detection method based on knowledge distillation and multi-task learning according to claim 1, wherein, The teacher network selects ConvNeXt, and the student network selects MobileNetV2.
5. The face key point detection method based on knowledge distillation and multi-task learning according to claim 1, wherein, Take face key point detection as task one and head pose estimation as task two, and take task two as an auxiliary task of task one to learn together with task one.
6. A face key point detection system based on knowledge distillation and multi-task learning, characterized in that, It comprises: a data acquisition module for acquiring a training data set, which comprises face picture data and corresponding real label data; a soft label data generation module for generating corresponding soft label data based on the training data set and a trained teacher network model; a student network training module for training a student network based on the real label data and the soft label data, transferring the knowledge learned by the teacher network to the student network, and simultaneously learning multiple related tasks together according to a multi-task learning method, sharing and supplementing the learned information through shallow shared representation during the learning process, and obtaining a finally trained student network; during the training of the student network, perform loss calculation on a coordinate point set generated by the student network and real labels by using a Wing loss; Then the Limit loss is used to calculate the loss between the coordinate point set generated by the student network and the coordinate point set generated by the teacher network; Finally, the overall loss is used for back propagation to update the weight parameters of the student network; When the loss between the coordinate point set generated by the student network and the coordinate point set generated by the teacher network is greater than the loss between the coordinate point set generated by the student network and the real label, the Limit loss is the loss between the coordinate point set generated by the student network and the coordinate point set generated by the teacher network, and is 0 in other cases; The multi-task recognition module is used to combine the recognized face picture and the finally trained student network to obtain the face key points and the head posture.
7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the face key point detection method based on knowledge distillation and multi-task learning in any one of claims 1-5.
8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the face key point detection method based on knowledge distillation and multi-task learning in any one of claims 1-5.