Face key point detection method and model training method, and terminal device

Abstract

The embodiment of the present application provides a facial key point detection method and its model training method, and a terminal device. The facial key point detection model training method mainly utilizes knowledge distillation technology. During the model training process, a teacher network is utilized to guide the training of a student network. In addition, based on the attention mechanism, the teacher network supervises the intermediate features of the student network. This can not only greatly improve the performance of the model without affecting the number of student network parameters, but also effectively avoid overfitting of the student network to the teacher network, thereby further improving the performance of the model, making the trained model more effective for face detection, and is particularly suitable for mobile devices with very limited computing power.

Description

Technical Field

[0001] The present application relates to the technical field of face detection, and in particular to a method for detecting key points of a face, a model training method thereof, and a terminal device. Background Art

[0002] In daily life, face detection technology is needed in many situations, which involves the detection of facial key points. For example, facial key point technology can be used to analyze the sleeping state of a baby. However, the existing mainstream solutions for facial key point detection are mostly implemented through computers and other devices with strong computing power. For some mobile devices with very limited computing power, directly deploying the existing facial key point detection model will greatly reduce the expressiveness of the model. However, if the network structure is increased, it will cause a huge algorithm burden on the mobile device. Summary of the invention

[0003] The embodiments of the present application provide a facial key point detection method and a model training method thereof, and a terminal device, which can be applied to mobile devices with limited computing power, such as mobile robots, smart phones, etc. The training method improves the model training efficiency and can also realize accurate detection of facial key points, etc.

[0004] In a first aspect, an embodiment of the present application provides a method for training a facial key point detection model, comprising:

[0005] Inputting sample images containing facial key point annotations into the student network and the teacher network of the facial key point detection model for feature extraction, respectively, to output corresponding intermediate feature maps and facial key point location prediction results; the intermediate feature maps include early feature maps and late feature maps;

[0006] Performing dimensionality reduction processing on the early feature maps output by the teacher network and the student network through the attention module of the face key point detection model to obtain respective significant feature maps;

[0007] If the teacher network has been pre-trained, the total loss value of the current training of the facial key point detection model is calculated according to a first loss value between the salient feature maps of the teacher network and the student network, a second loss value between the late feature maps output by the teacher network and the student network, and a third loss value between the positioning prediction value and the true value of the facial key point of the student network;

[0008] When the total loss value of this training does not meet the preset conditions, the next sample image is input to continue training until the calculated total loss value meets the preset conditions, thereby obtaining a trained face key point detection model.

[0009] In some embodiments, the facial key point detection model training method further includes:

[0010] If the teacher network has not been pre-trained, the total loss value of the facial key point detection model is calculated based on a first loss value between the significant feature maps of the teacher network and the student network, a second loss value between the teacher network and the late feature maps output by the student network, a third loss value between the positioning prediction value and the true value of the facial key points of the student network, and a fourth loss value between the positioning prediction value and the true value of the facial key points of the teacher network.

[0011] In some embodiments, the dimensionality reduction processing includes: performing an averaging operation on the intermediate feature maps output by the teacher network and the student network respectively in all channels.

[0012] In some embodiments, the teacher network and the student network both include: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a maximum global pooling layer, a first fully connected layer, and a second fully connected layer connected in sequence;

[0013] The second convolution layer includes a convolution kernel module and a bottleneck module; the third convolution layer includes a convolution kernel module and two identical bottleneck modules.

[0014] In some embodiments, the intermediate feature map includes a first early feature map, a second early feature map, and a third early feature map output by the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively, and a late feature map output by the first fully connected layer.

[0015] In some embodiments, the first loss value and the second loss value are both calculated using the Smooth L1 function as the loss function;

[0016] The third loss value and the fourth loss value are both calculated using the Wing loss function as the loss function.

[0017] In a second aspect, the embodiment of the present application further provides a method for detecting key points of a face, comprising:

[0018] Obtain a target face image, and input the target face image into a trained face key point detection model to perform face key point detection, wherein the face key point detection model is pre-trained using the above method;

[0019] Output the facial key point positioning prediction result corresponding to the target facial image.

[0020] In a third aspect, an embodiment of the present application further provides a method for detecting a user's sleep state, comprising:

[0021] The facial key point detection method is used to detect the facial key points of the user, and a facial key point positioning prediction result of the user is obtained;

[0022] The facial key point positioning prediction result is analyzed according to a preset sleeping posture to determine whether the user has entered a sleeping state.

[0023] In a fourth aspect, the embodiment of the present application further provides a facial key point detection model training device, comprising:

[0024] A network processing module, used for inputting sample images containing facial key point annotations into the student network and the teacher network of the facial key point detection model for feature extraction, so as to output corresponding intermediate feature maps and facial key point location prediction results respectively; the intermediate feature maps include early feature maps and late feature maps;

[0025] An attention module, used for performing dimensionality reduction processing on the early feature maps output by the teacher network and the student network through the attention module in the face key point detection model to obtain respective significant feature maps;

[0026] A loss calculation module, for calculating the total loss value of the current training of the face key point detection model according to a first loss value between the salient feature maps of the teacher network and the student network, a second loss value between the late feature maps output by the teacher network and the student network, and a third loss value between the positioning prediction value and the true value of the face key point of the student network, if the teacher network has been pre-trained;

[0027] The training judgment module is used to input the next sample image to continue training when the total loss value of this training does not meet the preset conditions, and stop when the calculated total loss value meets the preset conditions to obtain the trained face key point detection model.

[0028] In a fifth aspect, an embodiment of the present application further provides a terminal device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned face key point detection model training method or face key point detection method.

[0029] In a sixth aspect, an embodiment of the present application further provides a readable storage medium storing a computer program, which, when executed on a processor, implements the above-mentioned facial key point detection model training method or facial key point detection method.

[0030] The embodiments of the present application have the following beneficial effects:

[0031] The facial key point detection model training method provided in the embodiment of the present application mainly combines knowledge distillation technology to optimize the detection of facial key points. During the model training process, the teacher network is used to guide the training of the student network, and the teacher network is supervised by the student network in the intermediate features based on the attention mechanism. It can not only greatly improve the performance of the model without affecting the number of student network parameters, but also effectively avoid overfitting of the student network to the teacher network, thereby further improving the performance of the model, making the trained model more effective for face detection, and the facial key point detection model can be especially deployed on some mobile devices with very limited computing power, such as mobile robots, smart phones, tablets and some smart home devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A schematic diagram of the structure of a terminal device according to an embodiment of the present application is shown;

[0034] Figure 2 A schematic diagram showing a flow chart of a method for training a face key point detection model according to an embodiment of the present application is shown;

[0035] Figure 3 A schematic diagram showing the structure of a teacher network and a learning network of a face key point detection model according to an embodiment of the present application is shown;

[0036] Figure 4 Shown based on Figure 3 Schematic diagram of full-channel feature supervision by the teacher network and learning network;

[0037] Figure 5 A schematic diagram of the structure of a facial key point detection model including an attention module according to an embodiment of the present application is shown;

[0038] Figure 6 Another schematic diagram of the process of training a facial key point detection model according to an embodiment of the present application is shown;

[0039] Figure 7 A schematic diagram showing the flow of a method for detecting key points of a face according to an embodiment of the present application is shown;

[0040] Figure 8 A schematic diagram showing a process flow of a method for detecting a user's sleep state according to an embodiment of the present application is shown;

[0041] Fig. 9 A structural schematic diagram of a facial key point detection model according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0043] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0044] Hereinafter, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0045] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0046] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0047] Please refer to Figure 1, which is a schematic diagram of the structure of the terminal device 10 proposed in the embodiment of the present application. Exemplarily, the terminal device 10 may include a processor 11 and a memory 12, wherein the memory 12 is connected to the processor 11, and the memory 12 stores a corresponding computer program, and the processor 11 is used to execute the computer program to implement the face key point detection model training method or face key point detection method in the embodiment of the present application, which can effectively solve the problem that the existing face key point detection model is used in a mobile terminal device with very limited computing power, resulting in a significant reduction in the expressiveness of the model, but increasing the network will cause a great algorithm burden on the mobile terminal device.

[0048] In this embodiment, the processor 11 may be an integrated circuit chip with signal processing capability. The processor 11 may be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., which may implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application.

[0049] In this embodiment, the memory 12 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 12 is used to store a computer program, and the processor 11 may execute the computer program accordingly after receiving an execution instruction.

[0050] It is worth noting that the terminal device 10 in the embodiment of the present application can be a device with strong computing power, such as a computer, a laptop computer, etc., but it can also be some mobile devices or embedded devices with very limited computing power, such as mobile robots, smart phones, tablets, and even some smart home devices. The face key point detection model obtained by the training method in the embodiment of the present application is deployed in these terminal devices 10 to perform face key point detection in some required scenarios. For example, by obtaining the user's facial posture to further determine whether the user is in a sleeping state, whether he is physically unwell, or the current emotional state, etc. Among them, the above-mentioned face key points mainly refer to important feature points that can reflect the uniqueness of the face, for example, including but not limited to eyes, mouth, nose, eyebrows, etc., which can be selected according to actual needs and are not limited here.

[0051] Based on the above-mentioned terminal device 10, an embodiment of the present application proposes a facial key point detection model training method. The facial key point detection model obtained by the training method can be applied to the terminal devices 10 listed above, especially some mobile devices with limited computing power. It can still achieve fast and accurate detection of facial key points without increasing the burden on the mobile devices.

[0052] Considering that the trained face key point detection model will be deployed on some device platforms with very limited computing power, such as the int8 platform for deployment, this embodiment will implement face key point detection based on the regression method. Further, when constructing the network architecture of the above-mentioned face key point detection model, some lightweight neural network architectures are mainly considered, such as the Mobilenet series network, specifically using the Backbone of Mobilenet as the network foundation, etc.

[0053] Furthermore, in order to improve the expressiveness of these lightweight networks, this embodiment will combine knowledge distillation technology to optimize the constructed model. It is worth noting that the reason why knowledge distillation is proposed is that when the existing face key point detection model (such as RCNN, Fast RCNN, Faster RCNN, etc.) is used for training and deployment, taking the int8 platform as an example, the same model is trained and deployed on pytorch, fp32 and int8 platforms. It is found that the model performance on the int8 platform drops sharply, and the specific errors are shown in Table 1 below. Among them, the parameter ION is used to reflect the error of the key point, and the larger the ION, the greater the error. Therefore, for devices with limited computing power, it is not feasible to directly use the existing face detection model. For this reason, it is proposed to use a non-quantized model to guide the training of the quantitative model, that is, to use the teacher network in knowledge distillation to guide the training of the student network, so as to greatly improve the performance of the model without affecting the number of parameters of the quantitative model.

[0054] Table 1 Model errors of three platforms

[0055] platform ION Pytorch 4.969 fp32 5.0225 int8 5.6648

[0056] Based on this, with the Backbone of the above-mentioned Mobilenet as the basic structure, this embodiment constructs the teacher network and the student network in the model respectively, wherein the structure of the constructed teacher network and the learning network is the same. Exemplarily, Backbone may include a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a maximum global pooling (GMP) layer, a first fully connected layer and a second fully connected layer connected in sequence. In one embodiment, the specific structure of each layer may be as shown in Table 2, wherein the first convolutional layer is used as the input of the convolutional layer, which may include a convolutional module, such as using a 3*3 convolutional kernel (conv3*3), etc.; the second convolutional layer is used for the first stage extraction, and may include a convolutional kernel (such as conv3*3, etc.) and a bottleneck module, which can reduce the feature dimension, reduce the amount of calculation, etc.; the third convolutional layer is used for the second stage extraction, and may include a convolutional kernel (such as conv3*3, etc.) and two identical bottleneck modules, which can further reduce the dimension, reduce the amount of calculation, etc.; the fourth convolutional layer is used as the output of the convolutional layer, which may include a convolutional kernel (such as conv3*3, etc.). The sizes of convolution kernels in different convolution layers can be equal, but the number of channels of each convolution kernel may be different. The two fully connected modules can be implemented by using a linear function with 176 channels and a linear function with 196 channels, respectively.

[0057] Table 2 Backbone network structure

[0058]

[0059]

[0060] Based on the constructed face key point detection model, please refer to Figure 2 , exemplary, the facial key point detection model training method includes steps S110 to S140:

[0061] Step S110, input the sample images containing facial key point annotations into the student network and teacher network of the facial key point detection model for feature extraction, so as to output corresponding intermediate feature maps and facial key point positioning prediction results respectively, wherein the intermediate feature maps include early feature maps and late feature maps.

[0062] As an example, when constructing Figure 3After the student network and teacher network shown in the figure are constructed, the model needs to be trained. During the training process, images with pre-annotated facial key points can be input into the student network and the teacher network to perform feature extraction processing through the corresponding layers or modules in the corresponding networks, thereby obtaining feature maps output by some intermediate layers and the final facial key point location prediction results. Then, the parameters in the network are adjusted by calculating the loss value of these output data, and the training is iterated until the network is adjusted to achieve the expected detection effect, thereby obtaining a trained facial key point detection model.

[0063] It can be understood that the early feature map refers to the feature map output by the corresponding layer close to the input side of the network, such as the feature maps corresponding to the output of the first convolutional layer, the second convolutional layer, the third convolutional layer, etc.; and the late feature map refers to the feature map output by the corresponding layer close to the output side of the network, such as the feature map output by the first fully connected layer.

[0064] It is worth noting that when proposing to use the teacher network to guide the training of the student network, taking into account that the regression task is generally highly abstract and face recognition generally distills the previous feature layers, this embodiment will propose to use the feature map output by the first fully connected layer (i.e., fea_fc) in the network as the later feature map for feature extraction supervision. When the student network is trained, the feature map output by its first fully connected layer is used to calculate the loss with the feature map of the corresponding position obtained during the teacher network training. In this way, the purpose of the teacher leading the students to learn can be achieved.

[0065] Usually, after using the feature map output by the first fully connected layer to supervise to a certain extent, the feature map output by the previous layer is then combined for supervision, such as Figure 4 As shown, however, it was found that when the previous Feature map is used for supervision, the output result of the model is worse. In this regard, after analysis, it is found that: on the one hand, Featuremap supervision requires feature supervision of all channels, and each Feature map contains multiple channels, each channel is responsible for extracting the texture or detail features of a certain component, so it is possible to extract some meaningless information. These meaningless channel features, as supervision of the student network, will guide students to meaningless learning, thereby weakening the feature extraction ability of the model. On the other hand, the Feature map extracted by the teacher network is a complex hyperplane, and students will fall into overfitting if they learn too much about this plane. Therefore, in order to avoid the student network from overfitting the teacher network, during the training process, this embodiment will further combine the more significant features with lower dimensions to perform intermediate process supervision, which can further improve the performance of the student network. Specifically, some feature maps output by the front convolutional layer are processed accordingly through the attention mechanism. For example, based on Figure 4 The teacher network and student network structures shown in one embodiment can select the feature maps output by the first convolutional layer, the second convolutional layer, and the third convolutional layer for intermediate process supervision.

[0066] Step S120, the early feature maps output by the teacher network and the student network are subjected to dimensionality reduction processing through the attention module in the face key point detection model to obtain respective significant feature maps.

[0067] The above-mentioned dimensionality reduction processing mainly refers to the operation of taking the average value of the early feature maps output by the teacher network and the student network in all channels through the attention module. For example, Figure 5 As shown in the figure, an attention module with a channel number of 1 (corresponding to the attenmap module) is used to perform feature weighted calculations on each intermediate feature map with multiple channels, thereby obtaining a low-dimensional significant feature map. It can be understood that due to the full channel averaging operation, the overall characteristics of the object can be reflected, thereby ensuring that the intermediate process learned by the students is similar to that of the teacher, and the problem-solving ideas are similar, but better detection results can be obtained.

[0068] Step S130, if the teacher network has been pre-trained, the total loss value of the current training of the face key point detection model is calculated based on the first loss value between the salient feature maps of the teacher network and the student network, the second loss value between the late feature maps output by the teacher network and the student network, and the third loss value between the positioning prediction value and the true value of the face key point of the student network.

[0069] The teacher network can be pre-trained or trained synchronously with the student network. It is worth noting that if synchronous joint training is adopted, the loss function constructed by the face key point detection model will be different.

[0070] Taking the teacher network as an example after pre-training, in one embodiment, the loss function constructed by the face key point detection model includes three parts, such as Figure 5 As shown in the figure, the first part is the loss between the salient feature maps of the teacher network and the student network, for example, the Smooth L1 function can be used as the loss function; the second part is the loss between the late feature maps output by the teacher network and the student network, for example, the Smooth L1 function can also be used as the loss function; and the third part is the loss between the positioning prediction value of the facial key points of the student network and the true value (ground truth), for example, the Wing loss function can be used as the loss function. Among them, the use of Wing loss can better capture some minor errors of key points, which can improve the recognition accuracy of network training, etc.

[0071] Step S140, when the total loss value of this training does not meet the preset conditions, input the next sample image to continue training until the calculated total loss value meets the preset conditions, and then stop to obtain the trained face key point detection model.

[0072] The above preset conditions are also training stop conditions, which are used to determine when to stop training. For example, the preset conditions may include but are not limited to the total loss value being less than a preset threshold value. In addition, the preset conditions may also include the total loss value and the number of iterations meeting the requirements at the same time, which are not specifically limited here.

[0073] Exemplarily, through the above steps S110~130, the total loss value of the model after training the input sample image can be calculated, and whether the total loss value meets the preset conditions can be judged to determine whether the next iterative training is required. If the preset conditions are met, it means that the prediction result of this output meets the expectations, and the training can be stopped. Otherwise, it is necessary to continue training until the final calculated total loss value meets the preset conditions, then stop and obtain the face key point detection model. Then, the trained face key point detection model can be deployed for subsequent face key point detection. Among them, the number of key points of the predicted output is related to the setting of the last second fully connected layer, for example, Figure 5 The model shown will predict and output 196 key points. Of course, other numbers of key points can also be predicted according to actual needs, which is not limited here.

[0074] The face key point detection model training method proposed in this embodiment is based on knowledge distillation technology and adopts a method of using a non-quantized model to guide the training of a quantitative model, that is, using a teacher network to guide the training of a student network, so as to greatly improve the performance of the model without affecting the number of parameters of the quantitative model. At the same time, in order to avoid the student network from overfitting the teacher network, during the training process, the features after linear regression and the attention mechanism are combined for joint supervision, which can further improve the performance of the student network.

[0075] As another preferred solution, considering that when data update and model iteration are required, a teacher network needs to be trained each time, and it takes a long time to wait. In order to solve this problem, when training the face key point detection model, the embodiment of the present application also proposes a training method, that is, synchronously jointly training the teacher network and the student network without pre-training the teacher network, thereby improving the model training efficiency.

[0076] like Figure 6 As shown, exemplarily, the facial key point detection model training method includes steps S210 to S240:

[0077] In step S210, sample images containing facial key point annotations are respectively input into the student network and the teacher network of the facial key point detection model for feature extraction, so as to output corresponding intermediate feature maps and facial key point location prediction results respectively, wherein the intermediate feature maps include early feature maps and late feature maps.

[0078] Step S220, the attention module of the facial key point detection model is used to perform dimensionality reduction processing on the early feature maps output by the teacher network and the student network to obtain their respective significant feature maps.

[0079] Step S230, if the teacher network has not been pre-trained, the total loss value of the facial key point detection model is calculated based on the first loss value between the salient feature maps of the teacher network and the student network, the second loss value between the late feature maps output by the teacher network and the student network, the third loss value between the positioning prediction value and the true value of the facial key points of the student network, and the fourth loss value between the positioning prediction value and the true value of the facial key points of the teacher network.

[0080] Step S240, when the total loss value of this training does not meet the preset conditions, input the next sample image to continue training until the calculated total loss value meets the preset conditions, and then stop to obtain the trained face key point detection model.

[0081] It can be understood that when the teacher network and the student network are synchronously jointly trained, steps S210 to 240 of the training method of this embodiment are similar to the steps in the above embodiment, so they are not repeated here. The main difference between the two is that the total loss value of the model during synchronous joint training also includes the corresponding loss of the face key point prediction results of the teacher network. Specifically, in addition to the above three parts of loss, the loss function of the face key point detection model at this time also includes the loss function between the positioning prediction value and the true value of the face key point of the teacher network, such as using the Wing loss function.

[0082] The facial key point detection model training method proposed in this embodiment utilizes knowledge distillation to optimize the facial key point detection model, and jointly trains the teacher network and the student network, which not only improves the recognition accuracy of the model, but also improves the iteration efficiency and achieves better joint learning performance under certain circumstances.

[0083] Please refer to Figure 7 Based on the face key point detection model training method of the above embodiment, this embodiment proposes a face key point detection method. Exemplarily, the face key point detection method includes:

[0084] Step S310, obtaining a target face image.

[0085] For example, the target facial image may be directly captured by a camera installed on the terminal device 10, or may be a received facial picture, etc., which is not limited here.

[0086] Step S320: input the target facial image into a trained facial key point detection model to perform facial key point detection, wherein the facial key point detection model is pre-trained and deployed using the above-mentioned method.

[0087] Step S330, outputting the facial key point location prediction result corresponding to the target facial image.

[0088] Exemplarily, after inputting the target face image into the trained model, the corresponding face key point detection result can be obtained. It can be understood that the options of the face key point detection model training method in the above embodiment are also applicable to this embodiment, so they will not be described again here.

[0089] Please refer to Figure 8 Based on the facial key point detection method of the above embodiment, this embodiment proposes a user sleep state detection method, for example, it can be used to monitor the sleep state of patients, infants, etc. Exemplarily, the user sleep state detection method includes:

[0090] Step S410: Use the above-mentioned facial key point detection method to detect facial key points of the user to obtain the facial key point prediction results of the user.

[0091] Step S420: analyzing the predicted result of the facial key points according to the preset sleeping posture to determine whether the user has entered a sleeping state.

[0092] The preset sleeping posture can be obtained by learning the posture of the user in the sleeping state. Since the facial postures of people in different states are often different, for example, in the sleeping state, the eyes are usually closed, and the mouth of some users is open, etc., different postures can be used to further analyze the state of the user. For example, for a baby, it can be judged whether the baby is asleep based on the detected key point features of the face.

[0093] It can be understood that in addition to being used for sleep state detection, the facial key point detection method of this embodiment can be used for other scenarios that require facial key point detection, especially for those that need to be deployed in some mobile devices, embedded devices, etc. with very limited computing power.

[0094] Please refer to Fig. 9 Based on the face key point detection model training method of the above embodiment, this embodiment proposes a face key point detection model training device 100. Exemplarily, the face key point detection model training device 100 includes:

[0095] The network processing module 110 is used to input the sample images containing the facial key point annotations into the student network and the teacher network of the facial key point detection model for feature extraction, so as to output the corresponding intermediate feature map and the facial key point positioning prediction results respectively; the intermediate feature map includes the early feature map and the late feature map;

[0096] An attention module 120 is used to perform dimensionality reduction processing on the early feature maps output by the teacher network and the student network through the attention module in the face key point detection model to obtain respective significant feature maps;

[0097] The loss calculation module 130 is used to calculate the total loss value of the current training of the face key point detection model according to the first loss value between the salient feature maps of the teacher network and the student network, the second loss value between the late feature maps output by the teacher network and the student network, and the third loss value between the positioning prediction value and the true value of the face key point of the student network if the teacher network has been pre-trained;

[0098] The training judgment module 140 is used to input the next sample image to continue training when the total loss value of this training does not meet the preset conditions, and stop when the calculated total loss value meets the preset conditions to obtain the trained face key point detection model.

[0099] Furthermore, if the teacher network has not been pre-trained, when calculating the total loss value, the loss calculation module 130 is also used to calculate the total loss value of the facial key point detection model based on a first loss value between the significant feature maps of the teacher network and the student network, a second loss value between the late feature maps output by the teacher network and the student network, a third loss value between the positioning prediction value and the true value of the facial key points of the student network, and a fourth loss value between the positioning prediction value and the true value of the facial key points of the teacher network.

[0100] It can be understood that the facial key point detection model training device 100 of this embodiment corresponds to the facial key point detection model training method of the above embodiment, and the options in the above method are also applicable to this embodiment, so they will not be repeated here.

[0101] The present application also provides a readable storage medium for storing the computer program used in the above-mentioned terminal device 10.

[0102] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0104] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0105] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A facial key point detection model training method, characterized in that: include: Inputting sample images containing facial key point annotations into the student network and the teacher network of the facial key point detection model for feature extraction, respectively, to output corresponding intermediate feature maps and facial key point location prediction results; the intermediate feature maps include early feature maps and late feature maps; Performing dimensionality reduction processing on the early feature maps output by the teacher network and the student network through the attention module of the face key point detection model to obtain respective significant feature maps; If the teacher network has been pre-trained, the total loss value of the current training of the facial key point detection model is calculated according to a first loss value between the salient feature maps of the teacher network and the student network, a second loss value between the late feature maps output by the teacher network and the student network, and a third loss value between the positioning prediction value and the true value of the facial key point of the student network; When the total loss value of this training does not meet the preset condition, the next sample image is input to continue training until the calculated total loss value meets the preset condition, thereby obtaining a trained face key point detection model; Among them, the teacher network and the student network both include: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a maximum global pooling layer, a first fully connected layer and a second fully connected layer connected in sequence; the intermediate feature map includes an early feature map output by the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively, and a late feature map output by the first fully connected layer.

2. The facial key point detection model training method according to claim 1, characterized in that: Also includes: If the teacher network has not been pre-trained, the total loss value of the facial key point detection model is calculated based on a first loss value between the significant feature maps of the teacher network and the student network, a second loss value between the teacher network and the late feature maps output by the student network, a third loss value between the positioning prediction value and the true value of the facial key points of the student network, and a fourth loss value between the positioning prediction value and the true value of the facial key points of the teacher network.

3. The facial key point detection model training method according to claim 1, characterized in that: The dimensionality reduction process includes: The intermediate feature maps output by the teacher network and the student network are respectively averaged across all channels.

4. The method for training a facial key point detection model according to any one of claims 1 to 3, characterized in that: The second convolution layer includes a convolution kernel and a bottleneck module; the third convolution layer includes a convolution kernel and two identical bottleneck modules.

5. The facial key point detection model training method according to claim 2, characterized in that: The first loss value and the second loss value are both calculated using the Smooth L1 function as the loss function; The third loss value and the fourth loss value are both calculated using the Wing loss function as the loss function.

6. A method for detecting key points of a face, characterized in that: include: Acquire a target face image, and input the target face image into a trained face key point detection model to perform face key point detection, wherein the face key point detection model is pre-trained using the method described in any one of claims 1 to 5; Output the facial key point positioning prediction result corresponding to the target facial image.

7. A method for detecting a user's sleeping state, characterized in that: include: Using the method as claimed in claim 6 to detect key points of the user's face, and obtain the prediction result of the key points of the user's face; The facial key point positioning prediction result is analyzed according to a preset sleeping posture to determine whether the user has entered a sleeping state.

8. A facial key point detection model training device, characterized in that: include: A network processing module, used for inputting sample images containing facial key point annotations into the student network and the teacher network of the facial key point detection model for feature extraction, so as to output corresponding intermediate feature maps and facial key point location prediction results respectively; the intermediate feature maps include early feature maps and late feature maps; An attention module, used for performing dimensionality reduction processing on the early feature maps output by the teacher network and the student network through the attention module in the face key point detection model to obtain respective significant feature maps; A loss calculation module, for calculating the total loss value of the current training of the face key point detection model according to a first loss value between the salient feature maps of the teacher network and the student network, a second loss value between the late feature maps output by the teacher network and the student network, and a third loss value between the positioning prediction value and the true value of the face key point of the student network, if the teacher network has been pre-trained; A training judgment module is used to input the next sample image to continue training when the total loss value of this training does not meet the preset condition, and stop when the calculated total loss value meets the preset condition, thereby obtaining a trained face key point detection model; Among them, the teacher network and the student network both include: a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a maximum global pooling layer, a first fully connected layer and a second fully connected layer connected in sequence; the intermediate feature map includes an early feature map output by the first convolutional layer, the second convolutional layer, and the third convolutional layer, respectively, and a late feature map output by the first fully connected layer.

9. A terminal device, characterized in that: The terminal device comprises a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the method according to any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The device stores a computer program, which, when executed on a processor, implements the method according to any one of claims 1 to 7.

Images