Method, device, readable storage medium and terminal device for detecting key points of human face

Abstract

The present application belongs to the technical field of facial key point detection, and in particular, relates to a facial key point detection method, device, computer-readable storage medium and terminal device. The method comprises: obtaining a facial image to be detected; using a facial key point detection model to perform facial key point detection in the facial image to obtain a set of facial key points; wherein the facial key point detection model is used as a backbone network in the model training process, and the backbone network completes the model training process with the cooperation of a preset auxiliary network, and the auxiliary network is used to filter environmental information for the backbone network. In the present application, the facial key point detection model is used as the backbone network, and an auxiliary network is introduced on its basis to cooperate with it for model training, which can filter out environmental information irrelevant to the face as much as possible, so that the backbone network can focus more on information related to the face, thereby improving the accuracy of facial key point detection.

Description

Technical Field

[0001] The present application belongs to the technical field of facial key point detection, and in particular, relates to a facial key point detection method, device, computer-readable storage medium and terminal device. Background Art

[0002] Facial key point detection refers to locating the key points of a face, including eyebrows, eyes, nose, mouth, and facial contour areas, given a face image. This detection is the basis for other face-related tasks such as face recognition, expression analysis, and 3D face reconstruction.

[0003] In the prior art, although there are various mature facial key point detection models, since facial images often include more environmental information that is not related to the face, it will interfere with the detection of facial key points, thereby affecting the accuracy of facial key point detection. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a facial key point detection method, apparatus, computer-readable storage medium and terminal device to solve the problem that the existing facial key point detection method is easily interfered by environmental information unrelated to the face.

[0005] A first aspect of an embodiment of the present application provides a method for detecting key points of a face, which may include:

[0006] Obtain the face image to be detected;

[0007] Using a facial key point detection model to detect facial key points in the facial image, and obtain a facial key point set; wherein the facial key point detection model is used as a backbone network in the model training process, and the backbone network completes the model training process with the cooperation of a preset auxiliary network, and the auxiliary network is used to filter environmental information for the backbone network;

[0008] The model training process of the facial key point detection model includes: using the backbone network to perform facial key point detection on a preset sample face image to obtain an actual output of a face key point set; calculating a first training loss based on the actual output of the face key point set and an expected output of a preset face key point set; using the auxiliary network to perform facial key point detection on a preset mask face image to obtain a face key point set comparison output; wherein the mask face image is an image obtained after filtering the environmental information in the sample face image; calculating a third training loss based on the face key point set comparison output and the face key point set expected output; calculating a fourth training loss based on the actual output of the face key point set and the face key point set comparison output; and performing model training on the backbone network and the auxiliary network based on the first training loss, the third training loss and the fourth training loss.

[0009] In a specific implementation of the first aspect, performing model training on the backbone network and the auxiliary network according to the first training loss, the third training loss, and the fourth training loss may include:

[0010] Performing model training on the auxiliary network according to the third training loss;

[0011] Calculate a second superposition training loss according to the first training loss and the fourth training loss;

[0012] The backbone network is trained according to the second stacking training loss.

[0013] A second aspect of an embodiment of the present application provides a facial key point detection device, which may include:

[0014] A face image acquisition module, used to acquire a face image to be detected;

[0015] A detection model processing module, used to use a facial key point detection model to perform facial key point detection in the facial image to obtain a facial key point set; wherein the facial key point detection model is used as a backbone network in the model training process, and the backbone network completes the model training process with the cooperation of a preset auxiliary network, and the auxiliary network is used to filter environmental information for the backbone network;

[0016] A model training module is used to use the backbone network to perform facial key point detection on a preset sample face image to obtain an actual output of a face key point set; calculate a first training loss based on the actual output of the face key point set and an expected output of a preset face key point set; use the auxiliary network to perform facial key point detection on a preset mask face image to obtain a face key point set comparison output; wherein the mask face image is an image obtained after filtering the environmental information in the sample face image; calculate a third training loss based on the face key point set comparison output and the face key point set expected output; calculate a fourth training loss based on the actual output of the face key point set and the face key point set comparison output; and perform model training on the backbone network and the auxiliary network based on the first training loss, the third training loss, and the fourth training loss.

[0017] A third aspect of an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned facial key point detection methods are implemented.

[0018] The fourth aspect of an embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned facial key point detection methods when executing the computer program.

[0019] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the steps of any of the above-mentioned facial key point detection methods.

[0020] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain a face image to be detected; use a face key point detection model to detect face key points in the face image to obtain a face key point set; wherein the face key point detection model is used as a backbone network in the model training process, and the backbone network completes the model training process with the cooperation of a preset auxiliary network, and the auxiliary network is used to filter environmental information for the backbone network. In the embodiments of the present application, the face key point detection model is used as the backbone network, and an auxiliary network is introduced on its basis to cooperate with it for model training, which can filter out environmental information that is not related to the face as much as possible, so that the backbone network can focus more on information related to the face, thereby improving the accuracy of face key point detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A flow chart of an embodiment of a training process of a facial key point detection model;

[0023] Figure 2 A schematic diagram of the backbone network;

[0024] Figure 3 A schematic diagram of the overall network structure of the backbone network and the auxiliary network;

[0025] Figure 4 This is a schematic diagram of the distribution of key points of the face;

[0026] Figure 5 A schematic diagram showing that the backbone network and the auxiliary network share the same network structure;

[0027] Figure 6 A flowchart of another embodiment of a training process of a facial key point detection model;

[0028] Figure 7 This is a structural diagram of an embodiment of a facial key point detection device in an embodiment of the present application;

[0029] Figure 8 This is a schematic block diagram of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0032] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0033] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0034] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0035] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0036] In an embodiment of the present application, the facial key point detection model can be used as the backbone network, and an auxiliary network is introduced on its basis to cooperate with it for model training. Environmental information that is not related to the face can be filtered out as much as possible, so that the backbone network can focus more on information related to the face, thereby improving the accuracy of facial key point detection.

[0037] like Figure 1 As shown, in a specific implementation of the embodiment of the present application, the training process of the face key point detection model may include the following process:

[0038] Step S101: Use the backbone network to detect facial key points on a preset sample face image to obtain an actual output of a facial key point set.

[0039] The backbone network can select any neural network structure in the prior art according to the actual situation. Considering that the commonly used neural network structure has weak feature extraction ability and poor robustness when facing complex conditions, such as light disturbance and poor image quality, which may cause key point deviation. In a specific implementation of the embodiment of the present application, the following can be adopted: Figure 2 The backbone network shown in the figure. The backbone network adopts a lightweight network structure as shown in the following table:

[0040]

[0041]

[0042] Among them, Input is the input size of each layer, Operator is the data processing process of each layer, Conv is a 3*3 convolution layer, bottleneck is a bottleneck layer, GMP is a global maximum pooling layer, and Linear is a linear layer. The face image passes through Conv, stage1, stage2 and Conv in sequence, and feature data is extracted layer by layer, then processed by GMP, and finally passes through two linear layers for two fully connected processing processes, which are recorded as fea_fc and res_fc respectively. Stage 1 (stage1) includes a 3*3 convolution layer and a bottleneck layer, and stage 2 (stage2) includes a 3*3 convolution layer and two bottleneck layers. t is the transpose magnification factor of the bottleneck layer, c is the number of convolution kernel channels, n is the number of times the Operator is repeated, and s is the stride. The output of the face key point detection model is a set of 98 face key points. Since each key point consists of a horizontal coordinate and a vertical coordinate, it is 196-dimensional data.

[0043] Step S102: Calculate a first training loss according to the actual output of the face key point set and the expected output of the preset face key point set.

[0044] In an embodiment of the present application, annotation data corresponding to the sample facial image can be pre-set and used as the expected output of the facial key point set (i.e., groundtruth), and a preset first loss function is used to process the error between the actual output of the facial key point set and the expected output of the facial key point set to obtain a first training loss.

[0045] Among them, the first loss function can be any loss function in the prior art. Here, it is preferred to use a wingloss function that can better capture the key point error, so as to achieve more refined positioning of the key points.

[0046] Step S103: Use the auxiliary network to restore the facial contour of each layer of feature data of the backbone network to obtain the actual output of the facial contour.

[0047] Figure 3The figure shows the overall network structure of the backbone network and the auxiliary network. As shown in the figure, the last Conv of the backbone network outputs feature data of size 7×7×32. After being processed by the convolution layer (i.e. Conv_aux) in the auxiliary network, the output data size is 7×7×16. After being processed by unpooling (i.e. unpooling), the output data size is 14×14×16. It is concatenated with the feature data of size 14×14×16 output by stage2 of the backbone network to obtain concatenated data of size 14×14×32. After one convolution process, the concatenated data of 14×14×32 becomes 14×14×16. After another convolution process, the data size becomes 14×14×64. After being processed by unpooling, the output data size is 28×28×64. It is concatenated with the feature data of size 28×28×64 output by stage1 of the backbone network to obtain concatenated data of size 28×28×128. The 28×28×128 spliced ​​data undergoes one convolution process, and the data size becomes 28×28×64. After another convolution process, the data size becomes 28×28×64. After unpooling, the output data size is 56×56×64, which is spliced ​​with the feature data of size 56×56×64 output by the first Conv of the backbone network to obtain spliced ​​data of size 56×56×128. After a convolution process, the 56×56×128 spliced ​​data is processed once, and the data size becomes 56×56×64. After another convolution process, the data size becomes 56×56×64. After unpooling, the output data size is 112×112×64, and then it is spliced ​​with the sample face image convolution data with a data size of 112×112×64 to obtain spliced ​​data with a size of 112×112×128, where the sample face image convolution data is the sample face image (size 112×112×3) after the convolution layer in the auxiliary network. After a convolution process, the 112×112×128 spliced ​​data is processed once, and the data size becomes 112×112×64. After another convolution process, the result is the actual output of the face contour, and its data size is 112×112×2.

[0048] Step S104: Calculate a second training loss according to the actual output of the face contour and the preset expected output of the face contour.

[0049] The expected output of the face contour can be generated online from the expected output of the face key point set. Specifically, the face contour can be determined first according to the expected output of the face key point set. Figure 4The figure shows the distribution diagram of 98 facial key points. Key points 0 to 32 can be connected in sequence according to the serial numbers. In order to refine the drawing, Bezier curves can be used to connect the points. Since key points 0 to 32 do not form a closed loop, two additional points can be added here, namely the first additional point (p0_x, 0) and the second additional point (p32_x, 0), where p0_x is the horizontal coordinate of key point 0, and p32_x is the horizontal coordinate of key point 32. Key point 0 and the first additional point, the first additional point and the second additional point, and the second additional point and key point 32 are connected in sequence to form a closed loop figure with the connecting line from key point 0 to key point 32, that is, the outer contour of the face. Then, according to the outer contour of the face, a preset filling algorithm (filled) is used to set the pixels of the outer contour of the face and its interior to white pixels (pixel value is 1), and the pixels outside the outer contour of the face are set to black pixels (pixel value is 0), thereby generating the expected mask layer of the face contour, and forming the expected output of the face contour with the completely black background layer.

[0050] In an embodiment of the present application, a preset second loss function can be used to process the error between the actual output of the face contour and the expected output of the face contour to obtain a second training loss.

[0051] The second loss function can be any loss function in the prior art, for example, the celoss function or the diceloss function can be used, wherein the diceloss function converges faster, but the celoss function has a better final effect. This is because the diceloss function may focus more on the learning of the auxiliary network, and is not very direct in helping the backbone network.

[0052] Step S105: Perform model training on the backbone network and the auxiliary network according to the first training loss and the second training loss.

[0053] Specifically, the first superposition training loss can be calculated according to the first training loss and the second training loss. The calculation formula is as follows:

[0054] loss = wingloss + 10 * segloss

[0055] Among them, wingloss is the first training loss, segloss is the second training loss, and loss is the first superimposed training loss.

[0056] After obtaining the first superposition training loss, model training can be performed on the overall network structure of the backbone network and the auxiliary network according to the first superposition training loss.

[0057] After the model training is completed, the trained backbone network can be used as a face key point detection model. It should be noted that the auxiliary network is only used in the training process and no reasoning is performed in actual application. Therefore, the auxiliary network has no effect on the speed of actual application.

[0058] In the above training process, the reconstruction information may be partially obtained by the auxiliary network, while the backbone network may have incomplete information, and there may be an uncertainty problem in the intermediate process. In another specific implementation of the embodiment of the present application, Figure 5 As shown in FIG. 1 , the backbone network and the auxiliary network can share the same network structure. By drawing on the idea of ​​knowledge distillation, the learning process of the auxiliary network is used as a teacher, and the learning process of the backbone network is used as a student, so that the backbone network can directly obtain information related to the face as much as possible. In this way, the training process of the face key point detection model can include the following steps: Figure 6 The process shown:

[0059] Step S601: Use the backbone network to detect facial key points on a preset sample face image to obtain an actual output of a facial key point set.

[0060] Step S602: Calculate a first training loss according to the actual output of the face key point set and the expected output of the preset face key point set.

[0061] Among them, step S601 can refer to the specific description in step S101, and step S602 can refer to the specific description in step S102, which will not be repeated here.

[0062] Step S603: Use the auxiliary network to detect facial key points on the preset masked face image to obtain a facial key point set comparison output.

[0063] The masked face image is an image obtained after filtering the environmental information in the sample face image. Specifically, first, the expected mask layer of the face contour can be generated in the manner in step S104, and then the mask layer and the sample face image are ANDed pixel by pixel, that is, the face contour and the pixels inside the sample face image are kept unchanged, and the other pixels are set to black pixels, so as to obtain the masked face image.

[0064] Step S604: Calculate the third training loss based on the comparison output of the face key point set and the expected output of the face key point set.

[0065] The calculation process of the third training loss is similar to the calculation process of the first training loss. For details, please refer to the specific description in step S102, which will not be repeated here.

[0066] Step S605: Calculate a fourth training loss based on the actual output of the face key point set and the comparison output of the face key point set.

[0067] In an embodiment of the present application, a preset third loss function can be used to process the error between the actual output of the face key point set and the comparison output of the face key point set to obtain a fourth training loss (denoted as fealoss).

[0068] The third loss function may be any loss function in the prior art, for example, a smooth L1 function may be used.

[0069] Step S606: Perform model training on the backbone network and the auxiliary network according to the first training loss, the third training loss, and the fourth training loss.

[0070] Specifically, for the auxiliary network, model training can be performed on it according to the third training loss. For the backbone network, the second superimposed training loss can be calculated according to the first training loss and the fourth training loss. For example, the sum of the first training loss and the fourth training loss can be used as the second superimposed training loss, and then the backbone network can be model trained according to the second superimposed training loss.

[0071] In the above process, masked face images are used for auxiliary supervised learning. At the same time, the masked face images and sample face images are sent to the same network shared by the auxiliary network and the main network for learning. The input of the auxiliary network is clean and directly linked to the face. Features directly related to the face are extracted to predict key points. The input of the main network carries environmental information, and features that may be mixed with environmental information are extracted to predict key points. By minimizing the fourth training loss that reflects the difference between the two, the main network can be forced to learn features similar to those of the auxiliary network, that is, it can learn information directly linked to the face and remove environmental interference.

[0072] It should be noted that during the training process, the auxiliary network and the main network are trained together and share the same network. Only wingloss is used in the training of the auxiliary network, because its input is pure face, the convergence effect is better. Wingloss+fealoss is used in the training of the main network, forcing the network to extract information related to the face and help the key points converge better. After the model training is completed, the trained main network can be used as a face key point detection model.

[0073] In passing Figure 2 or Figure 5 After the face key point detection model is trained by the process shown, it can be used to perform the corresponding face key point detection task.

[0074] Specifically, a face image to be detected may be first obtained, and then a face key point detection model may be used to perform face key point detection in the face image, thereby obtaining a face key point set.

[0075] In summary, the embodiment of the present application obtains a face image to be detected; uses a face key point detection model to detect face key points in the face image to obtain a face key point set; wherein the face key point detection model serves as a backbone network during the model training process, and the backbone network completes the model training process with the cooperation of a preset auxiliary network, and the auxiliary network is used to filter environmental information for the backbone network. In the embodiment of the present application, the face key point detection model is used as the backbone network, and an auxiliary network is introduced on its basis to cooperate with it for model training, which can filter out environmental information that is not related to the face as much as possible, so that the backbone network can focus more on information related to the face, thereby improving the accuracy of face key point detection.

[0076] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0077] Corresponding to the method for detecting key points of a face described in the above embodiment, Figure 7 A structural diagram of an embodiment of a facial key point detection device provided in an embodiment of the present application is shown.

[0078] In this embodiment, a facial key point detection device may include:

[0079] The face image acquisition module 701 is used to acquire the face image to be detected;

[0080] The detection model processing module 702 is used to use the face key point detection model to perform face key point detection in the face image to obtain a face key point set; wherein the face key point detection model is used as the backbone network in the model training process, and the backbone network completes the model training process with the cooperation of a preset auxiliary network, and the auxiliary network is used to filter environmental information for the backbone network.

[0081] In a specific implementation of the embodiment of the present application, the facial key point detection device may further include a first model training module, and the first model training module may include:

[0082] A backbone network processing submodule, used to use the backbone network to perform facial key point detection on a preset sample face image to obtain an actual output of a facial key point set;

[0083] A first training loss calculation submodule, used to calculate a first training loss according to an actual output of the face key point set and an expected output of a preset face key point set;

[0084] An auxiliary network processing submodule, used to use the auxiliary network to restore the facial contour of each layer of feature data of the backbone network to obtain an actual output of the facial contour;

[0085] A second training loss calculation submodule, used for calculating a second training loss according to the actual output of the face contour and a preset expected output of the face contour;

[0086] The first model training submodule is used to perform model training on the backbone network and the auxiliary network according to the first training loss and the second training loss.

[0087] In a specific implementation of the embodiment of the present application, the first model training module may also include:

[0088] A face contour determination submodule, used to determine the face contour according to the expected output of the face key point set;

[0089] The face contour expected output generating submodule is used to generate the face contour expected output according to the face outer contour.

[0090] In a specific implementation of the embodiment of the present application, the first model training submodule may include:

[0091] A first superposition training loss calculation unit, configured to calculate a first superposition training loss according to the first training loss and the second training loss;

[0092] The first model training unit is used to perform model training on the backbone network and the auxiliary network according to the first superposition training loss.

[0093] In a specific implementation of the embodiment of the present application, the first superposition training loss calculation unit may be specifically configured to calculate the first superposition training loss according to the following formula:

[0094] loss = wingloss + 10 * segloss

[0095] Among them, wingloss is the first training loss, segloss is the second training loss, and loss is the first superposition training loss.

[0096] In a specific implementation of the embodiment of the present application, the facial key point detection device may further include a second model training module, and the second model training module may include:

[0097] A backbone network processing submodule, used to use the backbone network to perform facial key point detection on a preset sample face image to obtain an actual output of a facial key point set;

[0098] A first training loss calculation submodule, used to calculate a first training loss according to an actual output of the face key point set and an expected output of a preset face key point set;

[0099] The masked face image processing submodule is used to use the auxiliary network to perform face key point detection on the preset masked face image to obtain a face key point set comparison output; wherein the masked face image is an image obtained after filtering the environmental information in the sample face image;

[0100] A third training loss calculation submodule, used to calculate a third training loss according to a comparison output of the face key point set and an expected output of the face key point set;

[0101] A fourth training loss calculation submodule, configured to calculate a fourth training loss according to an actual output of the face key point set and a comparison output of the face key point set;

[0102] The second model training submodule is used to perform model training on the backbone network and the auxiliary network according to the first training loss, the third training loss and the fourth training loss.

[0103] In a specific implementation of the embodiment of the present application, the first model training submodule may include:

[0104] A second model training unit, configured to perform model training on the auxiliary network according to the third training loss;

[0105] A second superposition training loss calculation unit, used to calculate a second superposition training loss according to the first training loss and the fourth training loss;

[0106] The third model training unit is used to perform model training on the backbone network according to the second superposition training loss.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] Figure 8A schematic block diagram of a terminal device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0110] like Figure 8 As shown, the terminal device 8 of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps in the above-mentioned embodiments of the method for detecting key points of a face are implemented. Alternatively, when the processor 80 executes the computer program 82, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0111] Exemplarily, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 82 in the terminal device 8.

[0112] The terminal device 8 may be a computing device such as a mobile phone, a tablet computer, a desktop computer, a notebook, a PDA, a robot, etc. Those skilled in the art will appreciate that Figure 8 It is only an example of the terminal device 8 and does not constitute a limitation of the terminal device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device 8 may also include input and output devices, network access devices, buses, etc.

[0113] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0114] The memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. The memory 81 may also be an external storage device of the terminal device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 8. Further, the memory 81 may also include both an internal storage unit of the terminal device 8 and an external storage device. The memory 81 is used to store the computer program and other programs and data required by the terminal device 8. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0115] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0116] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0122] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for detecting key points of a face, characterized in that: include: Obtain the face image to be detected; Using a facial key point detection model to detect facial key points in the facial image, and obtain a facial key point set; wherein the facial key point detection model is used as a backbone network in the model training process, and the backbone network completes the model training process with the cooperation of a preset auxiliary network, and the auxiliary network is used to filter environmental information for the backbone network; The model training process of the facial key point detection model includes: using the backbone network to perform facial key point detection on a preset sample face image to obtain an actual output of a face key point set; calculating a first training loss based on the actual output of the face key point set and an expected output of a preset face key point set; using the auxiliary network to perform facial key point detection on a preset mask face image to obtain a face key point set comparison output; wherein the mask face image is an image obtained after filtering the environmental information in the sample face image; calculating a third training loss based on the face key point set comparison output and the face key point set expected output; calculating a fourth training loss based on the actual output of the face key point set and the face key point set comparison output; and performing model training on the backbone network and the auxiliary network based on the first training loss, the third training loss and the fourth training loss.

2. The method for detecting facial key points according to claim 1, characterized in that: The performing model training on the backbone network and the auxiliary network according to the first training loss, the third training loss and the fourth training loss includes: Performing model training on the auxiliary network according to the third training loss; Calculate a second superposition training loss according to the first training loss and the fourth training loss; The backbone network is trained according to the second stacking training loss.

3. A facial key point detection device, characterized in that: include: A face image acquisition module, used to acquire a face image to be detected; A detection model processing module, used to use a facial key point detection model to perform facial key point detection in the facial image to obtain a facial key point set; wherein the facial key point detection model is used as a backbone network in the model training process, and the backbone network completes the model training process with the cooperation of a preset auxiliary network, and the auxiliary network is used to filter environmental information for the backbone network; A model training module is used to use the backbone network to perform facial key point detection on a preset sample face image to obtain an actual output of a face key point set; calculate a first training loss based on the actual output of the face key point set and an expected output of a preset face key point set; use the auxiliary network to perform facial key point detection on a preset mask face image to obtain a face key point set comparison output; wherein the mask face image is an image obtained after filtering the environmental information in the sample face image; calculate a third training loss based on the face key point set comparison output and the face key point set expected output; calculate a fourth training loss based on the actual output of the face key point set and the face key point set comparison output; and perform model training on the backbone network and the auxiliary network based on the first training loss, the third training loss, and the fourth training loss.

4. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the facial key point detection method according to any one of claims 1 to 2 are implemented.

5. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the facial key point detection method according to any one of claims 1 to 2 are implemented.

Images