Face key point positioning method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a face key point positioning method and device, electronic equipment and a storage medium, and relates to the technical field of image processing. Obtaining a texture feature map of the to-be-positioned face image, and thermodynamic diagram coordinates and uncertainty of each face key point; for each face key point, determining an initial key point coordinate corresponding to the thermodynamic diagram coordinate in the to-be-positioned face image, and determining a dynamic adjustment size based on uncertainty; in the texture feature map, determining target mapping coordinates corresponding to the initial key point coordinates, and determining a target texture feature region; based on a preset corresponding relation, determining an initial deviation value corresponding to the target texture feature region and the target mapping coordinate, and determining a target deviation value; and determining a target key point coordinate after the initial key point coordinate is adjusted according to the target deviation value, and taking the target key point coordinate as a target coordinate in the to-be-positioned face image. By applying the scheme provided by the embodiment of the invention, the accuracy of face key point positioning can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, electronic device and storage medium for locating key points of a face. Background Art

[0002] With the rapid development of face recognition technology, face recognition has been widely used in related technical fields such as monitoring, payment, access control, entertainment, etc., providing more convenience for users. Among them, face recognition is achieved by locating the key points of each face.

[0003] In the related art, the location of facial key points can be achieved by using heat map prediction. Among them, the step of using heat map prediction to locate facial key points includes: obtaining a face image to be identified of a preset size, and sequentially performing texture feature extraction, downsampling and depth feature extraction on the face image to be identified to obtain a depth feature map of the face image to be identified. Afterwards, using the depth feature map, the heat map of each facial key point in the face image to be identified is determined respectively, and then the coordinates of each facial key point in the heat map can be determined, and the determined coordinates are called the heat map coordinates of the face key point. Then, according to the image size of the heat map and the image size of the image to be identified, the downsampling ratio is determined, and according to the determined downsampling ratio, the heat map coordinates of each facial key point are mapped to the face image to be identified, so that each facial key point can be located in the above-mentioned face image to be identified.

[0004] However, in the related art, since the heat map of each key point is obtained after downsampling the face image to be identified, the high-resolution detail features in the face image to be identified will be lost in the process of obtaining the heat map, resulting in low accuracy of the heat map coordinates determined by the heat map. Furthermore, when the above-mentioned heat map coordinates are used to locate the face key points in the face image to be identified, the accuracy of each face key point obtained is low. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for locating key points of a face, so as to improve the accuracy of locating key points of a face. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for locating key points of a face, the method comprising:

[0007] Obtain a texture feature map of the face image to be located, and the heat map coordinates and uncertainty of each face key point in the face image to be located; wherein the uncertainty of each face key point is used to characterize the accuracy of the heat map coordinates of the face key point;

[0008] For each facial key point, perform the following steps:

[0009] In the face image to be located, determining the initial key point coordinates corresponding to the heat map coordinates of the face key points;

[0010] Based on the uncertainty of the facial key point, determine the dynamic adjustment size corresponding to the facial key point in the texture feature map, and based on the mapping relationship between the face image to be located and the texture feature map, determine the target mapping coordinates of the initial key point coordinates in the texture feature map;

[0011] In the texture feature map, determining a target texture feature region centered on the target mapping coordinates according to the dynamic resizing;

[0012] Determine an initial offset value corresponding to the target texture feature region and the target mapping coordinates based on a preset correspondence relationship among the texture feature region, the mapping coordinates, and the mapping coordinate offset value, and determine a target offset value corresponding to the initial offset value based on the mapping relationship;

[0013] Determine the target key point coordinates after the initial key point coordinates are adjusted according to the target offset value as the target coordinates of the facial key point in the face image to be located.

[0014] Optionally, in a specific implementation, the step of obtaining the thermal map coordinates and uncertainty of each facial key point in the facial image to be located includes:

[0015] Inputting the face image to be located into a preset key point prediction model; wherein the key point prediction model includes a heat map sub-model and an uncertainty sub-model;

[0016] Obtaining the heat map coordinates of each facial key point output by the heat map sub-model and the uncertainty of each facial key point output by the uncertainty sub-model;

[0017] Among them, the key point prediction model is obtained by training a preset first sub-model based on a first sample face image and the sample heat map coordinates of each face key point in the first sample face image, and training a preset second sub-model based on the model prediction difference between a second sample face image and the second sample face image; the model prediction difference is: the difference between the prediction result of the heat map coordinate prediction of the second sample face image using the first sub-model and the sample heat map coordinates of each face key point in the second sample face image; the first sample face image and the second sample face image are different images.

[0018] Optionally, in a specific implementation, the training method of the key point prediction model includes:

[0019] Taking a first sample face image as input and taking sample heat map coordinates of each face key point in the first sample face image as output, training a preset first sub-model;

[0020] When the training of the first sub-model meets the specified training conditions, the second sample face image is used as input, and the model prediction difference of the second sample face image is used as output to train the preset second sub-model;

[0021] If the loss function of the second sub-model satisfies the first preset stop condition, the training is stopped to obtain a key point prediction model including the trained heat map sub-model and the uncertainty sub-model; wherein the heat map sub-model is the trained first sub-model, and the uncertainty sub-model is the trained second sub-model;

[0022] Otherwise, return to the step of training the preset first sub-model with the first sample face image as input and the sample heat map coordinates of each face key point in the first sample face image as output.

[0023] Optionally, in a specific implementation, the determining the initial offset value corresponding to the target texture feature area and the target mapping coordinate based on the correspondence between the preset texture feature area, the mapping coordinate and the mapping coordinate offset value includes:

[0024] Inputting the target texture feature area and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face, and obtaining an offset value output by the offset value prediction model as an initial offset value;

[0025] Among them, the offset value prediction model is determined based on the sample texture feature area and the label of the sample texture feature area, and the sample texture feature area is: the predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the texture feature area determined in the sample texture feature map of the third sample face image; the label of the sample texture feature area is: the offset value of the sample mapping coordinates and the true value of the mapping coordinates of each facial key point in the third sample face image in the sample texture feature image; the sample mapping coordinates are the mapping coordinates of the sample initial key point coordinates determined based on the mapping relationship between the third sample face image and the sample texture feature map; the sample initial key point coordinates are the predicted heat map coordinates of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the corresponding coordinates in the third sample face image.

[0026] Optionally, in a specific implementation, the training method of the offset value prediction model includes:

[0027] Obtaining predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample facial image with the key point prediction model, and obtaining a sample texture feature map of the third sample facial image;

[0028] For each facial key point, in the third sample facial image, determine the sample initial key point coordinates corresponding to the predicted heat map coordinates of the facial key point, and based on the mapping relationship between the third sample facial image and the sample texture feature map, determine the sample mapping coordinates corresponding to the sample initial key point coordinates in the sample texture feature map;

[0029] For each facial key point, based on the prediction uncertainty of the facial key point, determine the sample dynamic adjustment size corresponding to the facial key point in the sample texture feature map, and determine, in the sample texture feature map, a sample texture feature area centered on the sample mapping coordinates according to the sample dynamic adjustment size;

[0030] For each facial key point, calculating an offset value between the sample mapping coordinates and the true value of the mapping coordinates of the facial key point in the sample texture feature map as a label of the sample texture feature area;

[0031] Training a preset initial model with the sample texture feature region as output and the label of the sample texture feature region as output;

[0032] When the initial model meets the second preset stop condition, the training is stopped to obtain the offset value prediction model.

[0033] Optionally, in a specific implementation, determining the dynamic adjustment size corresponding to the facial key point in the texture feature map based on the uncertainty of the facial key point includes:

[0034] Based on the uncertainty of the facial key point and the sample size of a preset offset value prediction model, a dynamic adjustment size corresponding to the facial key point in the texture feature map is calculated.

[0035] Optionally, in a specific implementation, before inputting the target texture feature area and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face, the method further includes:

[0036] According to a preset sample size of the offset value prediction model, adjusting the area size of the target texture feature area;

[0037] The step of inputting the target texture feature area and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face includes:

[0038] The target texture feature region after the region size adjustment and the target mapping coordinates are input into a preset offset value prediction model corresponding to the key points of the human face.

[0039] Optionally, in a specific implementation, the calculating the dynamic adjustment size corresponding to the facial key point in the texture feature map based on the uncertainty of the facial key point and the sample size of the preset offset value prediction model includes:

[0040] The dynamic adjustment size corresponding to the key point of the face in the texture feature map is calculated using a preset formula; wherein the preset formula is:

[0041]

[0042] Among them, ds is the dynamically adjusted size corresponding to the facial key point in the texture feature map; u is the uncertainty of the heat map coordinates of the facial key point; sigmoid(u) is a preset nonlinear function related to uncertainty; k is the sample size of the offset value prediction model.

[0043] In a second aspect, an embodiment of the present application provides a facial key point positioning device, the device comprising:

[0044] A heat map coordinate acquisition module is used to acquire a texture feature map of a face image to be located, and heat map coordinates and uncertainties of each face key point in the face image to be located; wherein the uncertainty of each face key point is used to characterize the accuracy of the heat map coordinates of the face key point;

[0045] A key point coordinate determination module, for determining, for each facial key point, the initial key point coordinates corresponding to the heat map coordinates of the facial key point in the face image to be located;

[0046] A size determination module is used to determine, for each facial key point, the corresponding dynamic adjustment size of the facial key point in the texture feature map based on the uncertainty of the facial key point, and determine the target mapping coordinates of the initial key point coordinates in the texture feature map based on the mapping relationship between the face image to be located and the texture feature map;

[0047] A region determination module, for determining, for each facial key point, in the texture feature map, a target texture feature region centered on the mapping coordinates according to the dynamic resizing;

[0048] An offset value determination module is used to determine, for each facial key point, an initial offset value corresponding to the target texture feature area and the target mapping coordinates based on a preset correspondence relationship between the texture feature area, the mapping coordinates and the mapping coordinate offset value, and determine a target offset value corresponding to the initial offset value based on the mapping relationship;

[0049] The coordinate determination module is used to determine, for each facial key point, the target key point coordinates after the initial key point coordinates are adjusted according to the target offset value as the target coordinates of the facial key point in the facial image to be located.

[0050] Optionally, in a specific implementation, the heat map coordinate acquisition module is specifically used to:

[0051] Inputting the face image to be located into a preset key point prediction model; wherein the key point prediction model includes a heat map sub-model and an uncertainty sub-model;

[0052] Obtaining the heat map coordinates of each facial key point output by the heat map sub-model and the uncertainty of each facial key point output by the uncertainty sub-model;

[0053] Among them, the key point prediction model is obtained by training a preset first sub-model based on a first sample face image and the sample heat map coordinates of each face key point in the first sample face image, and training a preset second sub-model based on the model prediction difference between a second sample face image and the second sample face image; the model prediction difference is: the difference between the prediction result of the heat map coordinate prediction of the second sample face image using the first sub-model and the sample heat map coordinates of each face key point in the second sample face image; the first sample face image and the second sample face image are different images.

[0054] Optionally, in a specific implementation, the device further includes a training module for a key point prediction model, and the training module for the key point prediction model is specifically used to:

[0055] Taking a first sample face image as input and taking sample heat map coordinates of each face key point in the first sample face image as output, training a preset first sub-model;

[0056] When the training of the first sub-model meets the specified training conditions, the second sample face image is used as input, and the model prediction difference of the second sample face image is used as output to train the preset second sub-model;

[0057] If the loss function of the second sub-model satisfies the first preset stop condition, the training is stopped to obtain a key point prediction model including the trained heat map sub-model and the uncertainty sub-model; wherein the heat map sub-model is the trained first sub-model, and the uncertainty sub-model is the trained second sub-model;

[0058] Otherwise, return to the step of training the preset first sub-model with the first sample face image as input and the sample heat map coordinates of each face key point in the first sample face image as output.

[0059] Optionally, in a specific implementation, the offset value determination module includes:

[0060] An input submodule, used for inputting the target texture feature region and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face, and obtaining the offset value output by the offset value prediction model as an initial offset value;

[0061] Among them, the offset value prediction model is determined based on the sample texture feature area and the label of the sample texture feature area, and the sample texture feature area is: the predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the texture feature area determined in the sample texture feature map of the third sample face image; the label of the sample texture feature area is: the offset value of the sample mapping coordinates and the true value of the mapping coordinates of each facial key point in the third sample face image in the sample texture feature image; the sample mapping coordinates are the mapping coordinates of the sample initial key point coordinates determined based on the mapping relationship between the third sample face image and the sample texture feature map; the sample initial key point coordinates are the predicted heat map coordinates of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the corresponding coordinates in the third sample face image.

[0062] Optionally, in a specific implementation, the device further includes a training module for an offset value prediction model, and the training module for the offset value prediction model is specifically used to:

[0063] Obtaining predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample facial image with the key point prediction model, and obtaining a sample texture feature map of the third sample facial image;

[0064] For each facial key point, in the third sample facial image, determine the sample initial key point coordinates corresponding to the predicted heat map coordinates of the facial key point, and based on the mapping relationship between the third sample facial image and the sample texture feature map, determine the sample mapping coordinates corresponding to the sample initial key point coordinates in the sample texture feature map;

[0065] For each facial key point, based on the prediction uncertainty of the facial key point, determine the sample dynamic adjustment size corresponding to the facial key point in the sample texture feature map, and determine, in the sample texture feature map, a sample texture feature area centered on the sample mapping coordinates according to the sample dynamic adjustment size;

[0066] For each facial key point, calculating an offset value between the sample mapping coordinates and the true value of the mapping coordinates of the facial key point in the sample texture feature map as a label of the sample texture feature area;

[0067] Training a preset initial model with the sample texture feature region as output and the label of the sample texture feature region as output;

[0068] When the initial model meets the second preset stop condition, the training is stopped to obtain the offset value prediction model.

[0069] Optionally, in a specific implementation, the size determination module includes:

[0070] The calculation submodule is used to calculate the dynamic adjustment size corresponding to the facial key point in the texture feature map based on the uncertainty of the facial key point and the sample size of the preset offset value prediction model.

[0071] Optionally, in a specific implementation, the device further includes:

[0072] A region size adjustment module, used for adjusting the region size of the target texture feature region according to a preset sample size of the offset value prediction model before inputting the target texture feature region and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face;

[0073] The input submodule is specifically used for:

[0074] The target texture feature region after the region size adjustment and the target mapping coordinates are input into a preset offset value prediction model corresponding to the key points of the human face.

[0075] Optionally, in a specific implementation, the computing submodule is specifically used to:

[0076] The dynamic adjustment size corresponding to the key point of the face in the texture feature map is calculated using a preset formula; wherein the preset formula is:

[0077]

[0078] Among them, ds is the dynamically adjusted size corresponding to the facial key point in the texture feature map; u is the uncertainty of the heat map coordinates of the facial key point; sigmoid(u) is a preset nonlinear function related to uncertainty; k is the sample size of the offset value prediction model.

[0079] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0080] Memory, used to store computer programs;

[0081] The processor is used to implement the steps of any of the above method embodiments when executing the program stored in the memory.

[0082] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-described methods is implemented.

[0083] In a fifth aspect, an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the methods described above.

[0084] Beneficial effects of the embodiments of the present application:

[0085] As can be seen from the above, by applying the solution provided by the embodiment of the present application, when locating each facial key point in the face image to be located, the texture feature map of the face image to be located, as well as the heat map coordinates of each facial key point in the face image to be located and the uncertainty used to characterize the accuracy of the heat map coordinates can be first obtained; then, for each facial key point, the following steps can be performed: first, in the face image to be located, the initial key point coordinates corresponding to the heat map coordinates of the face key point are determined; then, based on the uncertainty of the face key point, the dynamic adjustment size corresponding to the face key point in the texture feature map is determined, and based on the mapping relationship between the face image to be located and the texture feature map, Determine the target mapping coordinates of the initial key point coordinates in the texture feature map; then, in the texture feature map, determine the target texture feature area centered on the target mapping coordinates based on the above-mentioned dynamic adjustment size; then, based on the corresponding relationship between the preset texture feature area, mapping coordinates and mapping coordinate offset values, determine the initial offset value corresponding to the target texture feature area and the above-mentioned target mapping coordinates, and based on the above-mentioned mapping relationship, determine the target offset value corresponding to the above-mentioned initial offset value; then, it is possible to determine the target key point coordinates after the above-mentioned initial key point coordinates are adjusted according to the target offset value, and use the above-mentioned target key point coordinates as the target coordinates of the face key point in the face image to be located.

[0086] Based on this, by applying the solution provided by the embodiment of the present application, for each facial key point in the facial image to be located, the thermal map coordinates of the facial key point and the uncertainty of the thermal map coordinates can be used to determine the initial key point coordinates and target offset value of the facial key point in the facial image to be located, and then, the facial key point can be located in the facial image to be located using the above initial key point coordinates and target offset value. In this way, by using the target deviation value determined based on the uncertainty of the thermal map coordinates to adjust the initial key point coordinates determined based on the thermal map coordinates, the coordinate deviation caused by the use of low-precision thermal map coordinates for facial key point location can be corrected, thereby reducing the impact of factors such as the difference in image size between the thermal map and the facial image to be located, and the loss of high-resolution detail features in the process of calculating the thermal map on the accuracy of facial key point location, thereby improving the accuracy of facial key point location. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0088] Figure 1 A schematic diagram of a flow chart of a method for locating key points of a face provided in an embodiment of the present application;

[0089] Figure 2 A schematic diagram of the structure of a key point prediction model provided in an embodiment of the present application;

[0090] Figure 3 A flowchart of a training method for a key point prediction model provided in an embodiment of the present application;

[0091] Figure 4 A flowchart of a method for training an offset value prediction model provided in an embodiment of the present application;

[0092] Figure 5 A schematic diagram of a sample texture feature area provided in an embodiment of the present application;

[0093] Figure 6 A flowchart of a specific example of a method for locating key points of a face provided in an embodiment of the present application;

[0094] Figure 7 A schematic diagram of the structure of a facial key point positioning device provided in an embodiment of the present application;

[0095] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0097] In the related art, since the heat map of each key point is obtained after downsampling the face image to be identified, the high-resolution detail features in the face image to be identified will be lost in the process of obtaining the heat map, resulting in low accuracy of the heat map coordinates determined by the heat map. Furthermore, when the face key points are located in the face image to be identified using the above-mentioned heat map coordinates, the accuracy of each face key point obtained is low.

[0098] In order to solve the above technical problems, an embodiment of the present application provides a method for locating key points of a face.

[0099] Among them, this method is suitable for various application scenarios where there is a need to locate facial key points. For example, payment software recognizes facial key points in user facial images to realize facial payment; access control systems open access by recognizing irises in user facial images, etc.

[0100] The method can be applied to various types of image acquisition devices with data processing functions. For example, the image acquisition device is equipped with a data processing module. After acquiring the face image to be located, the method is executed using the data processing module carried by the device. The method can also be applied to various types of electronic devices that can communicate with the image acquisition device, such as mobile phones, tablet computers, laptop computers, etc. Moreover, when the execution subject of the method is an electronic device, the electronic device can be an independent electronic device or a device cluster composed of multiple electronic devices, hereinafter referred to as the electronic device.

[0101] Based on this, the embodiments of the present application do not specifically limit the application scenarios and execution entities of the method.

[0102] It should be noted that in the technical solution of this application, the operations involved in obtaining, storing, using, processing, transmitting, providing and disclosing user personal information are all carried out with the user's authorization.

[0103] It should be noted that the head image in this embodiment is not a head image of a specific user and cannot reflect the personal information of a specific user.

[0104] It should be noted that the face images in this embodiment come from a public dataset.

[0105] A method for locating key points of a face provided in an embodiment of the present application may include the following steps:

[0106] Obtain a texture feature map of the face image to be located, and the heat map coordinates and uncertainty of each face key point in the face image to be located; wherein the uncertainty of each face key point is used to characterize the accuracy of the heat map coordinates of the face key point;

[0107] For each facial key point, perform the following steps:

[0108] In the face image to be located, determining the initial key point coordinates corresponding to the heat map coordinates of the face key points;

[0109] Based on the uncertainty of the facial key point, determine the dynamic adjustment size corresponding to the facial key point in the texture feature map, and based on the mapping relationship between the face image to be located and the texture feature map, determine the target mapping coordinates of the initial key point coordinates in the texture feature map;

[0110] In the texture feature map, determining a target texture feature region centered on the mapping coordinates according to the dynamic resizing;

[0111] Determine an initial offset value corresponding to the target texture feature region and the target mapping coordinates based on a preset correspondence relationship among the texture feature region, the mapping coordinates, and the mapping coordinate offset value, and determine a target offset value corresponding to the initial offset value based on the mapping relationship;

[0112] Determine the target key point coordinates after the initial key point coordinates are adjusted according to the target offset value as the target coordinates of the facial key point in the face image to be located.

[0113] As can be seen from the above, by applying the solution provided by the embodiment of the present application, when locating each facial key point in the face image to be located, the texture feature map of the face image to be located, as well as the heat map coordinates of each facial key point in the face image to be located and the uncertainty used to characterize the accuracy of the heat map coordinates can be first obtained; then, for each facial key point, the following steps can be performed: first, in the face image to be located, the initial key point coordinates corresponding to the heat map coordinates of the face key point are determined; then, based on the uncertainty of the face key point, the dynamic adjustment size corresponding to the face key point in the texture feature map is determined, and based on the mapping relationship between the face image to be located and the texture feature map, Determine the target mapping coordinates of the initial key point coordinates in the texture feature map; then, in the texture feature map, determine the target texture feature area centered on the target mapping coordinates based on the above-mentioned dynamic adjustment size; then, based on the corresponding relationship between the preset texture feature area, mapping coordinates and mapping coordinate offset values, determine the initial offset value corresponding to the target texture feature area and the above-mentioned target mapping coordinates, and based on the above-mentioned mapping relationship, determine the target offset value corresponding to the above-mentioned initial offset value; then, it is possible to determine the target key point coordinates after the above-mentioned initial key point coordinates are adjusted according to the target offset value, and use the above-mentioned target key point coordinates as the target coordinates of the face key point in the face image to be located.

[0114] Based on this, by applying the solution provided by the embodiment of the present application, for each facial key point in the facial image to be located, the thermal map coordinates of the facial key point and the uncertainty of the thermal map coordinates can be used to determine the initial key point coordinates and target offset value of the facial key point in the facial image to be located, and then, the facial key point can be located in the facial image to be located using the above initial key point coordinates and target offset value. In this way, by using the target deviation value determined based on the uncertainty of the thermal map coordinates to adjust the initial key point coordinates determined based on the thermal map coordinates, the coordinate deviation caused by the use of low-precision thermal map coordinates for facial key point location can be corrected, thereby reducing the impact of factors such as the difference in image size between the thermal map and the facial image to be located, and the loss of high-resolution detail features in the process of calculating the thermal map on the accuracy of facial key point location, thereby improving the accuracy of facial key point location.

[0115] Below, in conjunction with the accompanying drawings, a method for locating key points of a face provided in an embodiment of the present application is specifically described.

[0116] Figure 1 A schematic diagram of a flow chart of a method for locating key points of a face provided in an embodiment of the present application, such as Figure 1 As shown, the method may include the following steps S101-S106.

[0117] S101: Acquire a texture feature map of a face image to be located, and heat map coordinates and uncertainties of each face key point in the face image to be located;

[0118] Among them, the uncertainty of each facial key point is used to characterize the accuracy of the thermal map coordinates of the facial key point;

[0119] When locating each facial key point in the facial image to be located, the texture feature map of the facial image to be located can be firstly obtained, and the heat map coordinates of each facial key point in the facial image to be located can be obtained.

[0120] The texture feature map of the face image to be located is an image obtained by performing texture extraction on the face image to be located.

[0121] Optionally, a preset texture feature extraction algorithm may be used to extract texture features of the above-mentioned face image to be located, thereby obtaining a texture feature map of the face image to be located, so that the image size of the obtained texture feature map is the same as the image size of the face image to be located;

[0122] Optionally, the above-mentioned facial image to be located can be downsampled, and then a preset texture feature extraction algorithm is used to extract the texture features of the downsampled facial image to be located to obtain a texture feature map of the facial image to be located. In this way, the image size of the obtained texture feature map is smaller than the image size of the facial image to be located.

[0123] It can be seen that the image size of the texture feature map of the face image to be located and the image size of the face image to be located may be the same as or different from each other.

[0124] When the image size of the texture feature map of the face image to be located is the same as the image size of the face image to be located, for each pixel in the face image to be located, the pixel coordinates of the pixel in the face image to be located are the same as the pixel coordinates of the mapping point of the pixel in the texture feature map. For example, when the image size of the texture feature map of the face image to be located is the same as the image size of the face image to be located, the pixel coordinates of the pixel A in the face image to be located with the pixel coordinates (10, 10) are the same as the pixel coordinates of the mapping point A' in the texture feature map.

[0125] When the image size of the texture feature map of the face image to be located is different from the image size of the face image to be located, for each pixel point in the face image to be located, the pixel coordinates of the pixel point in the face image to be located are different from the pixel coordinates of the mapping point of the pixel point in the texture feature map. For example, when the image size of the face image to be located is 224*224, and the image size of the texture feature map of the face image to be located is 112*112, the pixel coordinates of the pixel point A in the face image to be located with the pixel coordinates (10, 10) are (5, 5) at the mapping point A' in the texture feature map.

[0126] Based on this, for each pixel of the face image to be located, the mapping point corresponding to the pixel in the texture feature map can be determined based on the proportional relationship between the image size of the face image to be located and the image size of the texture feature map. Therefore, in order to correspond the face image to be located and each pixel in the texture feature map, after obtaining the texture feature map of the face image to be located, the mapping relationship between the face image to be located and the texture feature map can be established.

[0127] The mapping relationship may include a proportional relationship between the image size of the face image to be located and the image size of the texture feature map, and a corresponding relationship between each pixel point in the face image to be located and each pixel point in the texture feature map.

[0128] In this way, for each pixel point of the face image to be located, the mapping point corresponding to the pixel point in the texture feature map can be determined based on the established mapping relationship.

[0129] In addition, in the face image to be located, the number of face key points to be located can be set according to actual needs, for example, 68, 72, 118, etc., which are all reasonable and are not specifically limited in the embodiments of the present application.

[0130] Since the heat map coordinates of each facial key point are the coordinates of the facial key point in the heat map corresponding to the facial key point. Generally, for each facial key point, in the process of obtaining the heat map of the facial key point based on the facial image to be located, there may be influencing factors such as loss of image feature data and calculation deviation in the facial image to be located, resulting in errors in the calculated heat map of the facial key point; and when determining the heat map coordinates of the facial key point in the heat map of the facial key point, there may also be influencing factors such as calculation error. Therefore, under the influence of multiple influencing factors, the accuracy of the heat map coordinates of the facial key point finally calculated is poor. In addition, for each facial key point, when using the heat map coordinates of the facial key point to locate the facial key point in the facial image to be located, the accuracy of the heat map coordinates will directly affect the accuracy of the positioning of the facial key point.

[0131] Based on this, for each facial key point, after obtaining the heat map coordinates of the facial key point, the accuracy of the heat map coordinates of the facial key point can be further determined.

[0132] That is to say, when locating facial key points of a face image to be located, for each facial key point in the face image to be located, after obtaining the heat map coordinates of the facial key point, the uncertainty of the accuracy of the heat map coordinates used to characterize the facial key point can be obtained.

[0133] In a specific implementation, the above step S101 may include the following steps 11-12:

[0134] Step 11: input the face image to be located into a preset key point prediction model; wherein the key point prediction model includes a heat map sub-model and an uncertainty sub-model;

[0135] Step 12: Obtain the heat map coordinates of each facial key point output by the heat map sub-model and the uncertainty of each facial key point output by the uncertainty sub-model;

[0136] Among them, the key point prediction model is obtained by training a preset first sub-model based on the first sample face image and the sample heat map coordinates of each face key point in the first sample face image, and training a preset second sub-model based on the second sample face image and the model prediction difference of the second sample face image; the model prediction difference is: the difference between the prediction result of the heat map coordinate prediction of the second sample face image using the first sub-model and the sample heat map coordinates of each face key point in the second sample face image; the first sample face image and the second sample face image are different images.

[0137] In this specific implementation, the first sample face image and the second sample face image can be used in advance to train a key point prediction model including a heat map sub-model and an uncertainty sub-model.

[0138] The key point prediction model can be obtained by training a preset first sub-model based on the first sample face image and the sample heat map coordinates of each face key point in the first sample face image, and training a preset second sub-model based on the second sample face image and the model prediction difference of the second sample face image; the model prediction difference is: the difference between the prediction result of the heat map coordinate prediction of the second sample face image using the first sub-model and the sample heat map coordinates of each face key point in the second sample face image; and the first sample face image and the second sample face image can be different images. For example, the first sample face image is the face of a young male A, and the second sample face image is the face of an elderly male B.

[0139] For the sake of clarity, a training method for a key point prediction model provided in an embodiment of the present application will be specifically introduced below.

[0140] After obtaining the face image to be located, the face image to be located can be input into a preset key point prediction model. In this way, the heat map sub-model in the key point prediction model can determine the heat map coordinates of each face key point in the face image to be located, and the uncertainty sub-model in the key point prediction model can determine the uncertainty of the accuracy of the heat map coordinates output by the heat map sub-model for characterizing each face key point.

[0141] Thus, the electronic device can obtain the thermal map coordinates of each facial key point output by the thermal map sub-model, and obtain the uncertainty of each thermal map coordinate output by the uncertainty sub-model.

[0142] For example, Figure 2As shown, after obtaining the face image to be located with an image size of 224*224, the face image to be located can be input into the key point prediction model, so that the key point prediction model can extract the texture features of the face image to be located, and then the texture feature image can be downsampled, and the depth feature can be extracted from the texture feature image after downsampling, thereby obtaining the depth feature map of the face image to be located. Then, the depth feature map of the face image to be located can be input into the heat map sub-model and uncertainty sub-model of the key point prediction model. Then, the heat map of the 68 face key points output by the heat map sub-model can be obtained to obtain a heat map of 68*56*56; and the uncertainty of the 68 heat map coordinate points output by the uncertainty sub-model can be obtained to obtain an uncertainty of 1*68. The above 68*56*56 heat map is a collection of 56*56 heat maps of each of the 68 facial key points, and the heat map coordinates of each facial key point are marked in the heat map of each facial key point; the above 1*68 uncertainty is a collection of the uncertainty of each of the 68 facial key points. Furthermore, for each facial key point, the heat map coordinates of the facial key point marked in the heat map of the facial key point can be obtained, and the uncertainty of the heat map coordinates of the facial key point can be obtained.

[0143] After obtaining the heat map coordinates and uncertainty of each facial key point in the face image to be located, the heat map coordinates and uncertainty of each facial key point can be used to locate the facial key point in the face image to be located.

[0144] In order to locate each facial key point in the face image to be located, the following steps S102-S106 may be performed:

[0145] S102: For each facial key point, in the face image to be located, determining the initial key point coordinates corresponding to the heat map coordinates of the facial key point;

[0146] For each facial key point, after determining the heat map coordinates of the facial key point, the heat map coordinates of the facial key point can be used to determine the initial key point coordinates corresponding to the heat map coordinates of the facial key point in the face image to be located.

[0147] Among them, we can first determine the image area corresponding to the heat map of the facial key point in the face image to be located, and then use the heat map coordinates of the facial key point to determine the image coordinates corresponding to the heat map coordinates in the determined image area as the initial key point coordinates corresponding to the heat map coordinates of the facial key point.

[0148] S103: for each facial key point, based on the uncertainty of the facial key point, determining the corresponding dynamic adjustment size of the facial key point in the texture feature map, and based on the mapping relationship between the face image to be located and the texture feature map, determining the target mapping coordinates of the initial key point coordinates in the texture feature map;

[0149] Usually, for each facial key point, the initial key point coordinates corresponding to the heat map coordinates of the facial key point in the face image to be located are the target coordinates of the facial key point in the face image to be located. However, as mentioned above, since there may be a variety of influencing factors in the process of determining the heat map coordinates of the facial key point, the accuracy of the determined heat map coordinates is poor under the influence of a variety of influencing factors. Therefore, after mapping the above heat map coordinates back to the face image to be located, the initial key point coordinates obtained deviate from the target coordinates of the facial key point in the face image to be located.

[0150] That is to say, for each facial key point, the initial key point coordinates corresponding to the heat map coordinates of the facial key point in the face image to be located can represent the approximate position of the facial key point in the face image to be located, and the target coordinates of the facial key point in the face image to be located are the coordinates in the face image to be located that have a certain coordinate offset from the above-mentioned initial key point coordinates.

[0151] As can be seen from the above, in order to determine the target coordinates of the facial key points in the face image to be located, the target offset value between the initial key point coordinates and the target coordinates can be determined first.

[0152] Since the image details included in the face image to be located may affect the determination process of the target offset value, resulting in poor accuracy of the determined target offset value, some image details can be eliminated in the process of extracting texture features from the feature map of the face to be located. Therefore, the texture feature map of the face image to be located can be used to determine the target offset value between the initial key point coordinates and the target coordinates.

[0153] Among them, since there is a mapping relationship between the texture feature map of the face image to be located and the face image to be located, the mapping relationship can be used to determine the initial key point coordinates and target coordinates of the face key points in the face image to be located, and the mapping coordinates in the texture feature map, and then the initial offset value between the mapping coordinates of the initial key point coordinates and the target coordinates in the texture feature map can be determined. And the obtained initial offset value and the above target offset value also have the above mapping relationship.

[0154] Based on this, for each facial key point, after determining the initial key point coordinates of the facial key point in the face image to be located, the above mapping relationship can be used to determine the target mapping coordinates corresponding to the above initial key point coordinates in the texture feature map.

[0155] Exemplarily, when the image size of the face image to be located is 224*224, and the image size of the texture feature map of the face image to be located is 112*112, if the initial key point coordinates are (50, 50), then the target mapping coordinates corresponding to the initial key point coordinates in the texture feature map are (25, 25); and when the image size of the face image to be located is 224*224, and the image size of the texture feature map of the face image to be located is 224*224, if the initial key point coordinates are (50, 50), then the target mapping coordinates corresponding to the initial key point coordinates in the texture feature map are (50, 50).

[0156] After determining the target mapping coordinates corresponding to the above-mentioned initial key point coordinates in the texture feature map, the initial offset values ​​of the above-mentioned target mapping coordinates and the mapping coordinates corresponding to the target coordinates in the texture feature map may be further calculated.

[0157] For each facial key point, in order to determine the above initial offset value, the target texture feature area of ​​the facial key point in the texture feature map can be first determined, and in order to determine the above target texture feature area, the dynamic adjustment size of the above target mapping coordinates can be first determined.

[0158] The dynamically adjusted size corresponding to the facial key point in the texture feature map may represent the size of the coordinate range of the target coordinates of the facial key point in the face image to be located in the texture feature map.

[0159] Usually, for each facial key point, when the accuracy of the heat map coordinates of the facial key point is poor, the offset value between the initial key point coordinates of the heat map coordinates in the face image to be located and the target coordinates of the facial key point in the face image to be located is large. Therefore, in the texture feature map, the larger the target texture feature area including the above-mentioned target mapping coordinates and the mapping coordinates corresponding to the target coordinates, that is, the larger the dynamic adjustment size of the target texture feature area of ​​the target mapping coordinates; and when the accuracy of the heat map coordinates of the facial key point is high, the offset value between the initial key point coordinates of the heat map coordinates in the face image to be located and the target coordinates of the facial key point in the face image to be located is small. Therefore, in the texture feature map, the smaller the target texture feature area including the above-mentioned target mapping coordinates and the mapping coordinates corresponding to the target coordinates, that is, the smaller the dynamic adjustment size of the target texture feature area of ​​the target mapping coordinates.

[0160] Among them, for each facial key point, the target texture feature area of ​​the target mapping coordinates corresponding to the initial key point coordinates of the facial key point in the texture feature map can be used as the target texture feature area corresponding to the facial key point in the texture feature map, and the dynamic adjustment size of the above target texture feature area can be used as the dynamic adjustment size corresponding to the facial key point in the texture feature map.

[0161] It can be seen from the above that the dynamic adjustment size corresponding to the facial key point in the texture feature map is related to the uncertainty of the thermal map coordinates of the facial key point.

[0162] Based on this, for each facial key point, the dynamic adjustment size of the facial key point in the texture feature map can be determined based on the uncertainty of the thermal map coordinates of the facial key point.

[0163] Optionally, the dynamically adjusted size corresponding to the uncertainty may be determined based on a preset relationship between the area size of the texture feature area and the uncertainty of the thermal map coordinates.

[0164] S104: for each facial key point, in the texture feature map, according to the dynamic adjustment of the size, determine the target texture feature area centered on the target mapping coordinates;

[0165] For each facial key point, after determining the target mapping coordinates corresponding to the initial key point coordinates of the facial key point in the face image to be located in the texture feature map, and after the dynamic adjustment of the size of the facial key point in the texture feature map, the target texture feature area centered on the target mapping coordinates can be determined in the texture feature map according to the dynamic adjustment of the size.

[0166] Optionally, a square area with the above-mentioned dynamically adjusted size as the side length and the target mapping coordinates as the center of the area can be determined in the texture feature map as the target texture feature area.

[0167] Optionally, a circular area with the above-mentioned dynamically adjusted size as the radius and the target mapping coordinate as the center may be determined in the texture feature map as the target texture feature area.

[0168] S105: for each facial key point, based on the correspondence between the preset texture feature region, the mapping coordinates and the mapping coordinate offset value, determining an initial offset value corresponding to the target texture feature region and the target mapping coordinate, and determining a target offset value corresponding to the initial offset value based on the mapping relationship;

[0169] For each facial key point, after determining the target texture feature area, an initial offset value between the target mapping coordinates and the mapping coordinates corresponding to the target coordinates of the facial key point in the face image to be located in the texture feature map can be further calculated.

[0170] Thus, for each facial key point, an initial offset value corresponding to the target texture feature region and the target mapping coordinates can be determined based on the correspondence between the preset texture feature region, the mapping coordinates and the mapping coordinate offset value.

[0171] Optionally, the correspondence between the texture feature area, the mapping coordinates and the mapping coordinate offset values ​​may be an offset value prediction model.

[0172] Based on this, optionally, in a specific implementation, the above step S105 may include the following step 21:

[0173] Step 21: input the target texture feature area and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face, and obtain the offset value output by the offset value prediction model as the initial offset value;

[0174] Among them, the offset value prediction model is determined based on the sample texture feature area and the label of the sample texture feature area. The sample texture feature area is: the predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the texture feature area determined in the sample texture feature map of the third sample face image; the label of the sample texture feature area is: the offset value of the sample mapping coordinates and the true value of the mapping coordinates of each facial key point in the third sample face image in the sample texture feature image; the sample mapping coordinates are the mapping coordinates of the sample initial key point coordinates determined based on the mapping relationship between the third sample face image and the sample texture feature map; the sample initial key point coordinates are the predicted heat map coordinates of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the corresponding coordinates in the third sample face image.

[0175] In this specific implementation, the offset value prediction model can be pre-trained based on the sample texture feature region and the label of the sample texture feature region. The above-mentioned sample texture feature region can be: the predicted heat map coordinates and prediction uncertainty of each face key point predicted by the key point prediction model for the third sample face image, and the texture feature region determined in the sample texture feature map of the third sample face image; the label of the sample texture feature region is: the sample mapping coordinates and the offset value of the true value of the mapping coordinates of each face key point in the third sample face image in the sample texture feature image; the sample mapping coordinates are the mapping coordinates of the sample initial key point coordinates determined based on the mapping relationship between the third sample face image and the sample texture feature map; the sample initial key point coordinates are the predicted heat map coordinates of each face key point predicted by the key point prediction model for the third sample face image, and the corresponding coordinates in the third sample face image.

[0176] For the sake of clarity, a training method for an offset value prediction model provided in an embodiment of the present application will be specifically introduced below.

[0177] For each facial key point, the key point types of different facial key points may be different, and there are certain differences in the offset value prediction rules between different key point types.

[0178] Based on this, for each key point type, an offset value prediction model belonging to each key point type can be trained separately. For example, an offset value prediction model for predicting the offset value of the eye, a key point of a human face, can be trained.

[0179] Thus, for each facial key point, after obtaining the target mapping coordinates of the initial key point coordinates of the facial key point in the face image to be located, and the target texture feature region of the target mapping coordinates, the target mapping coordinates and the target texture feature region can be input into the preset offset value prediction model corresponding to the facial key point. Then, the offset value output by the offset value prediction model can be obtained as the initial offset value of the target mapping coordinates of the initial key point coordinates of the target texture feature region and the facial key point in the face image to be located.

[0180] In addition, since the sizes and positions of facial key points corresponding to different key point types are different, and the positioning accuracy requirements for positioning different facial key points may also be different, based on this, the sample sizes supported by the offset value prediction model for different key point types are also different.

[0181] For example, the sample size of the offset value prediction model for the key facial point of the nose may be 28*28, while the sample size of the offset value prediction model for the key facial point of the forehead may be 56*56.

[0182] Therefore, in order to adapt to the sample size of the offset value prediction model and avoid introducing model prediction errors due to size mismatch, usually, the dynamic adjustment area to be input into the above offset value prediction model can be resized according to the sample size of the offset value prediction model.

[0183] Based on this, optionally, in the above step 21, before inputting the dynamic adjustment area and the initial key point coordinates into the preset offset value prediction model corresponding to the facial key point, the facial key point positioning method provided in the embodiment of the present application may further include the following step 20:

[0184] Step 20: adjusting the area size of the target texture feature area according to the sample size of the preset offset value prediction model;

[0185] Accordingly, the above step 21 may include the following step 211:

[0186] Step 211: inputting the target texture feature region after the region size adjustment and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face.

[0187] In this specific implementation, for each facial key point, a preset offset value prediction model corresponding to the facial key point can be determined, and a sample size of the offset value prediction model can be determined.

[0188] In this way, the area size of the determined target texture feature area can be adjusted according to the above sample size.

[0189] Afterwards, the adjusted target texture feature area and target mapping coordinates may be input into an offset value prediction model corresponding to the facial key point, and the offset value output by the offset value prediction model may be obtained.

[0190] Since the above-mentioned offset value is the offset value between the target mapping coordinates and the mapping coordinates of the target coordinates in the texture feature map in the target texture feature area after size adjustment; therefore, it is necessary to reversely adjust the above-mentioned offset value according to the size adjustment ratio of the above-mentioned target texture feature area, so that the offset value after adjustment is the initial offset value of the target texture feature area in the target texture feature area.

[0191] Exemplarily, when the area size of the target texture feature area of ​​the face key point in the texture feature map is 56*56, and the sample size of the offset value prediction model corresponding to the face key point is 28*28, the target texture feature area can be scaled by 4 times, thereby obtaining a target texture feature area with a scaled area size of 28*28. Afterwards, the target texture feature area with an area size of 28*28 is input into the offset value prediction model, and the offset value output by the offset value prediction model is obtained. Finally, the offset value is magnified by 4 times to obtain the initial offset value of the target texture feature area in the target texture feature area.

[0192] In addition, for each facial key point, the dynamic adjustment size of the above-mentioned target texture feature area can be determined based on the sample size of the offset value prediction model corresponding to the facial key point. In this way, the dynamic adjustment size of the sample size that meets the offset value prediction model can be obtained. Therefore, the determined target texture feature area is the area with the sample size that meets the offset value prediction model.

[0193] Based on this, optionally, the above step S103, based on the uncertainty of the facial key points, determines the dynamic adjustment size of the initial key point coordinates, and may include the following step 31:

[0194] Step 31: Based on the uncertainty of the facial key point and the sample size of the preset offset value prediction model, the dynamic adjustment size corresponding to the facial key point in the texture feature map is calculated.

[0195] In this specific implementation, for each facial key point, when determining the uncertainty of the facial key point and the preset sample size of the offset value prediction model corresponding to the facial key point, the dynamic adjustment size corresponding to the initial key point coordinates in the texture feature map can be calculated based on the above uncertainty and the above sample size.

[0196] Optionally, a preset correspondence between uncertainty, model sample size and region adjustment size may be used to determine a dynamic adjustment size corresponding to the uncertainty of the facial key points and the offset value prediction model sample size.

[0197] Optionally, in a specific implementation, step 31 may include the following step 311:

[0198] Step 311: Calculate the dynamic adjustment size corresponding to the key point of the face in the texture feature map using a preset formula;

[0199] Among them, the above preset formula is:

[0200]

[0201] Among them, ds is the dynamically adjusted size of the initial key point coordinates; k is the sample size of the offset value prediction model; u is the uncertainty of the heat map coordinates of the face key points; sigmoid(u) is a preset nonlinear function about uncertainty, and the nonlinear function can be expressed as:.

[0202]

[0203] In this specific implementation method, for each facial key point, when determining the uncertainty of the facial key point and the sample size of the preset offset value prediction model corresponding to the facial key point, the above uncertainty and sample size can be substituted into the preset formula to obtain the dynamically adjusted size of the facial key point in the texture feature map.

[0204] In this way, by using the uncertainty of the face key point and the sample size of the offset value prediction model corresponding to the face key point, the dynamic adjustment size of the face key point in the texture feature map is calculated, and the target texture feature area is calculated using the above dynamic adjustment size. The target texture feature area obtained is the area that meets the sample size of the offset value prediction model, so there is no need to adjust the area size of the target texture feature area before inputting the target texture feature area into the above offset value prediction model. In this way, the reduction in the resolution of the target texture feature area caused by scaling the target texture feature area can be avoided. In addition, the offset value output by the offset value prediction model is the initial offset value corresponding to the target texture feature area and the target mapping coordinates.

[0205] For each facial key point, since the initial offset value is the offset value between the target mapping coordinates corresponding to the initial key point coordinates of the facial key point in the face image to be located in the texture feature map and the mapping coordinates corresponding to the target coordinates of the facial key point in the face image to be located in the texture feature map, rather than the offset value between the initial key point coordinates and the target coordinates, it is impossible to use the initial offset value to directly adjust the initial key point coordinates to obtain the target coordinates.

[0206] Based on this, for each facial key point, after determining the above initial offset value, the target offset value corresponding to the above initial offset value can be determined by using the mapping relationship between the face image to be located and the texture feature map, so that the offset value between the above initial key point coordinates and the target coordinates can be obtained. For example, the image size of the face image to be located is 224*224, and the image size of the texture feature map is 112*112. After determining the initial offset value, the above initial offset value can be expanded by 2 times based on the mapping relationship between the face image to be located and the texture feature map, so as to obtain the target offset value corresponding to the initial offset value.

[0207] S106: For each facial key point, determine the target key point coordinates after the initial key point coordinates are adjusted according to the target offset value, as the target coordinates of the facial key point in the face image to be located.

[0208] For each facial key point, after determining the initial key point coordinates corresponding to the heat map coordinates of the facial key point in the face image to be located, and the target offset value between the above initial key point coordinates and the target coordinates of the facial key point in the face image to be located, the initial key point coordinates can be adjusted according to the above target offset value, and then the adjusted target key point coordinates are obtained. Therefore, the target key point coordinates are the target coordinates of the facial key point in the face image to be located.

[0209] As can be seen from the above, by applying the solution provided by the embodiment of the present application, for each facial key point in the facial image to be located, the thermal map coordinates of the facial key point and the uncertainty of the thermal map coordinates can be used to determine the initial key point coordinates and target offset value of the facial key point in the facial image to be located, and then, the facial key point can be located in the facial image to be located using the above initial key point coordinates and target offset value. In this way, by adjusting the initial key point coordinates determined based on the thermal map coordinates using the target deviation value determined based on the uncertainty of the thermal map coordinates, the coordinate deviation caused by the use of low-precision thermal map coordinates for facial key point location can be corrected, thereby reducing the impact of factors such as the image size difference between the thermal map and the facial image to be located, and the loss of high-resolution detail features in the process of calculating the thermal map on the accuracy of facial key point location, thereby improving the accuracy of facial key point location.

[0210] Below, the training method of the key point prediction model and the training method of the offset value prediction model involved in the embodiments of the present application are described in detail.

[0211] Optionally, in a specific implementation, such as Figure 3 As shown, the training method of the key point prediction model may include the following steps S301-S304:

[0212] S301: taking a first sample face image as input and taking sample heat map coordinates of each face key point in the first sample face image as output, training a preset first sub-model;

[0213] S302: When the training of the first sub-model meets the specified training conditions, the second sample face image is used as input, and the model prediction difference of the second sample face image is used as output to train the preset second sub-model;

[0214] S303: If the loss function of the second sub-model meets the preset stop condition, the training is stopped to obtain a key point prediction model including the trained heat map sub-model and the uncertainty sub-model;

[0215] Among them, the heat map sub-model is the first sub-model that has been trained, and the uncertainty sub-model is the second sub-model that has been trained;

[0216] S304: If the loss function of the second sub-model does not meet the preset stop condition, return to the step of training the preset first sub-model with the first sample face image as input and the sample heat map coordinates of each facial key point in the first sample face image as output.

[0217] In this specific implementation, when training the key point prediction model, you can first obtain multiple first sample face images, sample heat map coordinates of each face key point in each first sample face image, multiple second sample face images, and sample heat map coordinates of each face key point in each second sample face image.

[0218] Afterwards, the preset first sub-model can be trained with the first sample face image as input and the sample heat map coordinates of each face key point in the first sample face image as output. When the training of the first sub-model meets the specified training conditions, the training of the first sub-model can be suspended, and the second sample face image can be input into the first sub-model to obtain the prediction result of the heat map coordinate prediction of the second sample face image by the first sub-model. The prediction result is the predicted heat map coordinates of each face key point in the second sample face image.

[0219] Among them, the above-mentioned specified training conditions can be set according to actual needs, which can be that the training time reaches the specified time; or that the number of iterations of each first sample face image reaches the first preset number; or that the number of iterations of all first sample face images reaches the second preset number. These are all reasonable and are not specifically limited in the embodiments of the present application.

[0220] Then, for each second sample face image, the prediction results of each face key point in the second sample face image and the difference between the sample heat map coordinates of each face key point in the second sample face image are calculated as the model prediction difference of the second sample face image.

[0221] Next, the preset second sub-model may be trained by taking the second sample facial image as input and the model prediction difference of the second sample facial image as output.

[0222] Furthermore, it can be determined whether the loss function of the above-mentioned second sub-model satisfies the first preset stopping condition; wherein, the above-mentioned first preset stopping condition can be set according to actual needs, which can be that the result of the loss function of the above-mentioned second sub-model is not greater than the preset threshold, and the above-mentioned preset threshold can be 0.1 or 0.5, which are all reasonable and are not specifically limited in the embodiments of the present application.

[0223] If the loss function of the second sub-model satisfies the first preset stop condition, the training of both the first sub-model and the second sub-model can be stopped, thereby obtaining a key point prediction model including a heat map sub-model trained by the first sub-model and a key point prediction model of the uncertainty sub-model of the second sub-model.

[0224] If the loss function of the second sub-model satisfies the first preset stop condition, the method may return to step S301, and take the first sample face image as input, and take the sample heat map coordinates of each face key point in the first sample face image as output, to train the preset first sub-model, thereby continuing to train the first sub-model and the second sub-model, until the loss function of the second sub-model satisfies the first preset stop condition.

[0225] In this way, by alternately training the first sub-model and the second sub-model, the second sub-model can be used to verify the training results of the first sub-model without the need to use test samples to verify the training results of the first sub-model, thereby improving the efficiency of model training.

[0226] The uncertainty represents the confidence of the second sub-model for each prediction result predicted by the first sub-model. The higher the uncertainty, the lower the confidence of the second sub-model for the prediction result predicted by the first sub-model, and thus the error of the preset result of the first sub-model is greater.

[0227] The loss function Loss, as the target of model training optimization of the second sub-model, is correlated with the above-mentioned uncertainty.

[0228] When training the second sub-model, for the second sample face image x input into the second sub-model, let the prediction result of the first sub-model for the second sample face image be The standard deviation of the model prediction difference is σ, then the maximum likelihood estimate of the loss function of the second sub-model can be expressed as:

[0229]

[0230] Among them, p(y|x, θ) represents the probability of the observed result being y under the known condition x and the known parameter θ; specifically, it means the known model prediction result under the known condition and the standard deviation σ of the prediction results.

[0231] Assuming that the model prediction difference follows a Gaussian distribution, the above formula can be further transformed into:

[0232]

[0233] in, Indicates is the mean, is a Gaussian distribution with variance;

[0234] Since maximizing the likelihood estimation is equivalent to minimizing the loss, the loss function of the second sub-model can be further simplified as:

[0235]

[0236] use Instead, the loss function of the second sub-model is further transformed into:

[0237]

[0238] Thus, the uncertainty can be expressed as:

[0239]

[0240] It should be noted that the electronic device used for model training mentioned above and the electronic device for executing a method for locating facial key points provided in an embodiment of the present application may be the same electronic device or may be different electronic devices.

[0241] Optionally, in a specific implementation, such as Figure 4 As shown, the training method of the above offset value prediction model may include the following steps S401-S405:

[0242] S401: Obtain predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample facial image with a key point prediction model, and obtain a sample texture feature map of the third sample facial image;

[0243] S402: for each facial key point, in the third sample facial image, determine the sample initial key point coordinates corresponding to the predicted heat map coordinates of the facial key point, and based on the mapping relationship between the third sample facial image and the sample texture feature map, determine the sample mapping coordinates corresponding to the sample initial key point coordinates in the sample texture feature map;

[0244] S403: for each facial key point, based on the prediction uncertainty of the facial key point, determining the sample dynamic adjustment size corresponding to the facial key point in the sample texture feature map, and determining, in the sample texture feature map, a sample texture feature area centered on the sample mapping coordinate according to the sample dynamic adjustment size;

[0245] S404: For each facial key point, calculate the offset between the sample mapping coordinates and the true value of the mapping coordinates of the facial key point in the sample texture feature map as a label of the sample texture feature area;

[0246] S405: When the initial model meets the second preset stop condition, stop training to obtain an offset value prediction model.

[0247] In this specific implementation, after the key point prediction model is trained, the offset value prediction model can be trained using the key point prediction model. When training the offset value prediction model, the third sample face image, the sample texture feature map of the third sample face image, and the true value of the mapping coordinates of each face key point in the third sample face image in the sample texture feature map can be first obtained.

[0248] Afterwards, the third sample face image may be input into the key point prediction model, and the predicted heat map coordinates and prediction uncertainty of each face key point obtained by predicting the third sample face image with the key point prediction model may be obtained.

[0249] Then, for each facial key point, the sample initial key point coordinates corresponding to the predicted heat map coordinates of the facial key point can be determined in the third sample face image, and based on the mapping relationship between the above-mentioned third sample face image and the sample texture feature map, the sample mapping coordinates corresponding to the above-mentioned sample initial key point coordinates in the sample texture feature map can be determined.

[0250] Afterwards, for each facial key point, the sample dynamic adjustment size corresponding to the facial key point in the sample texture feature map can be determined based on the prediction uncertainty of the facial key point. Then, in the sample texture feature map, according to the above sample dynamic adjustment size, the sample texture feature area centered on the above sample mapping coordinates is determined, and the offset value between the sample mapping coordinates and the true value of the mapping coordinates of the facial key point in the sample texture feature map is calculated as the label of the sample texture feature area.

[0251] In this way, the preset initial model can be trained with the sample texture feature region as output and the label of the sample texture feature region as output.

[0252] Finally, when the initial model meets the second preset stop condition, the training of the initial model can be stopped to obtain the offset value prediction model.

[0253] Optionally, the second preset stop condition may be that the number of iterations of each sample texture feature region reaches a preset number.

[0254] Optionally, the second preset stop condition may be that the error between the true value and the predicted value of each sample texture feature region and the label of each sample texture feature region is less than a preset error.

[0255] In this way, after determining the target texture feature area and the target mapping coordinates, the above target texture feature area and the target mapping coordinates can be input into the offset value prediction model, so that the offset value prediction model learns the above target texture feature area and the target mapping coordinates, and outputs the initial offset value corresponding to the above target texture feature area and the above target mapping coordinates.

[0256] It should be noted that the electronic device used for model training mentioned above and the electronic device for executing a method for locating facial key points provided in an embodiment of the present application may be the same electronic device or may be different electronic devices.

[0257] In addition, for each key point type, an offset value prediction model for the key point type may be trained separately, thereby obtaining an offset value prediction model corresponding to each key point type.

[0258] Furthermore, in order to improve the clarity of local texture features of the sample texture feature region input into the above-mentioned offset value prediction model, the sample size of the sample texture feature region input into the offset value prediction model may be constrained.

[0259] Optionally, when the sample size of the sample texture feature region is too large, the range of the sample texture feature region is too large, which is easy to introduce non-local information and interfere with the model prediction results; when the sample size of the sample texture feature region is too small, the range of the sample texture feature region is too small and may not contain the required texture information around the true value point, so that the offset cannot be obtained. Based on this, the sample size of the sample texture feature region can be constrained.

[0260] Since the sample size of the sample texture feature region is calculated based on the uncertainty of the face key points, the purpose of constraining the sample size of the sample texture feature region can be achieved by constraining the uncertainty of the face key points.

[0261] For example, Figure 5As shown, for a certain facial key point, the offset value between the initial key point coordinate E of the thermal map coordinate of the facial key point in the facial image and the true value F of the coordinate of the facial key point is D. Therefore, through multiple experiments, it is found that when the sample size ds / 2=1.5D of the sample texture feature area, a better facial key point detection effect can be obtained. Then, the sample size ds / 2=1.5D of the sample texture feature area can be determined. The sample size ds=3D of the sample texture feature area can be determined. Then 510 in the figure is the determined sample texture feature area. 3D can be used as the constraint value of the sample size ds, and since there is a correlation between uncertainty and sample size, the purpose of constraining uncertainty u can be achieved by setting the constraint value of ds.

[0262] In order to facilitate understanding of a facial key point locating method provided in an embodiment of the present application, a specific example is provided below for explanation.

[0263] The face image to be located is a 224*224 face image, such as Figure 6 As shown, the process of locating facial key points of the face image to be located may include the following steps S601-S613:

[0264] S601: Input a face image to be located with an image size of 224*224;

[0265] S602: Extract texture features to obtain a C1*224*224 texture feature image;

[0266] S603: down-sampling the texture feature image;

[0267] S604: Extract depth features to obtain a C2*56*56 depth feature image;

[0268] S605: outputting a heat map;

[0269] S606: Output uncertainty;

[0270] S607: Determine the dynamic center;

[0271] S608: Determine dynamic dimensions using uncertainty;

[0272] S609: Determine the dynamic feature area;

[0273] S610: reduce the dynamic area;

[0274] S611: Determine an initial offset value;

[0275] S612: Determine a target offset value;

[0276] S613: Using the target offset value and the heat map coordinates, locate each facial key point in the face image to be located.

[0277] The input image size is 224*224 for the face image to be located. Then, a convolution layer with a step size of 1 is used to extract the texture features of the face image to be located to obtain a texture feature image of C1*224*224. The texture feature image is downsampled using multiple convolution layers and pooling layers.

[0278] Next, a convolutional layer with a step size of 2 is used to extract deep features from the downsampled texture feature image to obtain a C2*56*56 deep feature image.

[0279] Then, the above-mentioned deep feature image is input into a preset key point prediction model, and the 68*56*56 heat map and 1*68 uncertainty of the 68 facial key points of the face image to be located output by the above-mentioned key point prediction model are obtained.

[0280] Afterwards, for each facial key point, the heat map coordinates in the heat map of the facial key point are used to determine the initial key point coordinates of the facial key point in the face image to be located, and the mapping relationship between the above-mentioned face image to be located and the texture feature image is used to determine the target mapping coordinates of the above-mentioned initial key point coordinates in the texture feature image as the dynamic center.

[0281] Among them, since the image sizes of the above-mentioned face image to be located and the texture feature image are the same, the above-mentioned initial key point coordinates and the target mapping coordinates are the same.

[0282] For each facial key point, the uncertainty of the facial key point is used to determine the dynamic size of the facial key point in the texture feature image. Then, a dynamic area with the dynamic center as the center and the area size as the dynamic size can be determined in the texture feature image.

[0283] For each facial key point, according to the sample size of the offset value prediction model corresponding to the facial key point, the dynamic area is reduced by θ times to obtain the reduced dynamic area, and the dynamic area is input into the offset value prediction model to obtain the initial offset value output by the offset value prediction model.

[0284] For each facial key point, the initial offset value can be magnified by θ times, and the target offset value can be determined by using the mapping relationship between the facial image to be located and the texture feature image. Finally, the coordinates of the initial key point are adjusted according to the target offset value to obtain the target coordinates of the facial key point, and the obtained target coordinates are the coordinates of the facial key point in the facial image to be located. At this point, after determining the target coordinates of each facial key point in the facial image to be located, the positioning of each facial key point in the facial image to be located is completed.

[0285] Based on the same inventive concept, corresponding to the above-mentioned embodiment of the present application Figure 1 A method for locating key points of a face is shown, and an embodiment of the present application also provides a device for locating key points of a face.

[0286] Figure 7 A structural diagram of a facial key point positioning device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the device may include the following modules:

[0287] The heat map coordinate acquisition module 710 is used to acquire the texture feature map of the face image to be located, and the heat map coordinates and uncertainty of each face key point in the face image to be located; wherein the uncertainty of each face key point is used to characterize the accuracy of the heat map coordinates of the face key point;

[0288] A key point coordinate determination module 720 is used to determine, for each facial key point, the initial key point coordinates corresponding to the heat map coordinates of the facial key point in the face image to be located;

[0289] A size determination module 730 is used to determine, for each facial key point, the corresponding dynamic adjustment size of the facial key point in the texture feature map based on the uncertainty of the facial key point, and determine the target mapping coordinates of the initial key point coordinates in the texture feature map based on the mapping relationship between the face image to be located and the texture feature map;

[0290] A region determination module 740 is used to determine, for each facial key point, in the texture feature map, a target texture feature region centered on the mapping coordinates according to the dynamic resizing;

[0291] An offset value determination module 750 is used to determine, for each facial key point, an initial offset value corresponding to the target texture feature region and the target mapping coordinates based on a preset correspondence relationship among the texture feature region, the mapping coordinates, and the mapping coordinate offset value, and determine a target offset value corresponding to the initial offset value based on the mapping relationship;

[0292] The coordinate determination module 760 is used to determine, for each facial key point, the target key point coordinates after the initial key point coordinates are adjusted according to the target offset value, as the target coordinates of the facial key point in the face image to be located.

[0293] As can be seen from the above, by applying the solution provided by the embodiment of the present application, for each facial key point in the facial image to be located, the thermal map coordinates of the facial key point and the uncertainty of the thermal map coordinates can be used to determine the initial key point coordinates and target offset value of the facial key point in the facial image to be located, and then, the facial key point can be located in the facial image to be located using the above initial key point coordinates and target offset value. In this way, by adjusting the initial key point coordinates determined based on the thermal map coordinates using the target deviation value determined based on the uncertainty of the thermal map coordinates, the coordinate deviation caused by the use of low-precision thermal map coordinates for facial key point location can be corrected, thereby reducing the impact of factors such as the image size difference between the thermal map and the facial image to be located, and the loss of high-resolution detail features in the process of calculating the thermal map on the accuracy of facial key point location, thereby improving the accuracy of facial key point location.

[0294] Optionally, in a specific implementation, the heat map coordinate acquisition module 710 is specifically used to:

[0295] Inputting the face image to be located into a preset key point prediction model; wherein the key point prediction model includes a heat map sub-model and an uncertainty sub-model;

[0296] Obtaining the heat map coordinates of each facial key point output by the heat map sub-model and the uncertainty of each facial key point output by the uncertainty sub-model;

[0297] Among them, the key point prediction model is obtained by training a preset first sub-model based on a first sample face image and the sample heat map coordinates of each face key point in the first sample face image, and training a preset second sub-model based on the model prediction difference between a second sample face image and the second sample face image; the model prediction difference is: the difference between the prediction result of the heat map coordinate prediction of the second sample face image using the first sub-model and the sample heat map coordinates of each face key point in the second sample face image; the first sample face image and the second sample face image are different images.

[0298] Optionally, in a specific implementation, the device further includes a training module for a key point prediction model, and the training module for the key point prediction model is specifically used to:

[0299] Taking a first sample face image as input and taking sample heat map coordinates of each face key point in the first sample face image as output, training a preset first sub-model;

[0300] When the training of the first sub-model meets the specified training conditions, the second sample face image is used as input, and the model prediction difference of the second sample face image is used as output to train the preset second sub-model;

[0301] If the loss function of the second sub-model satisfies the first preset stop condition, the training is stopped to obtain a key point prediction model including the trained heat map sub-model and the uncertainty sub-model; wherein the heat map sub-model is the trained first sub-model, and the uncertainty sub-model is the trained second sub-model;

[0302] Otherwise, return to the step of training the preset first sub-model with the first sample face image as input and the sample heat map coordinates of each face key point in the first sample face image as output.

[0303] Optionally, in a specific implementation, the offset value determining module 750 includes:

[0304] An input submodule, used for inputting the target texture feature region and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face, and obtaining the offset value output by the offset value prediction model as an initial offset value;

[0305] Among them, the offset value prediction model is determined based on the sample texture feature area and the label of the sample texture feature area, and the sample texture feature area is: the predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the texture feature area determined in the sample texture feature map of the third sample face image; the label of the sample texture feature area is: the offset value of the sample mapping coordinates and the true value of the mapping coordinates of each facial key point in the third sample face image in the sample texture feature image; the sample mapping coordinates are the mapping coordinates of the sample initial key point coordinates determined based on the mapping relationship between the third sample face image and the sample texture feature map; the sample initial key point coordinates are the predicted heat map coordinates of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the corresponding coordinates in the third sample face image.

[0306] Optionally, in a specific implementation, the device further includes a training module for an offset value prediction model, and the training module for the offset value prediction model is specifically used to:

[0307] Obtaining predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample facial image with the key point prediction model, and obtaining a sample texture feature map of the third sample facial image;

[0308] For each facial key point, in the third sample facial image, determine the sample initial key point coordinates corresponding to the predicted heat map coordinates of the facial key point, and based on the mapping relationship between the third sample facial image and the sample texture feature map, determine the sample mapping coordinates corresponding to the sample initial key point coordinates in the sample texture feature map;

[0309] For each facial key point, based on the prediction uncertainty of the facial key point, determine the sample dynamic adjustment size corresponding to the facial key point in the sample texture feature map, and determine, in the sample texture feature map, a sample texture feature area centered on the sample mapping coordinates according to the sample dynamic adjustment size;

[0310] For each facial key point, calculating an offset value between the sample mapping coordinates and the true value of the mapping coordinates of the facial key point in the sample texture feature map as a label of the sample texture feature area;

[0311] Training a preset initial model with the sample texture feature region as output and the label of the sample texture feature region as output;

[0312] When the initial model meets the second preset stop condition, the training is stopped to obtain the offset value prediction model.

[0313] Optionally, in a specific implementation, the size determination module 730 includes:

[0314] The calculation submodule is used to calculate the dynamic adjustment size corresponding to the facial key point in the texture feature map based on the uncertainty of the facial key point and the sample size of the preset offset value prediction model.

[0315] Optionally, in a specific implementation, the device further includes:

[0316] A region size adjustment module, used for adjusting the region size of the target texture feature region according to a preset sample size of the offset value prediction model before inputting the target texture feature region and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face;

[0317] The input submodule is specifically used for:

[0318] The target texture feature region after the region size adjustment and the target mapping coordinates are input into a preset offset value prediction model corresponding to the key points of the human face.

[0319] Optionally, in a specific implementation, the computing submodule is specifically used to:

[0320] The dynamic adjustment size corresponding to the key point of the face in the texture feature map is calculated using a preset formula; wherein the preset formula is:

[0321]

[0322] Among them, ds is the dynamic adjustment size corresponding to the facial key point in the texture feature map; u is the uncertainty of the heat map coordinates of the facial key point; sigmoid(u) is a preset nonlinear function about uncertainty; k is the sample size of the offset value prediction model.

[0323] The present application also provides an electronic device, such as Figure 8 As shown, including:

[0324] Memory 801, used for storing computer programs;

[0325] The processor 802 is used to implement any of the above-mentioned methods for locating key points of a face when executing the program stored in the memory 801.

[0326] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other via the communication bus.

[0327] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0328] The communication interface is used for communication between the above electronic device and other devices.

[0329] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0330] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0331] In another embodiment provided in the present application, a computer-readable storage medium is also provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned facial key point locating methods are implemented.

[0332] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the facial key point locating methods in the above embodiments.

[0333] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0334] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0335] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, electronic device embodiment, computer-readable storage medium embodiment, and computer program product embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0336] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for locating key points of a face, characterized in that: The method comprises: Obtain a texture feature map of the face image to be located, and the heat map coordinates and uncertainty of each face key point in the face image to be located; wherein the uncertainty of each face key point is used to characterize the accuracy of the heat map coordinates of the face key point; For each facial key point, perform the following steps: In the face image to be located, determining the initial key point coordinates corresponding to the heat map coordinates of the face key points; Based on the uncertainty of the facial key point, determine the dynamic adjustment size corresponding to the facial key point in the texture feature map, and based on the mapping relationship between the face image to be located and the texture feature map, determine the target mapping coordinates of the initial key point coordinates in the texture feature map; In the texture feature map, determining a target texture feature region centered on the target mapping coordinates according to the dynamic resizing; Determine an initial offset value corresponding to the target texture feature region and the target mapping coordinates based on a preset correspondence relationship among the texture feature region, the mapping coordinates, and the mapping coordinate offset value, and determine a target offset value corresponding to the initial offset value based on the mapping relationship; Determine the target key point coordinates after the initial key point coordinates are adjusted according to the target offset value as the target coordinates of the facial key point in the face image to be located.

2. The method according to claim 1, characterized in that The step of obtaining the thermal map coordinates and uncertainty of each facial key point in the facial image to be located includes: Inputting the face image to be located into a preset key point prediction model; wherein the key point prediction model includes a heat map sub-model and an uncertainty sub-model; Obtaining the heat map coordinates of each facial key point output by the heat map sub-model and the uncertainty of each facial key point output by the uncertainty sub-model; Among them, the key point prediction model is obtained by training a preset first sub-model based on a first sample face image and the sample heat map coordinates of each face key point in the first sample face image, and training a preset second sub-model based on the model prediction difference between a second sample face image and the second sample face image; the model prediction difference is: the difference between the prediction result of the heat map coordinate prediction of the second sample face image using the first sub-model and the sample heat map coordinates of each face key point in the second sample face image; the first sample face image and the second sample face image are different images.

3. The method according to claim 2, characterized in that The training method of the key point prediction model includes: Taking a first sample face image as input and taking sample heat map coordinates of each face key point in the first sample face image as output, training a preset first sub-model; When the training of the first sub-model meets the specified training conditions, the second sample face image is used as input, and the model prediction difference of the second sample face image is used as output to train the preset second sub-model; If the loss function of the second sub-model satisfies the first preset stop condition, the training is stopped to obtain a key point prediction model including the trained heat map sub-model and the uncertainty sub-model; wherein the heat map sub-model is the trained first sub-model, and the uncertainty sub-model is the trained second sub-model; Otherwise, return to the step of training the preset first sub-model with the first sample face image as input and the sample heat map coordinates of each face key point in the first sample face image as output.

4. The method according to claim 2 or 3, characterized in that: The determining of the initial offset value corresponding to the target texture feature region and the target mapping coordinate based on the correspondence between the preset texture feature region, the mapping coordinate and the mapping coordinate offset value comprises: Inputting the target texture feature area and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face, and obtaining an offset value output by the offset value prediction model as an initial offset value; Among them, the offset value prediction model is determined based on the sample texture feature area and the label of the sample texture feature area, and the sample texture feature area is: the predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the texture feature area determined in the sample texture feature map of the third sample face image; the label of the sample texture feature area is: the offset value of the sample mapping coordinates and the true value of the mapping coordinates of each facial key point in the third sample face image in the sample texture feature image; the sample mapping coordinates are the mapping coordinates of the sample initial key point coordinates determined based on the mapping relationship between the third sample face image and the sample texture feature map; the sample initial key point coordinates are the predicted heat map coordinates of each facial key point obtained by predicting the third sample face image based on the key point prediction model, and the corresponding coordinates in the third sample face image.

5. The method according to claim 4, characterized in that The training method of the offset value prediction model includes: Obtaining predicted heat map coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample facial image with the key point prediction model, and obtaining a sample texture feature map of the third sample facial image; For each facial key point, in the third sample facial image, determine the sample initial key point coordinates corresponding to the predicted heat map coordinates of the facial key point, and based on the mapping relationship between the third sample facial image and the sample texture feature map, determine the sample mapping coordinates corresponding to the sample initial key point coordinates in the sample texture feature map; For each facial key point, based on the prediction uncertainty of the facial key point, determine the sample dynamic adjustment size corresponding to the facial key point in the sample texture feature map, and determine, in the sample texture feature map, a sample texture feature area centered on the sample mapping coordinates according to the sample dynamic adjustment size; For each facial key point, calculating an offset value between the sample mapping coordinates and the true value of the mapping coordinates of the facial key point in the sample texture feature map as a label of the sample texture feature area; Training a preset initial model with the sample texture feature region as output and the label of the sample texture feature region as output; When the initial model meets the second preset stop condition, the training is stopped to obtain the offset value prediction model.

6. The method according to claim 4, characterized in that The determining, based on the uncertainty of the facial key point, the dynamic adjustment size corresponding to the facial key point in the texture feature map comprises: Based on the uncertainty of the facial key point and the sample size of a preset offset value prediction model, a dynamic adjustment size corresponding to the facial key point in the texture feature map is calculated.

7. The method according to claim 6, characterized in that Before inputting the target texture feature area and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face, the method further includes: According to a preset sample size of the offset value prediction model, adjusting the area size of the target texture feature area; The step of inputting the target texture feature area and the target mapping coordinates into a preset offset value prediction model corresponding to the key points of the face includes: The target texture feature region after the region size adjustment and the target mapping coordinates are input into a preset offset value prediction model corresponding to the key points of the human face.

8. The method according to claim 7, characterized in that The calculating the dynamic adjustment size corresponding to the facial key point in the texture feature map based on the uncertainty of the facial key point and the sample size of the preset offset value prediction model comprises: The dynamic adjustment size corresponding to the key point of the face in the texture feature map is calculated using a preset formula; wherein the preset formula is: Among them, ds is the dynamically adjusted size corresponding to the facial key point in the texture feature map; u is the uncertainty of the heat map coordinates of the facial key point; sigmoid(u) is a preset nonlinear function related to uncertainty; k is the sample size of the offset value prediction model.

9. A facial key point positioning device, characterized in that: The device comprises: A heat map coordinate acquisition module is used to acquire a texture feature map of a face image to be located, and heat map coordinates and uncertainties of each face key point in the face image to be located; wherein the uncertainty of each face key point is used to characterize the accuracy of the heat map coordinates of the face key point; A key point coordinate determination module, for determining, for each facial key point, the initial key point coordinates corresponding to the heat map coordinates of the facial key point in the face image to be located; A size determination module is used to determine, for each facial key point, the corresponding dynamic adjustment size of the facial key point in the texture feature map based on the uncertainty of the facial key point, and determine the target mapping coordinates of the initial key point coordinates in the texture feature map based on the mapping relationship between the face image to be located and the texture feature map; A region determination module, for determining, for each facial key point, in the texture feature map, a target texture feature region centered on the mapping coordinates according to the dynamic resizing; An offset value determination module is used to determine, for each facial key point, an initial offset value corresponding to the target texture feature region and the target mapping coordinates based on a preset correspondence relationship among the texture feature region, the mapping coordinates and the mapping coordinate offset value, and determine a target offset value corresponding to the initial offset value based on the mapping relationship; The coordinate determination module is used to determine, for each facial key point, the target key point coordinates after the initial key point coordinates are adjusted according to the target offset value as the target coordinates of the facial key point in the face image to be located.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images