A face key point positioning method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a face key point positioning method and device, electronic equipment and storage medium, and relate to the technical field of image processing. The texture feature map of the face image to be positioned is obtained, and the heat map coordinates and uncertainty of each face key point are obtained. For each face key point: the initial key point coordinates corresponding to the heat map coordinates are determined in the face image to be positioned, and the dynamic adjustment size is determined based on the uncertainty. In the texture feature map, the target mapping coordinates corresponding to the initial key point coordinates are determined, and the target texture feature area is determined. Based on the preset correspondence relationship, the initial offset value corresponding to the target texture feature area and the target mapping coordinates is determined, and the target offset value is determined. The target key point coordinates obtained by adjusting the initial key point coordinates according to the target offset value are used as the target coordinates in the face image to be positioned. The application of the embodiments of the present application can improve the accuracy of face key point positioning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a face key point positioning method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the rapid development of face recognition technology, face recognition has been widely used in monitoring, payment, access control, entertainment and other related technical fields, providing more convenience for users. Among them, face recognition is realized by positioning each face key point.

[0003] In related technologies, face key point positioning can be realized by using heat map prediction. The steps of using heat map prediction to locate face key points include: obtaining a pre-set size of a to-be-identified face image, sequentially performing texture feature extraction, downsampling and depth feature extraction on the to-be-identified face image to obtain a depth feature map of the to-be-identified face image. Then, using the depth feature map, the heat map of each face key point in the to-be-identified face image is determined, then the coordinates of each face key point in the heat map can be determined, and the determined coordinates are called the heat map coordinates of the face key point. Then, according to the image size of the heat map and the image size of the to-be-identified image, the downsampling ratio is determined, and according to the determined downsampling ratio, the heat map coordinates of each face key point are mapped to the to-be-identified face image, so that each face key point can be located in the above to-be-identified face image.

[0004] However, in related technologies, since the heat map of each key point is an image obtained by downsampling the to-be-identified face image, the high-resolution detail features in the to-be-identified face image will be lost in the process of obtaining the heat map, resulting in low accuracy of the heat map coordinates determined by using the heat map, and further, the accuracy of each face key point obtained by using the above heat map coordinates to locate the face key point in the to-be-identified face image is low. SUMMARY

[0005] The purpose of the embodiments of the present application is to provide a face key point positioning method, device, electronic equipment and storage medium to improve the accuracy of face key point positioning. The specific technical solutions are as follows:

[0006] In a first aspect, the embodiments of the present application provide a face key point positioning method, which comprises:

[0007] obtaining a texture feature map of a to-be-positioned face image, and heat map coordinates and uncertainty of each face key point in the to-be-positioned face image; wherein the uncertainty of each face key point is used to represent the accuracy of the heat map coordinates of the face key point;

[0008] For each face key point, the following steps are performed:

[0009] In the face image to be positioned, the initial key point coordinates corresponding to the heat map coordinates of the face key point are determined;

[0010] Based on the uncertainty of the face key point, the dynamic adjustment size corresponding to the face key point in the texture feature map is determined, and based on the mapping relationship between the face image to be positioned and the texture feature map, the target mapping coordinates of the initial key point coordinates in the texture feature map are determined;

[0011] In the texture feature map, according to the dynamic adjustment size, a target texture feature region centered on the target mapping coordinates is determined;

[0012] Based on the correspondence relationship between the preset texture feature region, the mapping coordinates and the mapping coordinate offset value, the initial offset value corresponding to the target texture feature region and the target mapping coordinates is determined, and based on the mapping relationship, the target offset value corresponding to the initial offset value is determined;

[0013] The target key point coordinates obtained by adjusting the initial key point coordinates according to the target offset value are determined as the target coordinates of the face key point in the face image to be positioned.

[0014] Optionally, in a specific implementation manner, the heat map coordinates and the uncertainty of each face key point in the face image to be positioned are obtained, including:

[0015] The face image to be positioned is input into a preset key point prediction model; wherein the key point prediction model includes a heat map sub-model and an uncertainty sub-model;

[0016] The heat map coordinates of each face key point output by the heat map sub-model and the uncertainty of each face key point output by the uncertainty sub-model are obtained;

[0017] The key point prediction model is obtained by training a preset first sub-model based on a first sample face image and sample heat map coordinates of each face key point in the first sample face image, and training a preset second sub-model based on a second sample face image and a model prediction difference of the second sample face image; the model prediction difference is the difference between the prediction result of the first sub-model for predicting the heat map coordinates of the second sample face image and the sample heat map coordinates of each face key point in the second sample face image; the first sample face image and the second sample face image are different images.

[0018] Optionally, in an implementation manner, the training manner of the key point prediction model comprises:

[0019] training a preset first sub-model by taking a first sample face image as input and taking sample heat map coordinates of each face key point in the first sample face image as output;

[0020] training a preset second sub-model by taking a second sample face image as input and taking model prediction difference of the second sample face image as output when the training of the first sub-model satisfies a specified training condition;

[0021] if a loss function of the second sub-model satisfies a first preset stopping condition, stopping the training to obtain a key point prediction model comprising a trained heat map sub-model and an uncertainty sub-model; wherein the heat map sub-model is the trained first sub-model, and the uncertainty sub-model is the trained second sub-model;

[0022] otherwise, returning to the step of training the preset first sub-model by taking the first sample face image as input and taking the sample heat map coordinates of each face key point in the first sample face image as output.

[0023] Optionally, in an implementation manner, the determining of the initial offset value corresponding to the target texture feature region and the target mapping coordinate based on the preset correspondence relationship between the texture feature region, the mapping coordinate and the mapping coordinate offset value comprises:

[0024] inputting the target texture feature region and the target mapping coordinate into a preset offset value prediction model corresponding to the face key point, and obtaining an offset value output by the offset value prediction model as the initial offset value;

[0025] The offset value prediction model is determined based on a sample texture feature region and a label of the sample texture feature region. The sample texture feature region is determined based on the predicted heat map coordinates and the predicted uncertainty of each facial key point obtained by predicting the third sample face image by using the key point prediction model, and a texture feature region in a sample texture feature map of the third sample face image. The label of the sample texture feature region is an offset value of a sample mapping coordinate and a mapping coordinate true value of each facial key point in the third sample face image in the sample texture feature image. The sample mapping coordinate is a mapping coordinate of a sample initial key point coordinate in the sample texture feature map, which is determined based on a mapping relationship between the third sample face image and the sample texture feature map. The sample initial key point coordinate is the predicted heat map coordinate of each facial key point obtained by predicting the third sample face image by using the key point prediction model, and a corresponding coordinate in the third sample face image.

[0026] Optionally, in a specific implementation manner, the training manner of the offset value prediction model comprises:

[0027] obtaining the predicted heat map coordinates and the predicted uncertainty of each facial key point obtained by predicting the third sample face image by using the key point prediction model, and obtaining a sample texture feature map of the third sample face image;

[0028] for each facial key point, determining a sample initial key point coordinate corresponding to the predicted heat map coordinate of the facial key point in the third sample face image, and determining a sample mapping coordinate corresponding to the sample initial key point coordinate in the sample texture feature map based on a mapping relationship between the third sample face image and the sample texture feature map;

[0029] for each facial key point, determining a sample dynamic adjustment size corresponding to the facial key point in the sample texture feature map based on the predicted uncertainty of the facial key point, and determining a sample texture feature region centered on the sample mapping coordinate in the sample texture feature map according to the sample dynamic adjustment size;

[0030] for each facial key point, calculating an offset value of the sample mapping coordinate and a mapping coordinate true value of the facial key point in the sample texture feature map as the label of the sample texture feature region;

[0031] training a preset initial model with the sample texture feature region as an output and the label of the sample texture feature region as an output;

[0032] When the initial model meets a second preset stopping condition, the training is stopped to obtain the offset value prediction model.

[0033] Optionally, in a specific implementation, the method further includes:

[0034] Optionally, in a specific implementation, the method further includes:

[0035] Optionally, in a specific implementation, before the target texture feature region and the target mapping coordinates are input into the preset offset value prediction model corresponding to the face key point, the method further includes:

[0036] Optionally, in a specific implementation, before the target texture feature region and the target mapping coordinates are input into the preset offset value prediction model corresponding to the face key point, the method further includes:

[0037] Optionally, in a specific implementation, the method further includes:

[0038] Optionally, in a specific implementation, the method further includes:

[0039] Optionally, in a specific implementation, the method further includes:

[0040] Optionally, in a specific implementation, the method further includes:

[0041]

[0042] wherein, ds is the dynamic adjustment size of the face key point in the texture feature map; u is the uncertainty of the heat map coordinates of the face key point; sigmoid(u) is a preset nonlinear function about the uncertainty; and k is the sample size of the offset value prediction model.

[0043] In a second aspect, an embodiment of the present application provides a face key point positioning device, the device comprising:

[0044] The heat map coordinate acquisition module is configured to acquire a texture feature map of a to-be-located face image, and heat map coordinates and uncertainty of each face key point in the to-be-located face image; wherein the uncertainty of each face key point is used to represent the accuracy of the heat map coordinates of the face key point.

[0045] The key point coordinate determination module is configured to, for each face key point, determine an initial key point coordinate corresponding to the heat map coordinates of the face key point in the to-be-located face image.

[0046] The size determination module is configured to, for each face key point, determine a dynamic adjustment size corresponding to the face key point in the texture feature map based on the uncertainty of the face key point, and determine a target mapping coordinate of the initial key point coordinate in the texture feature map based on a mapping relationship between the to-be-located face image and the texture feature map.

[0047] The region determination module is configured to, for each face key point, determine a target texture feature region centered at the mapping coordinate in the texture feature map according to the dynamic adjustment size.

[0048] The offset value determination module is configured to, for each face key point, determine an initial offset value corresponding to the target texture feature region and the target mapping coordinate based on a preset correspondence relationship among texture feature regions, mapping coordinates and mapping coordinate offset values, and determine a target offset value corresponding to the initial offset value based on the mapping relationship.

[0049] The coordinate determination module is configured to, for each face key point, determine a target key point coordinate obtained by adjusting the initial key point coordinate according to the target offset value as a target coordinate of the face key point in the to-be-located face image.

[0050] Optionally, in a specific implementation, the heat map coordinate acquisition module is specifically configured to:

[0051] input the to-be-located face image into a preset key point prediction model; wherein the key point prediction model comprises a heat map sub-model and an uncertainty sub-model;

[0052] acquire the heat map coordinates of each face key point output by the heat map sub-model and the uncertainty of each face key point output by the uncertainty sub-model;

[0053] The key point prediction model is obtained by training a preset first sub-model based on a first sample face image and sample heat map coordinates of each face key point in the first sample face image, and training a preset second sub-model based on a second sample face image and a model prediction difference of the second sample face image.

[0054] Optionally, in a specific implementation, the apparatus further includes a key point prediction model training module, which is specifically configured to:

[0055] train the preset first sub-model by taking a first sample face image as input and taking sample heat map coordinates of each face key point in the first sample face image as output;

[0056] train the preset second sub-model by taking a second sample face image as input and taking a model prediction difference of the second sample face image as output when the training of the first sub-model meets a specified training condition;

[0057] stop training if a loss function of the second sub-model meets a first preset stop condition, and obtain a key point prediction model including a trained heat map sub-model and an uncertainty sub-model; the heat map sub-model is the trained first sub-model, and the uncertainty sub-model is the trained second sub-model;

[0058] otherwise, return to the step of training the preset first sub-model by taking the first sample face image as input and taking the sample heat map coordinates of each face key point in the first sample face image as output.

[0059] Optionally, in a specific implementation, the offset value determination module includes:

[0060] an input sub-module configured to input the target texture feature region and the target mapping coordinates into a preset offset value prediction model corresponding to the face key point, and obtain an offset value output by the offset value prediction model as an initial offset value;

[0061] The offset value prediction model is determined based on a sample texture feature region and a label of the sample texture feature region. The sample texture feature region is determined based on the predicted heat map coordinates and the predicted uncertainty of each facial key point obtained by predicting the third sample face image by using the key point prediction model, and a texture feature region in a sample texture feature map of the third sample face image. The label of the sample texture feature region is an offset value of a sample mapping coordinate and a mapping coordinate true value of each facial key point in the third sample face image in the sample texture feature image. The sample mapping coordinate is a mapping coordinate of a sample initial key point coordinate in the sample texture feature map, which is determined based on a mapping relationship between the third sample face image and the sample texture feature map. The sample initial key point coordinate is the predicted heat map coordinate of each facial key point obtained by predicting the third sample face image by using the key point prediction model, and a corresponding coordinate in the third sample face image.

[0062] Optionally, in a specific implementation manner, the apparatus further includes a training module of the offset value prediction model, and the training module of the offset value prediction model is specifically configured to:

[0063] obtain the predicted heat map coordinates and the predicted uncertainty of each facial key point obtained by predicting the third sample face image by using the key point prediction model, and obtain a sample texture feature map of the third sample face image;

[0064] for each facial key point, determine a sample initial key point coordinate corresponding to the predicted heat map coordinate of the facial key point in the third sample face image, and determine a sample mapping coordinate corresponding to the sample initial key point coordinate in the sample texture feature map based on a mapping relationship between the third sample face image and the sample texture feature map;

[0065] for each facial key point, determine a sample dynamic adjustment size corresponding to the facial key point in the sample texture feature map based on the predicted uncertainty of the facial key point, and determine a sample texture feature region centered on the sample mapping coordinate in the sample texture feature map according to the sample dynamic adjustment size;

[0066] for each facial key point, calculate an offset value of the sample mapping coordinate and a mapping coordinate true value of the facial key point in the sample texture feature map as the label of the sample texture feature region;

[0067] train a preset initial model by taking the sample texture feature region as an output and taking the label of the sample texture feature region as an output;

[0068] When the initial model meets a second preset stopping condition, the training is stopped to obtain the offset value prediction model.

[0069] Optionally, in an implementation, the size determining module comprises:

[0070] The computing sub-module is configured to calculate a dynamic adjustment size of the face key point in the texture feature map based on the uncertainty of the face key point and a sample size of the preset offset value prediction model.

[0071] Optionally, in an implementation, the device further comprises:

[0072] The region size adjustment module is configured to adjust a region size of the target texture feature region according to a sample size of the preset offset value prediction model before inputting the target texture feature region and the target mapping coordinates into the preset offset value prediction model corresponding to the face key point.

[0073] The input sub-module is specifically configured to:

[0074] input the target texture feature region with the adjusted region size and the target mapping coordinates into the preset offset value prediction model corresponding to the face key point.

[0075] Optionally, in an implementation, the computing sub-module is specifically configured to:

[0076] calculate the dynamic adjustment size of the face key point in the texture feature map by using a preset formula; and the preset formula is:

[0077]

[0078] wherein, ds is the dynamic adjustment size of the face key point in the texture feature map; u is the uncertainty of the heat map coordinates of the face key point; sigmoid(u) is a preset nonlinear function about the uncertainty; and k is the sample size of the offset value prediction model.

[0079] In a third aspect, an electronic device is provided, comprising:

[0080] a memory for storing a computer program;

[0081] a processor for executing the program stored in the memory to implement the steps of any of the method embodiments.

[0082] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, which, when executed by a processor, implements any of the methods described above.

[0083] In a fifth aspect, the embodiments of the present application further provide a computer program product containing instructions which, when executed on a computer, cause the computer to perform any of the above-described methods.

[0084] The embodiments of the present application have the following beneficial effects:

[0085] As can be seen above, when locating each facial key point in the face image to be located, the scheme provided by the embodiments of the present application can first acquire the texture feature map of the face image to be located, and the heat map coordinates of each facial key point in the face image to be located and the uncertainty used to represent the accuracy of the heat map coordinates; then, for each facial key point, the following steps can be performed: first, in the face image to be located, the initial key point coordinates corresponding to the heat map coordinates of the facial key point are determined; then, based on the uncertainty of the facial key point, the dynamically adjusted size corresponding to the facial key point in the texture feature map is determined, and based on the mapping relationship between the face image to be located and the texture feature map, the target mapping coordinates of the initial key point coordinates in the texture feature map are determined; then, in the texture feature map, the target texture feature region centered at the target mapping coordinates is determined according to the dynamically adjusted size; then, based on the correspondence between the preset texture feature region, the mapping coordinates and the mapping coordinate offset value, the initial offset value corresponding to the target texture feature region and the above-mentioned target mapping coordinates is determined, and based on the above-mentioned mapping relationship, the target offset value corresponding to the initial offset value is determined; further, the target key point coordinates of the initial key point coordinates adjusted according to the target offset value can be determined, and the target key point coordinates are taken as the target coordinates of the facial key point in the face image to be located.

[0086] Based on this, for each facial key point in the face image to be located, the scheme provided by the embodiments of the present application can utilize the heat map coordinates and the uncertainty of the heat map coordinates of the facial key point to determine the initial key point coordinates and the target offset value of the facial key point in the face image to be located, and then the initial key point coordinates and the target offset value can be used to locate the facial key point in the face image to be located. In this way, by adjusting the initial key point coordinates determined based on the heat map coordinates using the target offset value determined based on the uncertainty of the heat map coordinates, the coordinate deviation caused by using the low-precision heat map coordinates for facial key point location can be corrected, thereby reducing the influence of the difference between the image size of the heat map and the face image to be located and the loss of high-resolution detailed features in the process of calculating the heat map on the accuracy of facial key point location, and improving the accuracy of facial key point location. BRIEF DESCRIPTION OF DRAWINGS

[0087] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and other embodiments can also be obtained by those skilled in the art based on these drawings.

[0088] Figure 1 A flowchart of a face key point positioning method provided by an embodiment of the present application;

[0089] Figure 2 A structural diagram of a key point prediction model provided by an embodiment of the present application;

[0090] Figure 3 A flowchart of a training method of a key point prediction model provided by an embodiment of the present application;

[0091] Figure 4 A flowchart of a training method of an offset value prediction model provided by an embodiment of the present application;

[0092] Figure 5 An illustrative diagram of a sample texture feature region provided by an embodiment of the present application;

[0093] Figure 6 A flowchart of a specific example of a face key point positioning method provided by an embodiment of the present application;

[0094] Figure 7 A structural diagram of a face key point positioning device provided by an embodiment of the present application;

[0095] Figure 8 A structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0096] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art based on the present application belong to the scope of protection of the present application.

[0097] In the related art, since the heat map of each key point is an image obtained after down-sampling processing of the to-be-recognized face image, high-resolution detail features in the to-be-recognized face image will be lost in the process of obtaining the heat map, so that the accuracy of the heat map coordinates determined by using the heat map is low, and then the face key point positioning in the to-be-recognized face image by using the above heat map coordinates has low accuracy of each face key point.

[0098] To solve the above technical problems, the embodiment of the application provides a face key point positioning method.

[0099] The method is suitable for various application scenarios with face key point positioning requirements, for example, a payment software identifies face key points in a user face image to realize face payment, and an access control system opens access control by identifying irises in a user face image.

[0100] The method can be applied to various image acquisition devices with data processing functions, for example, an image acquisition device is equipped with a data processing module, and after acquiring a face image to be positioned, the image acquisition device executes the method by using the data processing module carried by the image acquisition device. The method can also be applied to various electronic devices that can communicate with the image acquisition device, for example, a mobile phone, a tablet computer, a notebook computer, and the like. When the execution subject of the method is an electronic device, the electronic device can be a stand-alone electronic device or a device cluster composed of multiple electronic devices, which is referred to as an electronic device hereinafter.

[0101] Therefore, the embodiment of the application does not specifically limit the application scenarios and execution subjects of the method.

[0102] It should be noted that in the technical solution of the application, the operations of obtaining, storing, using, processing, transmitting, providing, and disclosing user personal information are all performed with the authorization of the user.

[0103] It should be noted that the head image in the embodiment is not a head image of a specific user, and cannot reflect the personal information of the specific user.

[0104] It should be noted that the face image in the embodiment comes from a public data set.

[0105] The face key point positioning method provided by the embodiment of the application can include the following steps:

[0106] Obtaining a texture feature map of a face image to be positioned, and heat map coordinates and uncertainty of each face key point in the face image to be positioned; wherein the uncertainty of each face key point is used to represent the accuracy of the heat map coordinates of the face key point;

[0107] For each face key point, the following steps are performed:

[0108] In the face image to be positioned, determining initial key point coordinates corresponding to the heat map coordinates of the face key point;

[0109] determine a dynamic adjustment size corresponding to the facial key point in the texture feature map based on the uncertainty of the facial key point, and determine a target mapping coordinate of the initial key point coordinate in the texture feature map based on a mapping relationship between the face image to be positioned and the texture feature map;

[0110] In the texture feature map, a target texture feature region centered on the target mapping coordinate is determined according to the dynamic adjustment size;

[0111] An initial offset value corresponding to the target texture feature region and the target mapping coordinate is determined based on a preset correspondence relationship among a texture feature region, a mapping coordinate and a mapping coordinate offset value, and a target offset value corresponding to the initial offset value is determined based on the mapping relationship.

[0112] A target key point coordinate obtained by adjusting the initial key point coordinate according to the target offset value is determined as a target coordinate of the facial key point in the face image to be positioned.

[0113] As can be seen from the above, when positioning each facial key point in the face image to be positioned, the scheme provided in the embodiments of the present application can first acquire a texture feature map of the face image to be positioned, and a heat map coordinate of each facial key point in the face image to be positioned and an uncertainty used to represent the accuracy of the heat map coordinate. Then, for each facial key point, the following steps can be performed: first, in the face image to be positioned, an initial key point coordinate corresponding to the heat map coordinate of the facial key point is determined; then, based on the uncertainty of the facial key point, a dynamic adjustment size corresponding to the facial key point in the texture feature map is determined, and a target mapping coordinate of the initial key point coordinate in the texture feature map is determined based on a mapping relationship between the face image to be positioned and the texture feature map; then, in the texture feature map, a target texture feature region centered on the target mapping coordinate is determined according to the dynamic adjustment size; then, an initial offset value corresponding to the target texture feature region and the target mapping coordinate can be determined based on a preset correspondence relationship among a texture feature region, a mapping coordinate and a mapping coordinate offset value, and a target offset value corresponding to the initial offset value is determined based on the mapping relationship; further, a target key point coordinate obtained by adjusting the initial key point coordinate according to the target offset value can be determined, and the target key point coordinate is taken as a target coordinate of the facial key point in the face image to be positioned.

[0114] Based on this, by applying the scheme provided in the embodiments of the present application, for each facial key point in the to-be-located facial image, the initial key point coordinate and the target offset value of the facial key point in the to-be-located facial image can be determined by using the heat map coordinate of the facial key point and the uncertainty of the heat map coordinate, and then the facial key point in the to-be-located facial image can be located by using the initial key point coordinate and the target offset value. In this way, by adjusting the initial key point coordinate determined based on the heat map coordinate by using the target offset value determined based on the uncertainty of the heat map coordinate, the coordinate deviation caused by using the low-precision heat map coordinate for facial key point location can be corrected, thereby reducing the influence of the image size difference between the heat map and the to-be-located facial image and the loss of high-resolution detailed features in the process of calculating the heat map on the accuracy of facial key point location, and improving the accuracy of facial key point location.

[0115] Next, a facial key point location method provided by the embodiments of the present application will be described in detail in combination with the accompanying drawings.

[0116] Figure 1 A flowchart of a facial key point location method provided by the embodiments of the present application is shown in FIG. 1, which can include the following steps S101-S106. Figure 1

[0117] S101: Obtain a texture feature map of a to-be-located facial image, and a heat map coordinate and an uncertainty of each facial key point in the to-be-located facial image;

[0118] The uncertainty of each facial key point is used to represent the accuracy of the heat map coordinate of the facial key point.

[0119] When locating each facial key point in the to-be-located facial image, the texture feature map of the to-be-located facial image can be obtained first, and the heat map coordinate of each facial key point in the to-be-located facial image can be obtained.

[0120] The texture feature map of the to-be-located facial image is an image obtained by extracting the texture of the to-be-located facial image.

[0121] Optionally, the texture feature of the to-be-located facial image can be extracted by using a preset texture feature extraction algorithm, so as to obtain the texture feature map of the to-be-located facial image. In this way, the image size of the obtained texture feature map is the same as the image size of the to-be-located facial image.

[0122] ​Optionally, the to-be-positioned face image can be down-sampled, and then a preset texture feature extraction algorithm is used to extract the texture feature of the down-sampled to-be-positioned face image, to obtain a texture feature map of the to-be-positioned face image. In this way, the image size of the obtained texture feature map is smaller than the image size of the to-be-positioned face image.

[0123] It can be seen that the image size of the texture feature map of the to-be-positioned face image can be the same as or different from the image size of the to-be-positioned face image.

[0124] When the image size of the texture feature map of the to-be-positioned face image is the same as the image size of the to-be-positioned face image, for each pixel point in the to-be-positioned face image, the pixel coordinate of the pixel point in the to-be-positioned face image is the same as the pixel coordinate of the mapping point of the pixel point in the texture feature map. For example, when the image size of the texture feature map of the to-be-positioned face image is the same as the image size of the to-be-positioned face image, the pixel coordinate of the pixel point A with a pixel coordinate of (10, 10) in the to-be-positioned face image is (10, 10), and the pixel coordinate of the mapping point A' of the pixel point A in the texture feature map is (10, 10).

[0125] When the image size of the texture feature map of the to-be-positioned face image is different from the image size of the to-be-positioned face image, for each pixel point in the to-be-positioned face image, the pixel coordinate of the pixel point in the to-be-positioned face image is different from the pixel coordinate of the mapping point of the pixel point in the texture feature map. For example, when the image size of the to-be-positioned face image is 224*224 and the image size of the texture feature map of the to-be-positioned face image is 112*112, the pixel coordinate of the pixel point A with a pixel coordinate of (10, 10) in the to-be-positioned face image is (5, 5) in the texture feature map.

[0126] Based on this, for each pixel point of the to-be-positioned face image, the corresponding mapping point of the pixel point in the texture feature map can be determined based on the proportional relationship between the image size of the to-be-positioned face image and the image size of the texture feature map. Thus, in order to correspond the pixel points in the to-be-positioned face image and the texture feature map, a mapping relationship between the to-be-positioned face image and the texture feature map can be established after obtaining the texture feature map of the to-be-positioned face image.

[0127] The mapping relationship can include the proportional relationship between the image size of the to-be-positioned face image and the image size of the texture feature map, and the corresponding relationship between each pixel point in the to-be-positioned face image and each pixel point in the texture feature map.

[0128] In this way, for each pixel point of the to-be-positioned face image, the corresponding mapping point of the pixel point in the texture feature map can be determined based on the established mapping relationship.

[0129] In addition, the number of required positioning facial key points in the to-be-positioned facial image can be set according to actual needs, for example, 68, 72, 118, etc., which are all reasonable, and are not specifically limited in the embodiments of the present application.

[0130] Since the heat map coordinates of each facial key point are the coordinates of the facial key point in the heat map corresponding to the facial key point. Generally, for each facial key point, in the process of obtaining the heat map of the facial key point based on the to-be-positioned facial image, there may be influencing factors such as loss of image feature data in the to-be-positioned facial image, calculation deviation, etc., which result in errors in the calculated heat map of the facial key point; and when determining the heat map coordinates of the facial key point in the heat map of the facial key point, there may also be influencing factors such as calculation errors, etc., so that under the influence of various influencing factors, the accuracy of the finally calculated heat map coordinates of the facial key point is poor. Moreover, for each facial key point, when locating the facial key point in the to-be-positioned facial image by using the heat map coordinates of the facial key point, the accuracy of the heat map coordinates will directly affect the accuracy of the positioning of the facial key point.

[0131] Therefore, for each facial key point, after obtaining the heat map coordinates of the facial key point, the accuracy of the heat map coordinates of the facial key point can be further determined.

[0132] That is, when performing facial key point positioning on a to-be-positioned facial image, for each facial key point in the to-be-positioned facial image, after obtaining the heat map coordinates of the facial key point, the uncertainty representing the accuracy of the heat map coordinates of the facial key point can be obtained.

[0133] In one specific implementation, the step S101 can include the following steps 11-12:

[0134] Step 11: input the to-be-positioned facial image into a preset key point prediction model; wherein the key point prediction model includes a heat map sub-model and an uncertainty sub-model;

[0135] Step 12: obtain the heat map coordinates of each facial key point output by the heat map sub-model and the uncertainty of each facial key point output by the uncertainty sub-model;

[0136] The key point prediction model is trained based on a first sample face image, sample heat map coordinates of each face key point in the first sample face image, and a preset first sub-model, and trained based on a second sample face image, a model prediction difference of the second sample face image, and a preset second sub-model.

[0137] In the specific implementation, the key point prediction model including the heat map sub-model and the uncertainty sub-model can be trained in advance using the first sample face image and the second sample face image.

[0138] The key point prediction model can be trained based on a first sample face image, sample heat map coordinates of each face key point in the first sample face image, and a preset first sub-model, and trained based on a second sample face image, a model prediction difference of the second sample face image, and a preset second sub-model. The model prediction difference is a difference between a prediction result of heat map coordinate prediction of the second sample face image by the first sub-model and the sample heat map coordinates of each face key point in the second sample face image. The first sample face image and the second sample face image can be different images. For example, the first sample face image is a face of a young male A, and the second sample face image is a face of an old male B.

[0139] To make the description clear, a training method of a key point prediction model provided by an embodiment of the present application will be described in detail below.

[0140] After obtaining the face image to be positioned, the face image to be positioned can be input into the preset key point prediction model. In this way, the heat map sub-model in the key point prediction model can determine heat map coordinates of each face key point in the face image to be positioned, and the uncertainty sub-model in the key point prediction model can determine an uncertainty of the heat map coordinates output by the heat map sub-model to represent an accuracy of each face key point.

[0141] Therefore, the electronic device can obtain the heat map coordinates of each face key point output by the heat map sub-model, and obtain the uncertainty of each heat map coordinate output by the uncertainty sub-model.

[0142] For example, the heat map sub-model can be trained based on a first sample face image, sample heat map coordinates of each face key point in the first sample face image, and a preset first sub-model, and the uncertainty sub-model can be trained based on a second sample face image, a model prediction difference of the second sample face image, and a preset second sub-model. Figure 2As shown, after obtaining the to-be-positioned face image with an image size of 224*224, the to-be-positioned face image can be input into the key point prediction model, so that the key point prediction model can extract the texture feature of the to-be-positioned face image. Then, the texture feature image can be down-sampled, and the texture feature image after down-sampling can be subjected to deep feature extraction, so as to obtain the deep feature map of the to-be-positioned face image. Then, the deep feature map of the to-be-positioned face image can be input into the heat map sub-model and the uncertainty sub-model of the key point prediction model. Then, the heat map of 68 face key points output by the heat map sub-model is obtained, and the heat map of 68*56*56 is obtained; and the uncertainty of the 68 heat map coordinate points output by the uncertainty sub-model is obtained, and the uncertainty of 1*68 is obtained. Wherein, the heat map of 68*56*56 is a collection of the heat map of 56*56 of each face key point in the 68 face key points, and the heat map of each face key point is marked with the heat map coordinate of the face key point; the uncertainty of 1*68 is a collection of the uncertainty of each face key point in the 68 face key points. Further, for each face key point, the heat map coordinate of the face key point marked in the heat map of the face key point can be obtained, and the uncertainty of the heat map coordinate of the face key point can be obtained.

[0143] After obtaining the heat map coordinate and uncertainty of each face key point in the to-be-positioned face image, the heat map coordinate and uncertainty of each face key point can be used to position the face key point in the to-be-positioned face image.

[0144] Wherein, in order to position each face key point in the to-be-positioned face image, the following steps S102-S106 can be performed:

[0145] S102: for each face key point, determining the initial key point coordinate corresponding to the heat map coordinate of the face key point in the to-be-positioned face image;

[0146] For each face key point, after determining the heat map coordinate of the face key point, the heat map coordinate of the face key point can be used to determine the initial key point coordinate corresponding to the heat map coordinate of the face key point in the to-be-positioned face image.

[0147] Wherein, the image region corresponding to the heat map of the face key point can be first determined in the to-be-positioned face image, and then the image coordinate corresponding to the heat map coordinate in the determined image region can be determined as the initial key point coordinate corresponding to the heat map coordinate of the face key point by using the heat map coordinate of the face key point.

[0148] S103: For each face key point, based on the uncertainty of the face key point, determine the dynamically adjusted size corresponding to the face key point in the texture feature map, and based on the mapping relationship between the face image to be positioned and the texture feature map, determine the target mapping coordinates of the initial key point coordinates in the texture feature map;

[0149] Generally, for each face key point, the heat map coordinates of the face key point correspond to the initial key point coordinates in the face image to be positioned, that is, the target coordinates of the face key point in the face image to be positioned. However, as mentioned above, due to various influencing factors in the process of determining the heat map coordinates of the face key point, the accuracy of the determined heat map coordinates is poor, and thus after mapping the above heat map coordinates back to the face image to be positioned, the obtained initial key point coordinates deviate from the target coordinates of the face key point in the face image to be positioned.

[0150] That is, for each face key point, the initial key point coordinates corresponding to the heat map coordinates of the face key point in the face image to be positioned can represent the approximate position of the face key point in the face image to be positioned, and the target coordinates of the face key point in the face image to be positioned are coordinates that have a certain coordinate offset from the above initial key point coordinates.

[0151] As can be seen from the above, in order to determine the target coordinates of the face key point in the face image to be positioned, the target offset value between the above initial key point coordinates and the target coordinates can be determined first.

[0152] Since the image details included in the face image to be positioned can affect the determination process of the target offset value, resulting in poor accuracy of the determined target offset value, and in the process of extracting the texture features of the face image to be positioned, part of the image details can be eliminated. Thus, the target offset value between the above initial key point coordinates and the target coordinates can be determined using the texture feature map of the face image to be positioned.

[0153] Among them, since the texture feature map of the face image to be positioned has a mapping relationship with the face image to be positioned, the mapping relationship can be used to determine the mapping coordinates of the initial key point coordinates and the target coordinates of the face key point in the face image to be positioned in the texture feature map, and then the initial offset value between the mapping coordinates of the initial key point coordinates and the target coordinates in the texture feature map can be determined. And the obtained initial offset value and the above target offset value also have the above mapping relationship.

[0154] Based on this, for each face key point, after determining the initial key point coordinate of the face key point in the to-be-positioned face image, the mapping relationship can be used to determine the corresponding target mapping coordinate of the initial key point coordinate in the texture feature map.

[0155] For example, when the image size of the to-be-positioned face image is 224*224 and the image size of the texture feature map of the to-be-positioned face image is 112*112, if the initial key point coordinate is (50, 50), the corresponding target mapping coordinate of the initial key point coordinate in the texture feature map is (25, 25); and when the image size of the to-be-positioned face image is 224*224 and the image size of the texture feature map of the to-be-positioned face image is 224*224, if the initial key point coordinate is (50, 50), the corresponding target mapping coordinate of the initial key point coordinate in the texture feature map is (50, 50).

[0156] After determining the corresponding target mapping coordinate of the initial key point coordinate in the texture feature map, the initial offset value of the mapping coordinate corresponding to the target mapping coordinate and the target coordinate in the texture feature map can be further calculated.

[0157] For each face key point, in order to determine the initial offset value, the target texture feature region of the face key point in the texture feature map can be first determined, and in order to determine the target texture feature region, the dynamic adjustment size of the target mapping coordinate can be first determined.

[0158] The dynamic adjustment size corresponding to the face key point in the texture feature map can represent the size of the coordinate range in which the target coordinate of the face key point in the to-be-positioned face image is located in the texture feature map.

[0159] Generally, for each face key point, when the accuracy of the heat map coordinate of the face key point is poor, the offset value between the initial key point coordinate of the heat map coordinate in the to-be-positioned face image and the target coordinate of the face key point in the to-be-positioned face image is large, so that the target texture feature region including the mapping coordinate corresponding to the target mapping coordinate and the target coordinate in the texture feature map is larger, that is, the dynamic adjustment size of the target texture feature region of the target mapping coordinate is larger; and when the accuracy of the heat map coordinate of the face key point is high, the offset value between the initial key point coordinate of the heat map coordinate in the to-be-positioned face image and the target coordinate of the face key point in the to-be-positioned face image is small, so that the target texture feature region including the mapping coordinate corresponding to the target mapping coordinate and the target coordinate in the texture feature map is smaller, that is, the dynamic adjustment size of the target texture feature region of the target mapping coordinate is smaller.

[0160] Wherein, for each face key point, the initial key point coordinate of the face key point can be mapped to the target texture feature region in the texture feature map corresponding to the target mapping coordinate of the face key point, and the dynamic adjustment size of the above target texture feature region can be taken as the dynamic adjustment size corresponding to the face key point in the texture feature map.

[0161] As can be seen from the above, the dynamic adjustment size corresponding to the face key point in the texture feature map is related to the uncertainty of the heat map coordinate of the face key point.

[0162] Based on this, for each face key point, the dynamic adjustment size of the face key point in the texture feature map can be determined based on the uncertainty of the heat map coordinate of the face key point.

[0163] Optionally, the dynamic adjustment size corresponding to the uncertainty can be determined based on a preset relationship between the region size of the texture feature region and the uncertainty of the heat map coordinate.

[0164] S104: For each face key point, in the texture feature map, a target texture feature region centered on the target mapping coordinate is determined according to the dynamic adjustment size;

[0165] For each face key point, after determining the target mapping coordinate corresponding to the initial key point coordinate of the face key point in the texture feature map in the face image to be positioned, and the dynamic adjustment size of the face key point in the texture feature map, the target texture feature region centered on the target mapping coordinate can be determined in the texture feature map according to the dynamic adjustment size.

[0166] Wherein, optionally, in the texture feature map, a square region with the target mapping coordinate as the region center and the dynamic adjustment size as the side length can be determined as the target texture feature region.

[0167] Optionally, in the texture feature map, a circular region with the target mapping coordinate as the center and the dynamic adjustment size as the radius can be determined as the target texture feature region.

[0168] S105: For each face key point, based on the preset correspondence relationship between the texture feature region, the mapping coordinate and the mapping coordinate offset value, an initial offset value corresponding to the target texture feature region and the target mapping coordinate is determined, and based on the mapping relationship, a target offset value corresponding to the initial offset value is determined.

[0169] For each face key point, after determining the above target texture feature region, an initial offset value between the target mapping coordinate and the mapping coordinate corresponding to the target coordinate of the face key point in the face image to be positioned in the texture feature map can be further calculated.

[0170] Thus, for each face key point, an initial offset value corresponding to the target texture feature region and the target mapping coordinate can be determined based on a preset correspondence between the texture feature region, the mapping coordinate, and the mapping coordinate offset value.

[0171] Optionally, the correspondence between the texture feature region, the mapping coordinate, and the mapping coordinate offset value can be an offset value prediction model.

[0172] Based on this, in an optional specific implementation, the step S105 can include the following step 21.

[0173] Step 21: input the target texture feature region and the target mapping coordinate into a preset offset value prediction model corresponding to the face key point, and obtain an offset value output by the offset value prediction model as the initial offset value.

[0174] The offset value prediction model is determined based on a sample texture feature region and a label of the sample texture feature region. The sample texture feature region is a texture feature region in a sample texture feature map determined based on a predicted heat map coordinate and a predicted uncertainty of each face key point obtained by predicting a third sample face image by using a key point prediction model. The label of the sample texture feature region is an offset value of a sample mapping coordinate and a mapping coordinate true value of each face key point in the sample texture feature image in the third sample face image. The sample mapping coordinate is a mapping coordinate of a sample initial key point coordinate in the sample texture feature map determined based on a mapping relationship between the third sample face image and the sample texture feature map. The sample initial key point coordinate is a coordinate corresponding to a predicted heat map coordinate of each face key point obtained by predicting the third sample face image by using the key point prediction model.

[0175] In the specific implementation, the offset value prediction model can be trained in advance based on a sample texture feature region and a label of the sample texture feature region. The sample texture feature region can be a texture feature region determined in a sample texture feature map based on predicted heat map coordinates and predicted uncertainty of each facial landmark obtained by predicting a third sample face image by using the landmark prediction model. The label of the sample texture feature region is an offset value of a sample mapping coordinate and a ground truth mapping coordinate of each facial landmark in the third sample face image in the sample texture feature image. The sample mapping coordinate is a mapping coordinate of a sample initial landmark coordinate in the sample texture feature map based on a mapping relationship between the third sample face image and the sample texture feature map. The sample initial landmark coordinate is a coordinate corresponding to the predicted heat map coordinates of each facial landmark obtained by predicting the third sample face image by using the landmark prediction model in the third sample face image.

[0176] To make the writing clear, the training method of the offset value prediction model provided by the embodiments of the present application will be described in detail below.

[0177] For each facial landmark, the landmark type of different facial landmarks can be different, and the offset value prediction rules between different landmark types have certain differences.

[0178] Therefore, for each landmark type, an offset value prediction model belonging to each landmark type can be trained. For example, an offset value prediction model for predicting the offset value of an eye, a facial landmark, can be trained.

[0179] In this way, for each facial landmark, after obtaining the target mapping coordinate of the initial landmark coordinate of the facial landmark in the face image to be positioned and the target texture feature region of the target mapping coordinate, the target mapping coordinate and the target texture feature region can be input into the preset offset value prediction model corresponding to the facial landmark. Then, the offset value output by the offset value prediction model can be obtained as the initial offset value of the target mapping coordinate of the initial landmark coordinate of the facial landmark in the face image to be positioned and the target texture feature region.

[0180] In addition, the size and position of the facial landmark corresponding to different landmark types are different, and the positioning accuracy requirements for different facial landmarks can also be different. Therefore, for offset value prediction models of different landmark types, the sample size supported by the offset value prediction model is also different.

[0181] For example, the sample size of the offset value prediction model for the nose key point of the face can be 28*28, and the sample size of the offset value prediction model for the forehead key point of the face can be 56*56.

[0182] Therefore, in order to adapt to the sample size of the offset value prediction model and avoid introducing model prediction errors due to size mismatch, generally, the region size of the dynamic adjustment region to be input into the offset value prediction model can be adjusted according to the sample size of the offset value prediction model.

[0183] Based on this, before the step 21, the face key point positioning method provided by the embodiments of the present application can further include the following step 20:

[0184] Step 20: adjusting the region size of the target texture feature region according to the sample size of the preset offset value prediction model;

[0185] Correspondingly, the step 21 can include the following step 211:

[0186] Step 211: inputting the target texture feature region with the adjusted region size and the target mapping coordinates into the preset offset value prediction model corresponding to the face key point.

[0187] In the specific implementation, for each face key point, the preset offset value prediction model corresponding to the face key point can be determined, and the sample size of the offset value prediction model can be determined.

[0188] In this way, the region size of the determined target texture feature region can be adjusted according to the sample size.

[0189] Then, the adjusted target texture feature region and the target mapping coordinates can be input into the offset value prediction model corresponding to the face key point, and the offset value output by the offset value prediction model can be obtained.

[0190] Since the offset value is the offset value of the mapping coordinates of the target mapping coordinates and the target coordinates in the texture feature map in the target texture feature region with the adjusted size, the offset value needs to be inversely adjusted according to the size adjustment ratio of the target texture feature region. In this way, the offset value after the adjustment is the initial offset value of the target texture feature region in the target texture feature region.

[0191] For example, when the region size of the target texture feature region of the human face key point in the texture feature map is 56*56, and the sample size of the offset value prediction model corresponding to the human face key point is 28*28, the target texture feature region can be scaled by 4 times, so that the region size of the scaled target texture feature region is 28*28. Then, the target texture feature region with the region size of 28*28 is input into the offset value prediction model, and the offset value output by the offset value prediction model is obtained. Finally, the offset value is enlarged by 4 times to obtain the initial offset value of the target texture feature region in the target texture feature region.

[0192] In addition, for each human face key point, the dynamic adjustment size of the target texture feature region can also be determined based on the sample size of the offset value prediction model corresponding to the human face key point. In this way, the dynamic adjustment size that meets the sample size of the offset value prediction model can be obtained, so that the determined target texture feature region is a region that meets the sample size of the offset value prediction model.

[0193] Based on this, the step S103 of determining the dynamic adjustment size of the initial key point coordinate based on the uncertainty of the human face key point can include the following step 31:

[0194] Step 31: calculating the dynamic adjustment size corresponding to the human face key point in the texture feature map based on the uncertainty of the human face key point and the preset sample size of the offset value prediction model corresponding to the human face key point.

[0195] In the specific implementation, for each human face key point, when the uncertainty of the human face key point and the preset sample size of the offset value prediction model corresponding to the human face key point are determined, the dynamic adjustment size corresponding to the initial key point coordinate in the texture feature map can be calculated based on the uncertainty and the sample size.

[0196] Optionally, the dynamic adjustment size corresponding to the uncertainty of the human face key point and the sample size of the offset value prediction model can be determined by using a preset correspondence relationship among the uncertainty, the model sample size, and the region adjustment size.

[0197] Optionally, in one specific implementation, the step 31 can include the following step 311:

[0198] Step 311: calculating the dynamic adjustment size corresponding to the human face key point in the texture feature map by using a preset formula;

[0199] The preset formula is:

[0200]

[0201] wherein, ds is a dynamic adjustment size of the initial key point coordinate; k is a sample size of the offset value prediction model; u is an uncertainty of the heat map coordinate of the face key point; sigmoid(u) is a preset nonlinear function about the uncertainty, and the nonlinear function can be expressed as:

[0202]

[0203] In the specific implementation, for each face key point, when determining the uncertainty of the face key point and the preset sample size of the offset value prediction model corresponding to the face key point, the uncertainty and the sample size can be brought into the preset formula, so as to obtain the dynamic adjustment size of the face key point in the texture feature map.

[0204] In this way, by using the uncertainty of the face key point and the sample size of the offset value prediction model corresponding to the face key point, the dynamic adjustment size of the face key point in the texture feature map is calculated, and the target texture feature region is calculated by using the dynamic adjustment size, so that the target texture feature region obtained is a region that meets the sample size of the offset value prediction model, and it is not necessary to adjust the region size of the target texture feature region before inputting the target texture feature region into the offset value prediction model, so that the resolution reduction of the target texture feature region caused by scaling can be avoided. Moreover, the offset value output by the offset value prediction model is the initial offset value corresponding to the target texture feature region and the target mapping coordinate.

[0205] For each face key point, since the initial offset value is the offset value between the target mapping coordinate corresponding to the initial key point coordinate of the face key point in the texture feature map and the mapping coordinate corresponding to the target coordinate of the face key point in the texture feature map, rather than the offset value between the initial key point coordinate and the target coordinate, the initial key point coordinate cannot be directly adjusted by using the initial offset value to obtain the target coordinate.

[0206] Based on this, for each face key point, after determining the initial offset value, the target offset value corresponding to the initial offset value can be determined by using the mapping relationship between the face image to be located and the texture feature map, so that the offset value between the initial key point coordinate and the target coordinate can be obtained. For example, the image size of the face image to be located is 224*224, and the image size of the texture feature map is 112*112, so that after determining the initial offset value, the initial offset value can be expanded by 2 times based on the mapping relationship between the face image to be located and the texture feature map, so as to obtain the target offset value corresponding to the initial offset value.

[0207] S106: For each face key point, determine the target key point coordinate after adjusting the initial key point coordinate according to the target offset value as the target coordinate of the face key point in the face image to be positioned.

[0208] For each face key point, after determining the initial key point coordinate corresponding to the heat map coordinate of the face key point in the face image to be positioned, and the target offset value of the initial key point coordinate and the target coordinate of the face key point in the face image to be positioned, the initial key point coordinate can be adjusted according to the target offset value, and then the adjusted target key point coordinate is obtained. Thus, the target key point coordinate is the target coordinate of the face key point in the face image to be positioned.

[0209] As can be seen from the above, by applying the scheme provided in the embodiments of the present application, for each face key point in the face image to be positioned, the initial key point coordinate and the target offset value of the face key point in the face image to be positioned can be determined by using the heat map coordinate of the face key point and the uncertainty of the heat map coordinate. Then, the face key point in the face image to be positioned can be positioned by using the initial key point coordinate and the target offset value. In this way, by adjusting the initial key point coordinate determined based on the heat map coordinate by using the target offset value determined based on the uncertainty of the heat map coordinate, the coordinate deviation caused by positioning the face key point using the low-precision heat map coordinate can be corrected. Thus, the influence of the difference between the image size of the heat map and the image size of the face image to be positioned, and the loss of high-resolution detailed features in the process of calculating the heat map, etc. on the accuracy of the face key point positioning is reduced, and the accuracy of the face key point positioning is improved.

[0210] Next, the training method of the key point prediction model and the training method of the offset value prediction model involved in the embodiments of the present application are described in detail.

[0211] Optionally, in a specific implementation manner, as shown in Figure 3 the training method of the key point prediction model can include the following steps S301-S304:

[0212] S301: Training a preset first sub-model by taking a first sample face image as input and taking the sample heat map coordinates of each face key point in the first sample face image as output;

[0213] S302: When the training of the first sub-model satisfies a specified training condition, training a preset second sub-model by taking a second sample face image as input and taking the model prediction difference of the second sample face image as output;

[0214] S303: If the loss function of the second sub-model satisfies the preset stopping condition, stop training to obtain the key point prediction model including the trained heat map sub-model and the uncertainty sub-model.

[0215] The heat map sub-model is the trained first sub-model, and the uncertainty sub-model is the trained second sub-model.

[0216] S304: If the loss function of the second sub-model does not satisfy the preset stopping condition, return to the step of training the preset first sub-model by taking the first sample face image as input and taking the sample heat map coordinates of each facial key point in the first sample face image as output.

[0217] In the specific implementation manner, when training the key point prediction model, a plurality of first sample face images, sample heat map coordinates of each facial key point in each first sample face image, a plurality of second sample face images, and sample heat map coordinates of each facial key point in each second sample face image can be acquired.

[0218] Then, the preset first sub-model can be trained by taking the first sample face image as input and taking the sample heat map coordinates of each facial key point in the first sample face image as output. When the training of the first sub-model satisfies a specified training condition, the training of the first sub-model is paused, the second sample face image is input into the first sub-model, and a prediction result of the first sub-model in heat map coordinate prediction on the second sample face image is obtained. The prediction result is the predicted heat map coordinates of each facial key point in the second sample face image.

[0219] The specified training condition can be set according to actual needs, which can be that the training duration reaches a specified duration, or that the iteration number of each first sample face image reaches a first preset number, or that the iteration number of all first sample face images reaches a second preset number, all of which are reasonable and are not limited in the embodiments of the present application.

[0220] Then, for each second sample face image, a difference between the prediction result of each facial key point in the second sample face image and the sample heat map coordinates of each facial key point in the second sample face image is calculated as a model prediction difference of the second sample face image.

[0221] Then, the preset second sub-model can be trained by taking the second sample face image as input and taking the model prediction difference of the second sample face image as output.

[0222] Further, it can be determined whether the loss function of the second sub-model satisfies a first preset stopping condition. The first preset stopping condition can be set according to actual needs, and can be that the result of the loss function of the second sub-model is not greater than a preset threshold. The preset threshold can be 0.1 or 0.5, which are both reasonable, and are not specifically limited in the embodiments of the present application.

[0223] If the loss function of the second sub-model satisfies the first preset stopping condition, the training of the first sub-model and the second sub-model can be stopped, so as to obtain a key point prediction model including a heat map sub-model trained by the first sub-model and an uncertainty sub-model trained by the second sub-model.

[0224] If the loss function of the second sub-model satisfies the first preset stopping condition, the step of training the first sub-model with the first sample face image as input and the sample heat map coordinates of each face key point in the first sample face image as output can be returned to step S301, so as to continue training the first sub-model and the second sub-model until the loss function of the second sub-model satisfies the first preset stopping condition.

[0225] In this way, by alternately training the first sub-model and the second sub-model, the training result of the first sub-model can be verified by the second sub-model, without the need for additional test samples to verify the training result of the first sub-model, thereby improving the efficiency of model training.

[0226] The uncertainty represents the confidence of the second sub-model in the prediction result of the first sub-model. The higher the uncertainty, the lower the confidence of the second sub-model in the prediction result of the first sub-model, and thus the greater the error of the preset result of the first sub-model.

[0227] The loss function Loss is the optimization target of the model training of the second sub-model, and is related to the uncertainty.

[0228] When training the second sub-model, for the second sample face image x input to the second sub-model, the prediction result of the first sub-model for the second sample face image is denoted as The standard deviation of the model prediction difference is σ, and the maximum likelihood estimation of the loss function of the second sub-model can be represented as:

[0229]

[0230] wherein, p(y|x, θ) represents the possibility of occurrence of the observation result y under the condition that x is known and θ is known; here, specifically, the condition that x is known represents the model prediction result that is known and the standard deviation σ of the prediction result.

[0231] Assuming that the model prediction difference obeys a Gaussian distribution, the above formula can be further transformed into:

[0232]

[0233] wherein, represents a Gaussian distribution with as the mean value and as the variance;

[0234] Since the maximum likelihood estimation is equivalent to the minimization of the loss, the loss function of the second sub-model can be further simplified as:

[0235]

[0236] Using to replace, the loss function of the second sub-model is further transformed into:

[0237]

[0238] In this way, the uncertainty can be represented as:

[0239]

[0240] It should be noted that the electronic device used for model training and the electronic device for executing the face key point positioning method provided in the embodiments of the present application can be the same electronic device, or can be different electronic devices.

[0241] Optionally, in a specific implementation manner, as shown in Figure 4 , the training manner of the offset value prediction model can include the following steps S401-S405:

[0242] S401: Obtain the prediction heat map coordinates and prediction uncertainty of each face key point obtained by the key point prediction model by predicting the third sample face image, and obtain the sample texture feature map of the third sample face image;

[0243] S402: For each face key point, determine the sample initial key point coordinates corresponding to the prediction heat map coordinates of the face key point in the third sample face image, and determine the sample mapping coordinates corresponding to the sample initial key point coordinates in the sample texture feature map based on the mapping relationship between the third sample face image and the sample texture feature map;

[0244] S403: For each face key point, based on the prediction uncertainty of the face key point, a sample dynamic adjustment size corresponding to the face key point in the sample texture feature map is determined, and a sample texture feature region centered on the sample mapping coordinates is determined in the sample texture feature map according to the sample dynamic adjustment size;

[0245] S404: For each face key point, the offset value of the sample mapping coordinates and the mapping coordinate true value of the face key point in the sample texture feature map is calculated as the label of the sample texture feature region.

[0246] S405: When the initial model meets the second preset stopping condition, the training is stopped, and the offset value prediction model is obtained.

[0247] In the specific implementation manner, after the key point prediction model is trained, the offset value prediction model can be trained by using the key point prediction model. When the offset value prediction model is trained, the third sample face image, the sample texture feature map of the third sample face image, and the mapping coordinate true value of each face key point in the third sample face image in the sample texture feature map can be obtained.

[0248] Then, the third sample face image can be input into the key point prediction model, and the prediction heat map coordinates and the prediction uncertainty of each face key point obtained by the key point prediction model in predicting the third sample face image can be obtained.

[0249] Then, for each face key point, the sample initial key point coordinates corresponding to the prediction heat map coordinates of the face key point in the third sample face image can be determined, and based on the mapping relationship between the third sample face image and the sample texture feature map, the sample mapping coordinates corresponding to the sample initial key point coordinates in the sample texture feature map can be determined.

[0250] Then, for each face key point, the sample dynamic adjustment size corresponding to the face key point in the sample texture feature map can be determined based on the prediction uncertainty of the face key point. Then, in the sample texture feature map, the sample texture feature region centered on the sample mapping coordinates is determined according to the sample dynamic adjustment size, and the offset value of the sample mapping coordinates and the mapping coordinate true value of the face key point in the sample texture feature map is calculated as the label of the sample texture feature region.

[0251] In this way, the preset initial model can be trained with the sample texture feature region as the output and the label of the sample texture feature region as the output.

[0252] Finally, when the initial model meets the second preset stopping condition, the initial model can be stopped training to obtain the offset value prediction model.

[0253] Optionally, the second preset stopping condition can be that the iteration number of each sample texture feature region reaches a preset number.

[0254] Optionally, the second preset stopping condition can be that the error between the true value and the predicted value of each sample texture feature region and the label of each sample texture feature region is less than a preset error.

[0255] In this way, after the target texture feature region and the target mapping coordinates are determined, the target texture feature region and the target mapping coordinates can be input into the offset value prediction model, so that the offset value prediction model learns the target texture feature region and the target mapping coordinates, and outputs an initial offset value corresponding to the target texture feature region and the target mapping coordinates.

[0256] It should be noted that the electronic device for model training and the electronic device for performing the face key point positioning method provided in the embodiments of the present application can be the same electronic device or different electronic devices.

[0257] In addition, for each key point type, an offset value prediction model related to the key point type can be trained respectively, so as to obtain an offset value prediction model corresponding to each key point type.

[0258] In addition, in order to improve the clarity of the local texture features of the sample texture feature region input into the offset value prediction model, the sample size of the sample texture feature region input into the offset value prediction model can be constrained.

[0259] Optionally, when the sample size of the sample texture feature region is too large, the range of the sample texture feature region is too large, and non-local information is easily introduced, which interferes with the prediction result of the model; when the sample size of the sample texture feature region is too small, the range of the sample texture feature region is too small, and the texture information around the true value point required can not be contained, so that the offset amount cannot be obtained. Based on this, the sample size of the sample texture feature region can be constrained.

[0260] Since the sample size of the sample texture feature region is calculated based on the uncertainty of the face key point, the sample size of the sample texture feature region can be constrained by constraining the uncertainty of the face key point.

[0261] For example, the sample size of the sample texture feature region can be constrained by constraining the uncertainty of the face key point. Figure 5As shown, for a certain face key point, the offset value between the initial key point coordinate E of the face key point in the face image and the coordinate true value F of the face key point is D, so that through multiple experiments, it is found that when the sample texture feature region sample size ds / 2 = 1.5D, better face key point detection effect can be obtained, and then the sample texture feature region sample size ds / 2 = 1.5D can determine the sample texture feature region sample size ds = 3D, and 510 in the figure is the determined sample texture feature region. 3D can be used as the constraint value of the sample size ds, and since there is a correlation between the uncertainty and the sample size, the purpose of constraining the uncertainty u can be achieved by setting the constraint value of ds.

[0262] In order to facilitate understanding of the face key point positioning method provided by the embodiments of the present application, the following will be described in conjunction with specific examples.

[0263] The face image to be positioned is a 224*224 face image, as shown in Figure 6 As shown, the process of positioning the face key point of the face image to be positioned can include the following steps S601-S613:

[0264] S601: input the face image to be positioned with an image size of 224*224;

[0265] S602: extract texture features to obtain a texture feature image of C1*224*224;

[0266] S603: downsample the texture feature image;

[0267] S604: extract depth features to obtain a depth feature image of C2*56*56;

[0268] S605: output a heat map;

[0269] S606: output an uncertainty;

[0270] S607: determine a dynamic center;

[0271] S608: determine a dynamic size using the uncertainty;

[0272] S609: determine a dynamic feature region;

[0273] S610: reduce the dynamic region;

[0274] S611: determine an initial offset value;

[0275] S612: determine a target offset value;

[0276] S613: Locate each facial key point in the face image to be positioned by using the target offset value and the heat map coordinates.

[0277] The input image size of the face image to be positioned is 224*224. Then, the texture features of the face image to be positioned are extracted by using a convolution layer with a step size of 1 to obtain a texture feature image with a size of C1*224*224, and the texture feature image is down-sampled by using multiple convolution layers and pooling layers.

[0278] Then, the texture feature image after down-sampling is subjected to deep feature extraction by using a convolution layer with a step size of 2 to obtain a deep feature image with a size of C2*56*56.

[0279] Then, the deep feature image is input into a preset key point prediction model, and a heat map of 68*56*56 and an uncertainty of 1*68 about 68 facial key points of the face image to be positioned output by the key point prediction model are obtained.

[0280] Then, for each facial key point, the initial key point coordinates of the facial key point in the face image to be positioned are determined by using the heat map coordinates in the heat map of the facial key point, and the target mapping coordinates of the initial key point coordinates in the texture feature image are determined as the dynamic center by using the mapping relationship between the face image to be positioned and the texture feature image.

[0281] Since the image sizes of the face image to be positioned and the texture feature image are the same, the initial key point coordinates and the target mapping coordinates are the same.

[0282] For each facial key point, the dynamic size of the facial key point in the texture feature image is determined by using the uncertainty of the facial key point. Then, a dynamic region with the dynamic center and the dynamic size can be determined in the texture feature image.

[0283] For each facial key point, the dynamic region is reduced by θ times according to the sample size of the offset value prediction model corresponding to the facial key point to obtain a reduced dynamic region, and the dynamic region is input into the offset value prediction model to obtain the initial offset value output by the offset value prediction model.

[0284] For each face key point, the initial offset value can be enlarged by θ times, and the mapping relationship between the face image to be positioned and the texture feature image is used to determine the target offset value. Finally, the initial key point coordinates are adjusted according to the target offset value to obtain the target coordinates of the face key point. The obtained target coordinates are the coordinates of the face key point in the face image to be positioned. Thus, after determining the target coordinates of each face key point in the face image to be positioned, the positioning of each face key point in the face image to be positioned is completed.

[0285] Based on the same inventive concept, the embodiment of the present application provides a face key point positioning method as shown in the above Figure 1 The embodiment of the present application also provides a face key point positioning device.

[0286] Figure 7 A structural schematic diagram of a face key point positioning device provided by the embodiment of the present application is shown in the above Figure 7 The device can include the following modules:

[0287] The heat map coordinate acquisition module 710 is configured to acquire the texture feature map of the face image to be positioned, and the heat map coordinates and the uncertainty of each face key point in the face image to be positioned. The uncertainty of each face key point is used to represent the accuracy of the heat map coordinates of the face key point.

[0288] The key point coordinate determination module 720 is configured to determine, for each face key point, the initial key point coordinates corresponding to the heat map coordinates of the face key point in the face image to be positioned.

[0289] The size determination module 730 is configured to determine, for each face key point, the dynamic adjustment size corresponding to the face key point in the texture feature map based on the uncertainty of the face key point, and determine the target mapping coordinates of the initial key point coordinates in the texture feature map based on the mapping relationship between the face image to be positioned and the texture feature map.

[0290] The region determination module 740 is configured to determine, for each face key point, the target texture feature region centered at the mapping coordinates in the texture feature map according to the dynamic adjustment size.

[0291] The offset value determination module 750 is configured to determine, for each face key point, the initial offset value corresponding to the target texture feature region and the target mapping coordinates based on the preset correspondence relationship among the texture feature region, the mapping coordinates and the mapping coordinate offset value, and determine the target offset value corresponding to the initial offset value based on the mapping relationship.

[0292] The coordinate determination module 760 is configured to determine, for each facial key point, target key point coordinates of the initial key point coordinates adjusted according to the target offset value as the target coordinates of the facial key point in the face image to be positioned.

[0293] As can be seen from the above, by applying the scheme provided in the embodiments of the present application, for each facial key point in the face image to be positioned, the initial key point coordinates and the target offset value of the facial key point in the face image to be positioned can be determined by using the heat map coordinates and the uncertainty of the heat map coordinates of the facial key point. Then, the facial key point can be positioned in the face image to be positioned by using the initial key point coordinates and the target offset value. In this way, by adjusting the initial key point coordinates determined based on the heat map coordinates by using the target offset value determined based on the uncertainty of the heat map coordinates, the coordinate deviation caused by positioning the facial key point by using the low-precision heat map coordinates can be corrected. Thus, the influence of the difference between the heat map and the image size of the face image to be positioned and the loss of high-resolution detailed features in the process of calculating the heat map on the accuracy of the facial key point positioning is reduced, and the accuracy of the facial key point positioning is improved.

[0294] Optionally, in a specific implementation manner, the heat map coordinate acquisition module 710 is specifically configured to:

[0295] input the face image to be positioned into a preset key point prediction model; wherein the key point prediction model comprises a heat map sub-model and an uncertainty sub-model;

[0296] acquire the heat map coordinates of each facial key point output by the heat map sub-model and the uncertainty of each facial key point output by the uncertainty sub-model;

[0297] wherein the key point prediction model is obtained by training a preset first sub-model based on a first sample face image and sample heat map coordinates of each facial key point in the first sample face image, and training a preset second sub-model based on a second sample face image and a model prediction difference of the second sample face image; the model prediction difference is a difference between a prediction result of heat map coordinate prediction of the second sample face image by using the first sub-model and the sample heat map coordinates of each facial key point in the second sample face image; the first sample face image and the second sample face image are different images.

[0298] Optionally, in a specific implementation manner, the apparatus further comprises a key point prediction model training module, and the key point prediction model training module is specifically configured to:

[0299] training a first sub-model by taking a first sample face image as input and taking sample heat map coordinates of each face key point in the first sample face image as output;

[0300] training a second sub-model by taking a second sample face image as input and taking model prediction difference of the second sample face image as output when the training of the first sub-model meets a specified training condition;

[0301] stopping the training and obtaining a key point prediction model including a trained heat map sub-model and an uncertainty sub-model if a loss function of the second sub-model meets a first preset stopping condition, wherein the heat map sub-model is the trained first sub-model and the uncertainty sub-model is the trained second sub-model;

[0302] Otherwise, returning to the step of training the first sub-model by taking the first sample face image as input and taking the sample heat map coordinates of each face key point in the first sample face image as output.

[0303] Optionally, in a specific implementation manner, the offset value determination module 750 includes:

[0304] an input sub-module, configured to input the target texture feature region and the target mapping coordinate into a preset offset value prediction model corresponding to the face key point, and obtain an offset value output by the offset value prediction model as an initial offset value;

[0305] wherein the offset value prediction model is determined based on a sample texture feature region and a label of the sample texture feature region, the sample texture feature region is a texture feature region determined in a sample texture feature map based on predicted heat map coordinates and predicted uncertainty of each face key point obtained by predicting the third sample face image by using the key point prediction model, and the label of the sample texture feature region is an offset value of a sample mapping coordinate and a mapping coordinate true value of each face key point in the third sample face image in the sample texture feature image, the sample mapping coordinate is a mapping coordinate of a sample initial key point coordinate in the sample texture feature map determined based on a mapping relationship between the third sample face image and the sample texture feature map, and the sample initial key point coordinate is a coordinate corresponding to the predicted heat map coordinates of each face key point obtained by predicting the third sample face image by using the key point prediction model in the third sample face image.

[0306] Optionally, in a specific implementation manner, the apparatus further includes a training module of an offset value prediction model, and the training module of the offset value prediction model is specifically configured to:

[0307] obtain a prediction heat map coordinate and a prediction uncertainty of each facial landmark point obtained by predicting the third sample face image by using the key point prediction model, and obtain a sample texture feature map of the third sample face image;

[0308] For each facial landmark point, determine a sample initial landmark coordinate corresponding to the prediction heat map coordinate of the facial landmark point in the third sample face image, and determine a sample mapping coordinate corresponding to the sample initial landmark coordinate in the sample texture feature map based on a mapping relationship between the third sample face image and the sample texture feature map;

[0309] For each facial landmark point, determine a sample dynamic adjustment size corresponding to the facial landmark point in the sample texture feature map based on the prediction uncertainty of the facial landmark point, and determine a sample texture feature region centered at the sample mapping coordinate in the sample texture feature map according to the sample dynamic adjustment size;

[0310] For each facial landmark point, calculate an offset value of the sample mapping coordinate and a mapping coordinate ground truth of the facial landmark point in the sample texture feature map as a label of the sample texture feature region;

[0311] train a preset initial model with the sample texture feature region as the output and the label of the sample texture feature region as the output;

[0312] stop training when the initial model meets a second preset stopping condition, and obtain an offset value prediction model.

[0313] Optionally, in a specific implementation manner, the size determination module 730 includes:

[0314] The calculation sub-module is configured to calculate a dynamic adjustment size corresponding to the facial landmark point in the texture feature map based on the uncertainty of the facial landmark point and a sample size of the preset offset value prediction model.

[0315] Optionally, in a specific implementation manner, the apparatus further includes:

[0316] The region size adjustment module is configured to adjust a region size of the target texture feature region according to a sample size of the preset offset value prediction model before inputting the target texture feature region and the target mapping coordinate into the offset value prediction model corresponding to the facial landmark point.

[0317] The input sub-module is specifically configured to:

[0318] The target texture feature region with the adjusted region size and the target mapping coordinate are input into a preset offset value prediction model corresponding to the face key point.

[0319] Optionally, in a specific implementation, the calculation sub-module is specifically configured to:

[0320] The preset formula is used to calculate a dynamic adjustment size of the face key point in the texture feature map, wherein the preset formula is:

[0321]

[0322] wherein, ds is the dynamic adjustment size of the face key point in the texture feature map; u is an uncertainty of the heat map coordinate of the face key point; sigmoid(u) is a preset nonlinear function about the uncertainty; and k is a sample size of the offset value prediction model.

[0323] The embodiments of the present application also provide an electronic device, such as Figure 8 as shown in the drawings, comprising:

[0324] The memory 801 is used to store a computer program.

[0325] The processor 802 is used to execute the program stored in the memory 801, and realize the face key point positioning method described above.

[0326] The electronic device described above can also include a communication bus and / or a communication interface, and the processor 802, the communication interface and the memory 801 can complete mutual communication through the communication bus.

[0327] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0328] The communication interface is used for communication between the electronic device and other devices.

[0329] The memory can include a Random Access Memory (RAM) and can also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0330] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0331] In another embodiment provided in the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the face key point positioning methods described above.

[0332] In another embodiment provided in the present application, a computer program product containing instructions is provided, and when the computer program product is executed on a computer, the computer is caused to execute the face key point positioning method in any of the above embodiments.

[0333] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, DVD), or a solid state disk (SSD) and the like.

[0334] It is to be noted that, in the present document, relational terms such as first and second, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily implying any actual relationship or order between such entities or actions. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0335] Each of the above-described embodiments can be implemented in a variety of ways, and the same or similar parts among the embodiments can be mutually referred to. Each of the embodiments focuses on a difference from other embodiments. In particular, the device embodiments, the electronic device embodiments, the computer-readable storage medium embodiments, and the computer program product embodiments are described simply because they are substantially similar to the method embodiments.

[0336] The preferred embodiments of the present application have been described above with the aid of drawing figures, and are not limited to those embodiments; instead, they will include any modifications and variations of the preferred embodiments. It is intended that the application encompass all such modifications and variations as fall within the scope of the appended claims.

Claims

1. A method for locating facial key points, characterized in that, The method includes: The texture feature map of the face image to be located is obtained, as well as the heat map coordinates and uncertainty of each facial key point in the face image to be located; wherein, the uncertainty of each facial key point is used to characterize the accuracy of the heat map coordinates of the facial key point; For each facial landmark, perform the following steps: In the face image to be located, determine the initial key point coordinates corresponding to the heatmap coordinates of the face key points; Based on the uncertainty of the facial key point, the dynamic adjustment size of the facial key point corresponding to the texture feature map is determined, and based on the mapping relationship between the face image to be located and the texture feature map, the target mapping coordinates of the initial key point coordinates in the texture feature map are determined. In the texture feature map, the target texture feature region centered on the target mapping coordinates is determined according to the dynamically adjusted size; Based on the preset correspondence between texture feature regions, mapping coordinates, and mapping coordinate offset values, an initial offset value corresponding to the target texture feature region and the target mapping coordinates is determined, and based on the mapping relationship, a target offset value corresponding to the initial offset value is determined. The initial key point coordinates are adjusted according to the target offset value to determine the target key point coordinates, which are then used as the target coordinates of the facial key point in the face image to be located.

2. The method according to claim 1, characterized in that, The process of obtaining the heatmap coordinates and uncertainty of each facial key point in the image of the face to be located includes: The face image to be located is input into a preset key point prediction model; wherein, the key point prediction model includes a heatmap sub-model and an uncertainty sub-model; Obtain the heatmap coordinates of each facial key point output by the heatmap sub-model and the uncertainty of each facial key point output by the uncertainty sub-model; The key point prediction model is obtained by training a preset first sub-model based on the sample heatmap coordinates of each facial key point in the first sample face image, and by training a preset second sub-model based on the model prediction difference between the second sample face image and the second sample face image. The model prediction difference is the difference between the prediction result of the heatmap coordinate prediction of the second sample face image using the first sub-model and the sample heatmap coordinates of each facial key point in the second sample face image. The first sample face image and the second sample face image are different images.

3. The method according to claim 2, characterized in that, The training methods for the key point prediction model include: The first sample face image is used as input, and the sample heatmap coordinates of each facial key point in the first sample face image are used as output to train the preset first sub-model. When the training of the first sub-model meets the specified training conditions, the second sample face image is used as input and the model prediction difference of the second sample face image is used as output to train the preset second sub-model. If the loss function of the second sub-model satisfies the first preset stopping condition, then training stops, and a key point prediction model including a trained heatmap sub-model and an uncertainty sub-model is obtained; wherein, the heatmap sub-model is the trained first sub-model, and the uncertainty sub-model is the trained second sub-model; Otherwise, return to the step of training the preset first sub-model by taking the first sample face image as input and taking the sample heatmap coordinates of each face key point in the first sample face image as output.

4. The method according to claim 2 or 3, characterized in that, The determination of the initial offset value corresponding to the target texture feature region and the target mapped coordinates based on the preset correspondence between texture feature regions, mapped coordinates, and mapped coordinate offset values ​​includes: The target texture feature region and the target mapping coordinates are input into the preset offset value prediction model corresponding to the facial key point, and the offset value output by the offset value prediction model is obtained as the initial offset value. The offset prediction model is determined based on the sample texture feature region and its label. The sample texture feature region is the texture feature region determined in the sample texture feature map of the third sample face image by the predicted heatmap coordinates and prediction uncertainty of each facial key point obtained by predicting the third sample face image based on the key point prediction model. The label of the sample texture feature region is the offset value of the sample mapping coordinates and the true values ​​of the mapping coordinates of each facial key point in the third sample face image in the sample texture feature map. The sample mapping coordinates are the mapping coordinates of the initial key point coordinates of the sample determined based on the mapping relationship between the third sample face image and the sample texture feature map in the sample texture feature map. The initial key point coordinates of the sample are the coordinates corresponding to the predicted heatmap coordinates of each facial key point obtained by predicting the third sample face image based on the key point prediction model in the third sample face image.

5. The method according to claim 4, characterized in that, The training methods for the offset prediction model include: Obtain the predicted heatmap coordinates and prediction uncertainty of each facial key point obtained by the key point prediction model in predicting the third sample face image, and obtain the sample texture feature map of the third sample face image. For each facial key point, in the third sample face image, the initial key point coordinates of the sample corresponding to the predicted heatmap coordinates of the facial key point are determined, and based on the mapping relationship between the third sample face image and the sample texture feature map, the sample mapping coordinates of the initial key point coordinates of the sample are determined in the sample texture feature map. For each facial key point, based on the prediction uncertainty of the facial key point, the dynamic adjustment size of the sample corresponding to the facial key point in the sample texture feature map is determined, and in the sample texture feature map, the sample texture feature region centered on the sample mapping coordinates is determined according to the dynamic adjustment size of the sample. For each facial key point, calculate the offset value between the sample mapping coordinates and the ground truth value of the mapping coordinates of the facial key point in the sample texture feature map, and use it as the label of the sample texture feature region; The preset initial model is trained using the sample texture feature region as input and the label of the sample texture feature region as output. Training stops when the initial model meets the second preset stopping condition, and the offset prediction model is obtained.

6. The method according to claim 4, characterized in that, The step of determining the dynamic adjustment size of the facial key point in the texture feature map based on the uncertainty of the facial key point includes: Based on the uncertainty of the facial key point and the sample size of the preset offset prediction model, the dynamic adjustment size of the facial key point in the texture feature map is calculated.

7. The method according to claim 6, characterized in that, Before inputting the target texture feature region and the target mapping coordinates into a preset offset prediction model corresponding to the facial key point, the method further includes: Based on the sample size of the prediction model according to the preset offset value, adjust the region size of the target texture feature region; The step of inputting the target texture feature region and the target mapping coordinates into a preset offset prediction model corresponding to the facial key point includes: The target texture feature region after the region size adjustment and the target mapping coordinates are input into the preset offset value prediction model corresponding to the facial key point.

8. The method according to claim 7, characterized in that, The calculation of the dynamic adjustment size of the facial key point in the texture feature map based on the uncertainty of the facial key point and the sample size of the preset offset prediction model includes: Using a preset formula, the dynamic adjustment size of the facial key points corresponding to the texture feature map is calculated; wherein, the preset formula is: Where ds is the dynamic adjustment size of the facial key point in the texture feature map; u is the uncertainty of the heatmap coordinates of the facial key point; sigmoid(u) is a preset nonlinear function with respect to the uncertainty; and k is the sample size of the offset prediction model.

9. A facial landmark localization device, characterized in that, The device includes: The heatmap coordinate acquisition module is used to acquire the texture feature map of the face image to be located, as well as the heatmap coordinates and uncertainties of each facial key point in the face image to be located; wherein, the uncertainty of each facial key point is used to characterize the accuracy of the heatmap coordinates of the facial key point. The key point coordinate determination module is used to determine the initial key point coordinates corresponding to the heatmap coordinates of each facial key point in the face image to be located. The size determination module is used to determine the dynamic adjustment size of each facial key point in the texture feature map based on the uncertainty of the facial key point, and to determine the target mapping coordinates of the initial key point coordinates in the texture feature map based on the mapping relationship between the face image to be located and the texture feature map. The region determination module is used to determine, for each facial key point, a target texture feature region centered on the mapped coordinates in the texture feature map based on the dynamically adjusted size. The offset value determination module is used to determine, for each facial key point, an initial offset value corresponding to the target texture feature region and the target mapping coordinates based on a preset correspondence between texture feature regions, mapping coordinates and mapping coordinate offset values, and to determine the target offset value corresponding to the initial offset value based on the mapping relationship. The coordinate determination module is used to determine the target key point coordinates after adjusting the initial key point coordinates according to the target offset value for each facial key point, and use these as the target coordinates of the facial key point in the face image to be located.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-8.

Images