A face key point positioning method, a task response method and device

Abstract

This invention provides a method, task response method, and apparatus for locating facial key points, relating to the field of computer technology. The specific implementation scheme is as follows: 1. Determine sparse key points in a facial image; 2. Based on the sparse key points, segment the facial image into local regions to obtain local images; 3. Utilize the local images and a multi-level key point prediction model to determine key point prediction results; wherein, in the multi-level key point prediction model, the input of the first-level model includes a first image obtained by scaling the resolution of the local image, and the input of other levels includes a second image obtained by scaling the resolution of a specified image; furthermore, the local resolution of the input image and the number of key points output by each level model increase progressively; 4. Based on the key point prediction results, determine dense key points in the facial image. Therefore, this scheme can improve the localization accuracy of dense key points.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to the field of key point localization technology, specifically to a method, task response method and device for locating facial key points. Background Technology

[0002] Facial landmark localization technology is a popular area of ​​research in face-related applications, with different applications having varying requirements for the accuracy of these landmarks. For example, in the field of facial recognition, only a small number of facial landmarks need to be identified from a facial image, and the accuracy requirement for these landmarks is relatively low; while in the field of beauty and makeup, a dense number of facial landmarks may be needed to accurately guide the adjustment of the facial features.

[0003] In related technologies, the method for locating dense key points on the face is usually the same as the method for locating sparse key points on the face, which results in low accuracy for locating dense key points.

[0004] Therefore, improving the positioning accuracy of dense key points has become an urgent technical problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method and apparatus for locating facial key points, thereby improving the positioning accuracy of dense key points. Furthermore, this invention also provides a task response method for facial images to meet the needs of tasks requiring high processing accuracy. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide a method for locating facial key points, the method comprising:

[0007] Identify sparse keypoints in facial images;

[0008] Based on the sparse keypoints, the facial image is segmented into local regions to obtain a local image; wherein, the local region is any facial feature region;

[0009] Using the local image and a multi-level keypoint prediction model, the keypoint prediction result is determined. In the multi-level keypoint prediction model, the input to the first-level model includes a first image obtained by scaling the resolution of the local image; the input to other levels includes a second image obtained by scaling the resolution of a specified image, which is an image obtained by segmenting the facial image using the keypoints output by the previous level model. Furthermore, the local resolution of the input image and the number of output keypoints for each level model increase progressively.

[0010] Based on the key point prediction results, dense key points in the facial image are determined.

[0011] Optionally, the step of performing local region segmentation on the facial image based on the sparse keypoints to obtain a local image includes:

[0012] The sparse keypoints are mapped onto the facial image to obtain initial keypoints;

[0013] Based on the facial features represented by the initial key points, the facial image is segmented locally to obtain a local image.

[0014] Optionally, the step of performing local region segmentation on the facial image based on the sparse keypoints to obtain a local image includes:

[0015] Identify the facial features area where key point localization is to be performed, and designate it as the designated area;

[0016] Based on the sparse keypoints belonging to the specified region, the facial image is segmented for the specified region to obtain a local image.

[0017] Optionally, the input content of each level of the multi-level key point prediction model may further include: auxiliary information;

[0018] The auxiliary information is used to characterize the facial contour information determined based on the key points in the image corresponding to the model at that level; the image corresponding to the first-level model is the first image, and the images corresponding to other-level models are the second images.

[0019] Optionally, the auxiliary information includes a contour image;

[0020] The background area of ​​the contour image is a blank image, and the foreground area of ​​the contour image is a contour line determined based on the facial contour information.

[0021] Optionally, determining the keypoint prediction result using the local image and the multi-level keypoint prediction model includes:

[0022] The local image is scaled up to obtain the first image;

[0023] The first image is input into the first-level model of the multi-level key point prediction model to obtain the key points to be utilized.

[0024] Use the next level model of the first level model as the current model;

[0025] Based on the key points to be utilized, the facial image is segmented into local regions to obtain a specified image, and the specified image is then scaled to obtain a second image.

[0026] The second image is input into the current model to obtain key points, which are used as the key points to be used. The next level model of the current model is used as the new current model. The process of performing local region segmentation on the facial image based on the key points to be used is then returned to obtain the specified image.

[0027] The key points output by the last-level model are determined as the key point prediction results.

[0028] Optionally, after inputting the second image into the current model to obtain key points as the currently utilized key points, and before using the next level model of the current model as the new current model, the method further includes:

[0029] Determine whether the number of keypoints output by the current model is lower than the number of keypoints represented by a specified requirement; wherein, the specified requirement includes a requirement for the number of keypoints;

[0030] If it is lower, then the step of taking the next level model of the current model as the new current model is executed;

[0031] If it is not lower than the threshold, then based on the currently available key points, the dense key points in the facial image are determined.

[0032] Secondly, embodiments of the present invention provide a task response method for facial images, the method comprising:

[0033] Define the processing task for facial images;

[0034] According to the above method for locating facial key points, the facial key points in the facial image are located to obtain dense key points in the facial image.

[0035] The processing task is performed on the facial image using dense key points in the facial image.

[0036] Thirdly, embodiments of the present invention provide a facial key point localization device, the device comprising:

[0037] The determination module is used to determine sparse keypoints in facial images;

[0038] The segmentation module is used to perform local region segmentation on the facial image based on the sparse keypoints to obtain a local image; wherein, the local region is any facial feature region;

[0039] The prediction module is used to determine the keypoint prediction result using the local image and the multi-level keypoint prediction model. The multi-level keypoint prediction model's input includes a first image obtained by scaling the resolution of the local image, and the inputs of other levels include a second image obtained by scaling the resolution of a specified image. The specified image is an image obtained by segmenting the facial image using the keypoints output by the previous level model. Furthermore, the local resolution of the input image and the number of output keypoints for each level model increase progressively.

[0040] The output module is used to determine the dense key points in the facial image based on the key point prediction results.

[0041] Optionally, the segmentation module includes:

[0042] The mapping submodule is used to map the sparse key points onto the facial image to obtain initial key points;

[0043] The first segmentation submodule is used to perform local region segmentation on the facial image based on the facial feature regions represented by the initial key points to obtain a local image.

[0044] Optionally, the segmentation module includes:

[0045] The determination submodule is used to determine a facial region for key point localization, which is then designated as the specified region.

[0046] The second segmentation submodule is used to segment the facial image for the specified region based on the sparse keypoints belonging to the specified region, and obtain a local image.

[0047] Optionally, the input content of each level of the multi-level key point prediction model may further include: auxiliary information;

[0048] The auxiliary information is used to characterize the facial contour information determined based on the key points in the image corresponding to the model at that level; the image corresponding to the first-level model is the first image, and the images corresponding to other-level models are the second images.

[0049] Optionally, the auxiliary information includes a contour image;

[0050] The background area of ​​the contour image is a blank image, and the foreground area of ​​the contour image is a contour line determined based on the facial contour information.

[0051] Optionally, the prediction module is specifically used for:

[0052] The local image is scaled up to obtain the first image;

[0053] The first image is input into the first-level model of the multi-level key point prediction model to obtain the key points to be utilized.

[0054] Use the next level model of the first level model as the current model;

[0055] Based on the key points to be utilized, the facial image is segmented into local regions to obtain a specified image, and the specified image is then scaled to obtain a second image.

[0056] The second image is input into the current model to obtain key points, which are used as the key points to be used. The next level model of the current model is used as the new current model. The process of performing local region segmentation on the facial image based on the key points to be used is then returned to obtain the specified image.

[0057] The key points output by the last-level model are determined as the key point prediction results.

[0058] Optionally, after inputting the second image into the current model to obtain key points as the currently utilized key points, and before using the next-level model of the current model as the new current model, the device is further configured to:

[0059] Determine whether the number of keypoints output by the current model is lower than the number of keypoints represented by a specified requirement; wherein, the specified requirement includes a requirement for the number of keypoints;

[0060] If it is lower, then the step of taking the next level model of the current model as the new current model is executed;

[0061] If it is not lower than the threshold, then based on the currently available key points, the dense key points in the facial image are determined.

[0062] Fourthly, embodiments of the present invention provide a task response device for facial images, the device comprising:

[0063] The task determination module is used to determine the processing tasks for facial images;

[0064] The positioning module is used to locate the facial key points in the facial image according to the above-described facial key point positioning method, so as to obtain the dense key points in the facial image.

[0065] A processing module is used to perform the processing task on the facial image by utilizing the dense key points in the facial image.

[0066] Fifthly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0067] Memory, used to store computer programs;

[0068] When a processor executes a program stored in memory, it implements the steps of the facial key point localization method described above, or the steps of a task response method for a facial image.

[0069] Sixthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for locating facial key points, or the steps of a task response method for a facial image.

[0070] Beneficial effects of the embodiments of the present invention:

[0071] In the facial keypoint localization method provided by this invention, the facial image is segmented into local regions based on sparse keypoints in the facial image to obtain a local image. Then, a multi-level keypoint prediction model is used to predict keypoints in this local image to obtain the keypoint prediction result. Since the input of each level of this multi-level keypoint prediction model is a local image, it ensures that, compared to the entire facial image, it contains more detailed information at the same resolution, and the output of each level model has high accuracy. Simultaneously, the input of each subsequent level model is based on the keypoints output by the previous level model, and the number of keypoints output by the subsequent level model is greater than that of the previous level model. Therefore, the input of the subsequent level model contains more detailed information, thus predicting facial keypoints more accurately than the previous level model. Furthermore, since the local resolution of the image input to each level model increases progressively, mapping the facial keypoints output by the last level model in this multi-level keypoint prediction model back to the original facial image can reduce the problem of error amplification to a certain extent. Therefore, this solution can improve the localization accuracy of dense keypoints.

[0072] Furthermore, the task response method for facial images provided in this embodiment of the invention utilizes dense key points in the facial image to process the facial image processing task, which can meet the needs of tasks with high processing accuracy requirements.

[0073] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0075] Figure 1 A flowchart illustrating a method for locating facial key points provided in an embodiment of the present invention;

[0076] Figure 2 This is another flowchart of the facial key point localization method provided in an embodiment of the present invention;

[0077] Figure 3 This is another flowchart of the facial key point localization method provided in the embodiments of the present invention;

[0078] Figure 4 This is a flowchart of a task response method for facial images provided in an embodiment of the present invention;

[0079] Figure 5 This is a flowchart for locating facial landmarks provided in related technologies;

[0080] Figure 6 A flowchart illustrating a specific example of the facial key point localization method provided in an embodiment of the present invention;

[0081] Figure 7 A flowchart illustrating another specific example of the facial key point localization method provided in this embodiment of the invention;

[0082] Figure 8 This is a schematic diagram of the structure of a facial key point positioning device provided in an embodiment of the present invention;

[0083] Figure 9 This is a schematic diagram of the structure of a task response device for facial images provided in an embodiment of the present invention;

[0084] Figure 10 A schematic diagram of the structure of an electronic device for implementing the method provided in the embodiments of the present invention. Detailed Implementation

[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0086] Below, we will first introduce the technical terms involved in the embodiments of this invention:

[0087] Facial key points: refers to a set of points that are artificially defined to represent specific key locations on the face in order to determine the location of certain special and important areas of the face (such as the eyes, nose, mouth, eyebrows, etc.).

[0088] Facial landmark localization refers to the technical method of determining the coordinates of facial landmarks using computers or sensors in a certain way.

[0089] Facial detection refers to the technical method of determining the approximate location of facial areas using devices such as computers or sensors in a certain way.

[0090] The concepts of sparse and dense keypoints on the face are relative. Sparse keypoints are keypoints with a relatively small number of keypoints, while dense keypoints are keypoints with a relatively large number of keypoints. Generally, 68 keypoints can be used as the threshold. If the number of keypoints on the face is less than 68, it is considered a sparse keypoint, and if the number of keypoints on the face is not less than 68, it is considered a dense keypoint.

[0091] Facial landmark localization technology has always been a fundamental and popular technology in face-related applications. Different face-related applications have different requirements for the accuracy and number of facial landmarks. For example, in the field of face recognition, only a few facial landmarks may be needed to normalize the face to a uniform size, and the requirements for the number and accuracy of facial landmarks are relatively low. However, in the field of beauty and makeup, more precise and dense facial landmarks may be needed to accurately guide the adjustment of facial features.

[0092] In the current field of dense facial landmark prediction, facial 3D reconstruction technology can be used to first reconstruct a 3D facial model, then establish a projection mapping relationship between the 3D model and the 2D facial image, and finally project predefined points on the 3D facial model onto a 2D plane to obtain dense landmarks. However, facial 3D reconstruction technology is generally complex, time-consuming, and consumes a large amount of video memory, so it can only be applied to servers with abundant computing resources and cannot be applied to mobile devices or edge chips with limited computing resources.

[0093] To conserve computational resources, related technologies involve directly scaling facial images to a specified size, then inputting the scaled image into a keypoint prediction model based on convolutional neural networks to predict keypoints in the scaled image. These keypoints are then mapped back to the facial image to obtain dense keypoints of the face. However, considering computational resources and hardware performance, this specified size must be much smaller than the size of the facial image. This results in two problems: firstly, the input image to the model contains less detailed information; secondly, mapping the keypoints predicted by the keypoint prediction model back to the facial image leads to error amplification. Both of these factors undoubtedly result in significant accuracy loss in the obtained dense keypoints. For example, if the size of the facial image is five times this specified size, and the predicted keypoints in the scaled image have a 2-pixel error, mapping this 2-pixel error back to the facial image will amplify the error to 10 pixels.

[0094] Based on the above, in order to improve the positioning accuracy of dense key points, embodiments of the present invention provide a method for locating facial key points, a task response method, and an apparatus.

[0095] Below, we will first introduce a method for locating facial key points provided by an embodiment of the present invention.

[0096] The facial key point localization method provided in this invention can be applied to electronic devices. In specific applications, the electronic device can be a server or a terminal device, both of which are reasonable. In practical applications, the terminal device can be a tablet computer, desktop computer, etc.

[0097] Specifically, the entity executing this facial key point localization method can be a facial key point localization device. For example, when the facial key point localization method is applied to a terminal device, the facial key point localization device can be functional software running on the terminal device, such as software for facial key point detection; of course, the facial key point localization device can also be a plugin in existing functional software, such as a key point localization plugin in facial recognition software. For example, when the facial key point localization method is applied to a server, the facial key point localization device can be a computer program running on the server, which can be used to determine facial key points.

[0098] The method for locating facial key points provided in this embodiment of the invention may include the following steps:

[0099] Identify sparse keypoints in facial images;

[0100] Based on the sparse keypoints, the facial image is segmented into local regions to obtain a local image; wherein, the local region is any facial feature region;

[0101] Using the local image and a multi-level keypoint prediction model, the keypoint prediction result is determined. In the multi-level keypoint prediction model, the input to the first-level model includes a first image obtained by scaling the resolution of the local image; the input to other levels includes a second image obtained by scaling the resolution of a specified image, which is an image obtained by segmenting the facial image using the keypoints output by the previous level model. Furthermore, the local resolution of the input image and the number of output keypoints for each level model increase progressively.

[0102] Based on the key point prediction results, dense key points in the facial image are determined.

[0103] The solution provided in this invention performs local region segmentation on a facial image based on sparse keypoints in the facial image to obtain a local image. Then, a multi-level keypoint prediction model is used to predict keypoints in this local image to obtain the keypoint prediction result. Since the input of each level of this multi-level keypoint prediction model is a local image, it ensures that, compared to the entire facial image, it contains more detailed information at the same resolution, and the output of each level model has high accuracy. Simultaneously, the input of each subsequent level model is based on the keypoints output by the previous level model, and the number of keypoints output by the subsequent level model is greater than that of the previous level model. Therefore, the input of the subsequent level model contains more detailed information, thus predicting facial keypoints more accurately than the previous level model. Furthermore, since the local resolution of the image input to each level model increases progressively, mapping the facial keypoints output by the last level model in this multi-level keypoint prediction model back to the original facial image can reduce the problem of error amplification to a certain extent. Therefore, this solution can improve the localization accuracy of dense keypoints.

[0104] The method for locating facial key points provided by the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0105] like Figure 1 As shown, the facial key point localization method provided in this embodiment of the invention may include steps S101-S104:

[0106] S101, Identify sparse keypoints in the facial image;

[0107] In this embodiment, the facial image can be an image pre-stored locally on the electronic device or an image acquired in real time. For example, the facial image can be an image containing the facial region stored on a mobile phone, or an image containing the facial region captured in real time by the mobile phone's camera. Furthermore, the facial image described in this embodiment can be a human face image; however, the facial image is not limited to human faces. For example, in some scenarios, the facial image can also be an animal face image. It should be noted that the facial image in this embodiment can come from a public dataset, or the acquisition of the facial image has been authorized by the corresponding user; and the collection, storage, use, and processing of the facial image all comply with relevant laws and regulations and do not violate public order and good morals.

[0108] To obtain dense keypoints in a facial image, the image can first be scaled to a specified size. Then, sparse keypoint prediction is performed on the scaled image to obtain sparse keypoints. This specified size can be much smaller than the original size of the facial image. The sparse keypoints can be several keypoints used to locate facial features; for example, the sparse keypoints could include nine points used to locate the left and right ears, left and right eyebrows, left and right eyes, the tip of the nose, and the left and right corners of the mouth. It is understandable that scaling the facial image to a specified size for sparse keypoint prediction can quickly locate facial features while saving computational resources.

[0109] For example, determining sparse keypoints in a facial image can be achieved by inputting a scaled facial image into a sparse keypoint prediction model to obtain the sparse keypoints of the facial image. This sparse keypoint prediction model can be a keypoint prediction model based on a convolutional neural network. The input of this model is a facial image of a specified size, and the output is the positional information of a specified number of facial keypoints. Based on the positional information of each facial keypoint, each facial keypoint can be determined. Of course, the sparse keypoint prediction model in this embodiment can also be a keypoint prediction model based on other neural networks, such as a keypoint prediction model based on a deep neural network. It should be noted that this embodiment of the invention does not limit the specific type and structure of the keypoint prediction model.

[0110] S102, Based on the sparse keypoints, perform local region segmentation on the facial image to obtain a local image; wherein, the local region is any facial feature region;

[0111] After obtaining sparse keypoints in step S101, since these sparse keypoints are facial keypoints used to locate the facial feature regions, the facial image can be segmented based on these sparse keypoints to obtain local images belonging to each facial feature region. It is understandable that since the sparse keypoints are obtained by predicting keypoints on a scaled facial image, and the specified size of the scaled facial image is much smaller than the original size, the scaled facial image contains less detailed information about the facial feature regions, resulting in lower accuracy of the predicted sparse keypoints. Therefore, since the specific locations of the facial features cannot be accurately determined based on these sparse keypoints, the segmentation area can be appropriately enlarged during local region segmentation to ensure that the obtained local image covers the complete facial feature regions.

[0112] Optionally, in one implementation, based on the sparse keypoints, local region segmentation is performed on the facial image to obtain a local image, which may include steps A1-A2:

[0113] A1, map the sparse keypoints onto the facial image to obtain the initial keypoints;

[0114] A2. Based on the facial features represented by the initial key point, perform local region segmentation on the facial image to obtain a local image.

[0115] Understandably, since sparse keypoints are obtained by scaling a facial image to a specified size and then predicting keypoints on that scaled image, in order to perform local region segmentation on the facial image, these sparse keypoints can first be mapped onto the facial image to obtain initial keypoints. For example, based on the corresponding positional relationship between each pixel in the facial image and the scaled facial image, positional information matching the positional information of the sparse keypoints can be determined from the facial image and used as the positional information of the initial keypoints, thus obtaining the initial keypoints. Then, based on the facial feature regions represented by these initial keypoints, local region segmentation is performed on the facial image to obtain local images. For example, contour lines of each facial feature region can be constructed based on the initial keypoints, and then the region containing each facial feature contour line can be segmented from the facial image to obtain various local images. For example, each local image can include local images representing the left and right ears, left and right eyebrows, left and right eyes, nose, mouth, upper half of the face (ear-upward region), and lower half of the face (ear-downward region), respectively.

[0116] Additionally, it should be noted that when determining dense keypoints in a facial image, the facial image can be segmented into facial feature regions according to the user's needs to obtain a local image. For example, if the user only needs dense keypoints in the eye region, the eye region in the facial image can be segmented using sparse keypoints in the eye region to obtain a local image. Subsequently, this local image of the eye region can be processed to obtain dense keypoints for locating the eye region.

[0117] S103, using the local image and the multi-level keypoint prediction model, the keypoint prediction result is determined; wherein, in the multi-level keypoint prediction model, the input of the first-level model includes a first image obtained by scaling the resolution of the local image, and the input of other levels of the model includes a second image obtained by scaling the resolution of a specified image, which is an image obtained by segmenting the facial image using the keypoints output by the previous level of the model; and the local resolution of the input image and the number of output keypoints of each level of the model increase progressively.

[0118] In this embodiment, each level of the multi-level keypoint prediction model can include multiple models, which can be used to perform keypoint recognition on images corresponding to different facial feature regions. For example, each level model can include one or more of an eye keypoint recognition model, a nose keypoint recognition model, and a mouth keypoint recognition model. Of course, each level model can also include models for keypoint recognition on other facial feature regions, such as a pupil keypoint recognition model. This embodiment of the invention does not limit the type or number of models included in each level model. Furthermore, the training process for each level model can involve using sample images containing the corresponding facial feature regions as input. Each sample image has a specified label, which can be the ground truth coordinates of facial keypoints. The model is trained based on the loss value between the coordinates of the facial keypoints output for each sample image and the corresponding label, until the model converges.

[0119] Furthermore, it's understandable that in electronic devices with a specific resolution, a larger image size corresponds to higher resolution, and a smaller image size to lower resolution. Therefore, scaling the resolution of a local image yields a first image smaller than the local image itself. Inputting this first image into the first-level model of a multi-level keypoint prediction model can similarly conserve computational resources while obtaining keypoints corresponding to this first image. Since the local region contained in the local image is any facial feature region in the face image, and the size of this local image is smaller than the size of the face image, even if the local image and the face image are scaled to the same specified size, the local resolution corresponding to the facial feature region in the local image is higher, containing more detailed information about that region. Thus, this first-level model can output more and more accurate facial keypoints than sparse keypoints.

[0120] It should be noted that the number of levels in the multi-level keypoint prediction model is not limited in this embodiment of the invention. Furthermore, after the first-level model outputs keypoints corresponding to the first image, the aforementioned local region segmentation and resolution scaling steps can be repeated using these keypoints to obtain a second image as the input content of the second-level model. Moreover, the input content of subsequent keypoint prediction models can all be obtained by performing local region segmentation and resolution scaling based on the keypoints output by the previous level model. The local resolution of the input image and the number of output keypoints increase progressively with each level of the model.

[0121] For example, if the multi-level keypoint prediction model is a three-level keypoint prediction model, that is, the multi-level keypoint model is composed of three keypoint prediction models connected in series. If 2000 In a 2000-pixel facial image, the size of the nose region is 500. Due to the low number and accuracy of sparse keypoints, a 500-pixel image is used for local segmentation of the nose region in a facial image. Therefore, the segmentation area is appropriately expanded to 1000 pixels. A 1000-pixel local image is scaled down to obtain a 320-pixel local image. A first image of 320 pixels is input into a first-level model to obtain facial key points output by the first-level model. Since the number of facial key points output by the first-level model is greater than the number of sparse key points, using these first-level key points for local region segmentation of the facial image can reduce the scaling required for segmenting local regions, for example, segmenting into 600 pixels. A specified image of the nose region, at 600 pixels. This specified image is then scaled down to obtain a 320 pixel image. A second image of 320 pixels. Because this second image consists of 600... The second image is obtained by scaling up a specified image of 600 pixels. Therefore, the resolution of the nose region in the second image is higher than that in the first image, resulting in richer details. Inputting this second image into the second-level model allows for the prediction of more and more accurate facial key points compared to the first-level model. The third-level model, also based on the second-level key points output from the second-level model, performs local region segmentation on the facial image. Since the number of second-level key points is greater than that of first-level key points, using these key points for local region segmentation allows for more precise localization of the nose region, resulting in a more accurate specified image. This specified image is then scaled up to obtain a second image containing even richer details. Processing this second image using the third-level model yields more and more accurate facial key points than the first-level model.

[0122] It is understandable that, in this multi-level keypoint prediction model, the input content of the later level model contains more detailed information than the previous level model, and the number of keypoints output by the later level model is greater than the number of keypoints output by the previous level model. Therefore, by utilizing this step-by-step amplification idea of ​​the multi-level keypoint prediction model, the last level model of the multi-level keypoint prediction model will output more accurate keypoint prediction results.

[0123] Furthermore, it's worth mentioning that if the determination of sparse keypoints in the facial image in step S101 utilizes a sparse keypoint prediction model, this sparse keypoint prediction model can be combined with the multi-level keypoint prediction model in step S103 to form a new multi-level keypoint prediction model, serving as a model system. In this case, the sparse keypoint prediction model is the first-level model in this system, while the original first-level model in the multi-level keypoint prediction model becomes the second-level model. It can be understood that by using the sparse keypoint prediction model as the first-level model in this system, subsequent levels output more keypoints, resulting in higher accuracy of the input content for each subsequent level compared to the previous level, thus enabling faster and more accurate acquisition of facial keypoints.

[0124] S104. Based on the key point prediction results, determine the dense key points in the facial image.

[0125] In this embodiment, the keypoint prediction result obtained through step S103 contains a large number of highly accurate facial keypoints. By mapping the location information of these multiple facial keypoints onto the original facial image, the dense keypoints in the facial image can be determined. It is understood that as the number and accuracy of keypoints output by each level of the model increase progressively, the later level model uses the keypoints output by the previous level model to perform local region segmentation on the facial image, resulting in a more precise segmentation of the specified local region. Therefore, when mapping the facial keypoints output by the last level model in this multi-level keypoint prediction model back to the original facial image, mapping these keypoints back to a more precise local region in the facial image can reduce the problem of error amplification to a certain extent, thereby obtaining highly accurate dense keypoints.

[0126] Additionally, it should be noted that if mapping the positional information of these multiple facial key points onto the original facial image does not yield a sufficient number of dense key points as required by the user, interpolation can be used to obtain the required number of dense key points. For example, curve fitting can be performed on each facial key point to obtain the facial contour lines, and interpolation can be used to fill the contour lines between pixels. Then, the pixels at the interpolation points can be used as facial key points.

[0127] The solution provided in this embodiment performs local region segmentation on a facial image based on sparse keypoints in the facial image to obtain a local image. Then, a multi-level keypoint prediction model is used to predict keypoints in this local image to obtain the keypoint prediction result. Since the input of each level of this multi-level keypoint prediction model is a local image, it ensures that, compared to the entire facial image, it contains more detailed information at the same resolution, and the output of each level model has high accuracy. Simultaneously, the input of each subsequent level model is based on the keypoints output by the previous level model, and the number of keypoints output by the subsequent level model is greater than that of the previous level model. Therefore, the input of the subsequent level model contains more detailed information, thus predicting facial keypoints more accurately than the previous level model. Furthermore, since the local resolution of the image input to each level model increases progressively, mapping the facial keypoints output by the last level model in this multi-level keypoint prediction model back to the original facial image can reduce the problem of error amplification to some extent. Therefore, this solution can improve the localization accuracy of dense keypoints.

[0128] Optionally, in another embodiment of the invention, in Figure 1 Based on the illustrated embodiments, as Figure 2As shown, in step S102 above, based on the sparse keypoints, local region segmentation is performed on the facial image to obtain a local image, which may include steps S1021-S1022:

[0129] S1021, Determine a facial region for key point localization, and designate it as the designated region;

[0130] In this embodiment, the facial feature area to be located for key points can be a facial feature area determined according to user needs. For example, in fields such as beauty filters and makeup, if fine adjustments to the nose area are required, the nose area can be used as the designated area.

[0131] S1022, Based on the sparse keypoints belonging to the specified region among the sparse keypoints, the facial region is segmented for the specified region to obtain a local image.

[0132] In this embodiment, the location information of sparse keypoints belonging to the specified region can be used to map these sparse keypoints into the facial image, obtaining initial keypoints corresponding to the specified region. Then, based on the facial features represented by the initial keypoints of the specified region, the facial image is segmented into the specified region to obtain a local image. For example, if the specified region is the nose region, the sparse keypoints belonging to the nose region in the facial image can be mapped into the facial image to obtain the initial keypoints of the nose region. Then, the outline of the nose region is constructed based on the initial keypoints of the nose region, thereby segmenting the region containing the nose outline from the facial image, and the image of the segmented region containing the nose outline is used as a local image.

[0133] It is understandable that by segmenting a specified region in a facial image to obtain a local image, the local image of the specified region and the key point prediction model can be used to obtain the key point prediction results of the specified region, thereby determining the dense key points of the specified region.

[0134] As can be seen, this scheme allows for the localization of key points in a specific facial feature region within a facial image. By treating this region as the designated area, the facial image is segmented to obtain a local image. Subsequent processing can then focus only on this local image, thereby reducing computational load.

[0135] Optionally, in another embodiment of the present invention, the input content of each level of the multi-level key point prediction model further includes: auxiliary information; wherein, the auxiliary information is used to characterize the facial contour information determined based on the key points in the image corresponding to the level model; the image corresponding to the first level model is the first image, and the images corresponding to other levels of the model are the second images.

[0136] In this embodiment, when predicting key points, the input of each level of the model may also include auxiliary information corresponding to the facial contours in the input image. This auxiliary information can be obtained based on the key points output by the previous level of the model. For example, when mapping the key points output by the previous level of the model back to the original image, multiple initial key points for locating the facial regions can be obtained. Using these initial key points, the contour lines of each facial feature can be constructed, and then the facial contour lines that match the image corresponding to each level of the model are used as auxiliary information for that image. For example, the training process for each level of the model may use sample images containing the corresponding facial regions and sample contour lines containing sample contour lines as input, wherein the sample contour lines are the facial contour lines corresponding to the corresponding facial regions. Each sample image has a specified label, which may be the ground truth coordinates of facial key points. The model is trained based on the loss value between the coordinates of the facial key points output for each sample image and the label corresponding to the sample image, until the model converges.

[0137] It is understandable that since this auxiliary information is obtained based on the key points output by the previous level model, it can locate the facial features in the facial image. This allows the current key point prediction model to more accurately and quickly determine the facial key points of the facial features to be located based on this auxiliary key point.

[0138] Optionally, in one implementation, the auxiliary information includes a contour image;

[0139] In this image, the background area is a blank image, and the foreground area is the outline determined based on the facial contour information.

[0140] In this implementation, the contour image is used as auxiliary information for each level of the model, and the foreground region of the contour image is the contour line determined based on the facial feature contour information. It can be understood that by inputting the determined facial feature contour information into each level of the model in the form of an image, each level of the model can quickly predict the facial key points that match the facial feature contour information based on the corresponding positional relationship between the image corresponding to that level of the model and the pixels in the contour image.

[0141] As can be seen, by utilizing auxiliary information corresponding to the facial contours in the input image, this solution can accurately and quickly determine the facial key points of the facial region to be localized.

[0142] Optionally, in another embodiment of the invention, such as Figure 3As shown, using this local image and the multi-level keypoint prediction model, the keypoint prediction result is determined, which may include steps S1031-S1036:

[0143] S1031, The resolution of the local image is scaled to obtain the first image;

[0144] For example, the resolution scaling of the local image can be achieved by performing pooling on the local image to obtain the first image. This pooling process can be max pooling, average pooling, etc.

[0145] S1032, Input the first image into the first-level model of the multi-level keypoint prediction model to obtain the keypoints to be utilized;

[0146] It is understandable that since the first image input to the first-level model is obtained by scaling the resolution of that local image, the first-level model can obtain the key points to be used by predicting key points in the first image with only a small amount of computing resources, thus saving computing resources.

[0147] S1033, take the next level model of the first level model as the current model;

[0148] It is understandable that after obtaining the key points to be used in step S1032, in order to obtain dense key points, the next level model can be used to continue processing the facial image. At this time, the next level model of the first level model can be used as the current model.

[0149] S1034, Based on the key points to be utilized, perform local region segmentation on the facial image to obtain a specified image, and perform resolution scaling on the specified image to obtain a second image;

[0150] In this embodiment, the local resolution of the input image increases progressively with each level of the model, and the second image contains more detailed information than the first image. It is understood that because the second image contains more detailed information about the facial features than the first image, when the current model is used to predict key points on the second-level image, it will output more facial key points than the first-level model.

[0151] S1035, Input the second image into the current model to obtain key points, use them as the key points to be used, and use the next level model of the current model as the new current model, return the step of performing local region segmentation on the facial image based on the key points to be used to obtain the specified image;

[0152] In this embodiment, the key points output by the current model are used to perform local region segmentation and resolution scaling on the facial image, resulting in a second image that serves as the input to the next-level model. It is understood that because the local resolution of the second image input to the next-level model is greater than that of the second image input to the current model, it contains more detailed information than the previous model. Therefore, the next-level model can predict more facial key points based on this increased detail. Furthermore, since the number of currently usable key points output by the next-level model is greater than the number of currently usable key points output by the current model, this increased number of currently usable key points allows for the segmentation of a specified image containing more precise facial feature regions from the facial image, leading to more accurate prediction of subsequent facial key points.

[0153] S1036, the key points output by the last-level model are determined as the key point prediction results.

[0154] In this embodiment, when the current model is not the last-level model, steps S1034 and S1035 are repeated until the current model becomes the last-level model. Based on the key points output by the last-level model, the key point prediction result is determined. The method for determining the key point prediction result can be the same as in step S104 above, and will not be repeated here.

[0155] It is understandable that, as the local resolution of the input images of each level of the multi-level keypoint prediction model increases progressively and the number of output keypoints increases progressively, by continuously using the keypoints output by the current model as the keypoints to be used, and using the next level of the current model as the new current model, the final level of the multi-level keypoint prediction model will output relatively accurate keypoint prediction results through this progressive scaling method.

[0156] Accordingly, in this embodiment, after inputting the second image into the current model to obtain key points, and before using the next level model of the current model as the new current model, the method further includes steps B1-B3:

[0157] B1 determines whether the number of keypoints output by the current model is lower than the number of keypoints represented by the specified requirement; wherein, the specified requirement includes the requirement for the number of keypoints;

[0158] If B2 is lower, then the step of using the next level model of the current model as the new current model is executed;

[0159] B3, if not lower than, then based on the current key points to be utilized, determine the dense key points in the facial image.

[0160] It is understandable that, since each level of the multi-level keypoint prediction model outputs a different number of keypoints, and in actual application scenarios, there may be different requirements for the number and accuracy of facial keypoints, in this embodiment, it can be determined whether the number of keypoints output by the current model is lower than the number of keypoints represented by the specified requirements.

[0161] In this embodiment, if the number of keypoints output by the current model is lower than the number of keypoints represented by the specified requirement, then the next level model of the current model is used for keypoint prediction; otherwise, based on the currently available keypoints output by the current model, dense keypoints in the facial image are determined. In this way, dense keypoints in a facial image that meet the specified requirements can be obtained in a timely manner, thereby saving computational resources.

[0162] As can be seen, this scheme continuously uses the keypoints output by the current model as the keypoints to be utilized, and uses the next level of the current model as the new current model. This step-by-step scaling approach ensures that the final level of the multi-level keypoint prediction model outputs relatively accurate keypoint prediction results. Furthermore, after each level outputs the keypoints to be utilized, by determining whether the number of these keypoints is lower than the number of keypoints represented by the specified requirement, dense keypoints in the facial image that meet the requirement can be obtained promptly, thus saving computational resources.

[0163] Based on the above embodiments of the facial key point localization method, this invention also provides a task response method for facial images, such as... Figure 4 As shown, steps S401-S403 are included:

[0164] S401, Define the processing task for the facial image;

[0165] For example, facial image processing tasks can include beautification tasks, identity recognition tasks, facial feature annotation tasks, and so on.

[0166] S402, according to the above method for locating facial key points, the facial key points of the facial image are located to obtain the dense key points in the facial image.

[0167] It is understandable that after determining the processing task for the facial image in step S401, the facial key point localization method provided in the above embodiment can obtain dense key points in the facial image, so that the facial features can be accurately located in the facial image using the dense key points in the facial image.

[0168] S403, Perform the processing task on the facial image using the dense key points in the facial image.

[0169] In this embodiment, after obtaining the dense key points in the facial image through S402, when performing the processing task on the facial image, these dense key points can be used to accurately guide the facial features to be processed, thereby meeting the requirements of tasks with high processing accuracy. It should be noted that the embodiments of the present invention do not limit the specific implementation method of using these dense key points to perform the processing task.

[0170] As can be seen, this solution utilizes dense key points in facial images to process facial images, which can meet the needs of tasks requiring high processing accuracy.

[0171] For example, the above-mentioned facial image processing tasks may include one or more of the following: face recognition, face modeling, face beautification, face attribute recognition, face classification, etc.

[0172] To better understand the embodiments of the present invention, the following is combined with... Figures 5-7 Specific examples of embodiments of the present invention will be described below.

[0173] Figure 5 A flowchart illustrating a facial landmark localization method in related technologies is provided, including the following steps:

[0174] (1) Input image; wherein, the input image is image data containing human faces that has been collected in advance;

[0175] (2) Face detection, that is, using a face detector to detect the region where the face is located in the input image, which can be represented by a rectangular box;

[0176] (3) If face detection fails, prompt to change the image, that is, re-enter an image. If successful, proceed to step (4).

[0177] (4) Extracting the face region: that is, extracting the area where the face is located from the input image to obtain the facial image;

[0178] (5) Scale the facial image obtained in step (4) to a specified size, which is commonly 224. 224 pixels;

[0179] (6) Input the scaled image into the face key point prediction model to predict the coordinates of the face key points (corresponding to the location information in the above text).

[0180] (7) Map the key points predicted by the model back to the original image and output the coordinates of the mapped face key points.

[0181] In the solutions provided by related technologies, due to considerations of computing resources and hardware performance, when scaling facial images to a specified size, this specified size is much smaller than the size of the high-resolution facial image. This approach has two drawbacks. First, it fails to utilize the detailed information from the high-resolution image. For example, when predicting pupil keypoints, in a low-resolution image, the pupil might only occupy a few pixels, making high-precision localization of pupil keypoints difficult. Second, since the final requirement is the coordinates of keypoints in the original high-resolution image, it is often necessary to map the keypoints predicted by the model back to the high-resolution original image. This step can amplify errors. For example, the model itself might have a prediction error of 2 pixels, but when the predicted keypoint coordinates are scaled from 224 pixels... A 224 resolution image is mapped back to 1120. At a high resolution of 1120, this error is magnified to 10 pixels.

[0182] To address the aforementioned problems, this invention decomposes the high-precision dense keypoint localization task step-by-step and gradually increases the local resolution of facial feature regions in the input image, thereby reducing the loss of detail information and minimizing mapping errors. Taking keypoint localization of a face as an example... Figure 6 A flowchart illustrating a specific example of the facial key point localization method provided by an embodiment of the present invention is shown, including the following steps:

[0183] (1) The acquisition device reads facial image data from the storage medium as the input image;

[0184] (2) Perform face detection on the input image. If the face detection fails, prompt to change the image. If successful, the face detection box representing the approximate location of the face is obtained.

[0185] (3) Based on the obtained face detection bounding box, segment the face region from the original high-resolution input image and scale it to a specified size of the first resolution, which is 72. 72 pixels;

[0186] (4) Input the scaled image into the pre-trained sparse keypoint prediction model and output sparse keypoints. The sparse keypoint prediction model can output 9 sparse keypoints, which are keypoints used to represent the left and right ears, left and right eyebrows, left and right eyes, nose tip and left and right corners of the mouth.

[0187] (5) Map the sparse key points back to the original image and perform alignment correction on the original image based on the obtained initial key points;

[0188] (6) Based on the initial key points, segment the local images of the left and right ears, left and right eyebrows, left and right eyes, nose, mouth, upper half of the face (ears up area) and lower half of the face (ears down area) from the original image respectively;

[0189] (7) Scale the local image to a second resolution to obtain the first image; wherein the second resolution can be 5 times the first resolution;

[0190] (8) Input the first image into the first-level model of the multi-level key point prediction model, and use the facial contour line constructed by the initial key points as auxiliary information to input into the first-level model, and output the first-level facial key points. The first-level model can output 68 first-level facial key points.

[0191] (9) Based on the first-level facial key points, the local image can be segmented more accurately to obtain the specified image. At this time, the specified image can be scaled to the third resolution, which can be twice the second resolution to obtain the second image.

[0192] (10) Input the second image into the second-level model, and combine it with the more accurate facial contour lines determined by the first-level facial key points as auxiliary information to predict more dense and accurate facial key points. The second-level model can output 273 second-level facial key points.

[0193] (11) By performing curve fitting on the second-level facial key points and interpolating to fill the contour lines between the points, dense key points that are closely connected are obtained, and the final dense key points are output. Among them, the dense key points contain at least 1000 facial key points.

[0194] It should be noted that since the number of key points output by the second-level model already covers almost all the face regions that may be used, this multi-level key point prediction model only contains two levels in this example. In practical applications, the number of levels of this multi-level key point prediction model can be increased as needed to obtain facial key points that meet the requirements of accuracy and quantity.

[0195] Furthermore, in practical applications, different projects may have varying requirements for the accuracy and number of key points. For example, the number and accuracy of key points required for face alignment can be met by the output of a sparse point prediction model, eliminating the need for subsequent processes. Similarly, in makeup applications, only fine-tuning of the nose shape may be needed. In this case, when determining the input content of the multi-level key point prediction model, a specific region can be selected for prediction instead of outputting the complete facial key point locations, thus reducing computational load. Furthermore, when performing face recognition on people wearing masks, since the mask covers the nose and mouth areas, key points in other facial features not covered by the mask can be predicted to obtain accurate face recognition results. Based on these considerations, this invention can also flexibly configure parameters such as the input content and model level of the multi-level key point prediction model according to the user's required number of key points and positioning area to meet the needs of face recognition projects on different platforms and for different tasks.

[0196] Taking a model system composed of a sparse keypoint prediction model and a multi-level keypoint prediction model as an example, Figure 7 The flowchart illustrates another specific example of the facial key point localization method provided in an embodiment of the present invention, including the following steps:

[0197] (1) Determine the designated area for key point localization and the specified requirements for the number of key points;

[0198] (2) The acquisition device reads facial image data from the storage medium as the input image;

[0199] (3) Perform face detection on the input image. If the face detection is successful, the image of the specified region is segmented from the original high-resolution input image based on the obtained face detection box and scaled to the specified size of the first resolution.

[0200] (4) Input the scaled image into the first-level model in the model system and output the first-level key points;

[0201] (5) Determine whether the number of first-level key points meets the requirements. If yes, output the first-level key point; otherwise, proceed to step (6).

[0202] (6) Based on the first-level key points, determine the input content of the second-level model in the model system. The input content is the input image representing the specified region and the corresponding auxiliary information. Input the input content of the second-level model into the second-level model to predict key points and output the second-level key points.

[0203] (7) Determine whether the number of second-level key points meets the requirements. If yes, output the second-level key point; otherwise, proceed to step (8).

[0204] (8) Based on the second-level key points, determine the input content of the third-level model. The input content is the input image representing the specified region and the corresponding auxiliary information. Input the input content of the third-level model into the third-level model to predict key points and output the third-level key points.

[0205] (9) Determine whether the number of third-level key points meets the requirements. If yes, output the third-level key point; otherwise, perform curve fitting interpolation on the third-level key point and output it.

[0206] As can be seen, this solution enables high-precision dense keypoint localization of high-resolution face images even when hardware or computing resources are insufficient. Furthermore, it supports flexible configuration of parameters for multi-level keypoint prediction models to meet the needs of face recognition projects on different platforms and for different tasks.

[0207] Corresponding to the embodiments of the above-described facial key point localization method, this invention also provides a facial key point localization device, such as... Figure 8 As shown, the device includes:

[0208] The determination module 810 is used to determine sparse key points in a facial image;

[0209] The segmentation module 820 is used to perform local region segmentation on the facial image based on the sparse key points to obtain a local image; wherein, the local region is any facial feature region;

[0210] The prediction module 830 is used to determine the keypoint prediction result using the local image and the multi-level keypoint prediction model. The input to the first-level model of the multi-level keypoint prediction model includes a first image obtained by scaling the resolution of the local image. The input to other levels of the model includes a second image obtained by scaling the resolution of a specified image, where the specified image is an image obtained by segmenting the facial image using the keypoints output by the previous level model. Furthermore, the local resolution of the input image and the number of output keypoints for each level of the model increase progressively.

[0211] The output module 840 is used to determine the dense key points in the facial image based on the key point prediction results.

[0212] Optionally, the segmentation module includes:

[0213] The mapping submodule is used to map the sparse key points onto the facial image to obtain initial key points;

[0214] The first segmentation submodule is used to perform local region segmentation on the facial image based on the facial feature regions represented by the initial key points to obtain a local image.

[0215] Optionally, the segmentation module includes:

[0216] The determination submodule is used to determine a facial region for key point localization, which is then designated as the specified region.

[0217] The second segmentation submodule is used to segment the facial image for the specified region based on the sparse keypoints belonging to the specified region, and obtain a local image.

[0218] Optionally, the input content of each level of the multi-level key point prediction model may further include: auxiliary information;

[0219] The auxiliary information is used to characterize the facial contour information determined based on the key points in the image corresponding to the model at that level; the image corresponding to the first-level model is the first image, and the images corresponding to other-level models are the second images.

[0220] Optionally, the auxiliary information includes a contour image;

[0221] The background area of ​​the contour image is a blank image, and the foreground area of ​​the contour image is a contour line determined based on the facial contour information.

[0222] Optionally, the prediction module is specifically used for:

[0223] The local image is scaled up to obtain the first image;

[0224] The first image is input into the first-level model of the multi-level key point prediction model to obtain the key points to be utilized.

[0225] Use the next level model of the first level model as the current model;

[0226] Based on the key points to be utilized, the facial image is segmented into local regions to obtain a specified image, and the specified image is then scaled to obtain a second image.

[0227] The second image is input into the current model to obtain key points, which are used as the key points to be used. The next level model of the current model is used as the new current model. The process of performing local region segmentation on the facial image based on the key points to be used is then returned to obtain the specified image.

[0228] The key points output by the last-level model are determined as the key point prediction results.

[0229] Optionally, after inputting the second image into the current model to obtain key points as the currently utilized key points, and before using the next-level model of the current model as the new current model, the device is further configured to:

[0230] Determine whether the number of keypoints output by the current model is lower than the number of keypoints represented by a specified requirement; wherein, the specified requirement includes a requirement for the number of keypoints;

[0231] If it is lower, then the step of taking the next level model of the current model as the new current model is executed;

[0232] If it is not lower than the threshold, then based on the currently available key points, the dense key points in the facial image are determined.

[0233] Corresponding to the above embodiments of the task response method for facial images, this embodiment of the invention also provides a task response device for facial images, such as... Figure 9 As shown, the device includes:

[0234] Task determination module 910 is used to determine the processing task for the facial image;

[0235] The positioning module 920 is used to locate the facial key points in the facial image according to the above-described facial key point positioning method, so as to obtain the dense key points in the facial image.

[0236] The processing module 930 is used to perform the processing task on the facial image by utilizing the dense key points in the facial image.

[0237] This invention also provides an electronic device, such as... Figure 10 As shown, it includes a processor 1001, a communication interface 1002, a memory 1003, and a communication bus 1004, wherein the processor 1001, the communication interface 1002, and the memory 1003 communicate with each other through the communication bus 1004.

[0238] Memory 1003 is used to store computer programs;

[0239] When the processor 1001 executes the program stored in the memory 1003, it implements the steps of the facial key point localization method described in any of the above embodiments, or the steps of the task response method for facial images.

[0240] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0241] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0242] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0243] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0244] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of the facial key point localization method described in any of the above embodiments, or the steps of a task response method for a facial image.

[0245] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of the facial key point localization method described in any of the above embodiments, or the steps of the task response method for a facial image.

[0246] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0247] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0248] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0249] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for locating facial landmarks, the method comprising: The method includes: Identify sparse keypoints in facial images; Based on the sparse keypoints, the facial image is segmented into local regions to obtain a local image; wherein, the local region is any facial feature region; Using the local image and a multi-level keypoint prediction model, keypoint prediction results are determined. In the multi-level keypoint prediction model, the input to the first-level model includes a first image obtained by scaling the resolution of the local image; the input to other levels includes a second image obtained by scaling the resolution of a specified image, which is an image obtained by segmenting the facial image using keypoints output from the previous level model. Furthermore, the local resolution of the input image and the number of output keypoints for each level model increase progressively. The input to each level of the multi-level keypoint prediction model also includes auxiliary information, which characterizes the facial contour information determined based on the keypoints in the image corresponding to that level model. The image corresponding to the first-level model is the first image, and the images corresponding to other levels are the second images. Based on the key point prediction results, dense key points in the facial image are determined.

2. The method of claim 1, wherein, The step of performing local region segmentation on the facial image based on the sparse keypoints to obtain a local image includes: The sparse keypoints are mapped onto the facial image to obtain initial keypoints; Based on the facial features represented by the initial key points, the facial image is segmented locally to obtain a local image.

3. The method of claim 1, wherein, The step of performing local region segmentation on the facial image based on the sparse keypoints to obtain a local image includes: Identify the facial features area where key point localization is to be performed, and designate it as the designated area; Based on the sparse keypoints belonging to the specified region, the facial image is segmented for the specified region to obtain a local image.

4. The method of claim 1, wherein, The auxiliary information includes a contour image; The background area of ​​the contour image is a blank image, and the foreground area of ​​the contour image is a contour line determined based on the facial contour information.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the keypoint prediction result using the local image and the multi-level keypoint prediction model includes: The local image is scaled up to obtain the first image; The first image and corresponding auxiliary information are input into the first-level model of the multi-level key point prediction model to obtain the key points to be utilized. Use the next level model of the first level model as the current model; Based on the key points to be utilized, the facial image is segmented into local regions to obtain a specified image, and the specified image is then scaled to obtain a second image. The second image and corresponding auxiliary information are input into the current model to obtain key points, which are used as the key points to be utilized. The next level model of the current model is used as the new current model. The process of performing local region segmentation on the facial image based on the key points to be utilized to obtain the specified image is then returned. The key points output by the last-level model are determined as the key point prediction results.

6. The method of claim 5, wherein, After inputting the second image and corresponding auxiliary information into the current model to obtain key points, and before using the next level model of the current model as the new current model, the method further includes: Determine whether the number of keypoints output by the current model is lower than the number of keypoints represented by a specified requirement; wherein, the specified requirement includes a requirement for the number of keypoints; If it is lower, then the step of taking the next level model of the current model as the new current model is executed; If it is not lower than the threshold, then based on the currently available key points, the dense key points in the facial image are determined.

7. A task response method for facial images, characterized in that, The method includes: Define the processing task for facial images; According to the facial key point localization method of any one of claims 1-6, the facial key points are located in the facial image to obtain dense key points in the facial image; The processing task is performed on the facial image using dense key points in the facial image.

8. A device for locating facial key points, characterized in that, The device includes: The determination module is used to determine sparse keypoints in facial images; The segmentation module is used to perform local region segmentation on the facial image based on the sparse keypoints to obtain a local image; wherein, the local region is any facial feature region; The prediction module is used to determine the keypoint prediction result using the local image and a multi-level keypoint prediction model. The multi-level keypoint prediction model's input includes a first image obtained by scaling the resolution of the local image, and the input of other levels includes a second image obtained by scaling the resolution of a specified image. The specified image is an image obtained by segmenting the facial image using the keypoints output by the previous level model. Furthermore, the local resolution of the input image and the number of output keypoints increase progressively with each level. The input of each level of the multi-level keypoint prediction model also includes auxiliary information, which characterizes the facial contour information determined based on the keypoints in the image corresponding to that level model. The image corresponding to the first level model is the first image, and the images corresponding to other levels are the second images. The output module is used to determine the dense key points in the facial image based on the key point prediction results.

9. A task response device for facial images, characterized in that, The device includes: The task determination module is used to determine the processing tasks for facial images; A positioning module is used to locate facial key points in the facial image according to the facial key point positioning method according to any one of claims 1-6, and obtain dense key points in the facial image. A processing module is used to perform the processing task on the facial image by utilizing the dense key points in the facial image.

Images