Facial landmark detection method and device

Abstract

This invention provides a method and apparatus for facial landmark detection. The method includes: acquiring a face image; inputting the face image into a facial landmark detection model to obtain the facial landmark detection result output by the model; wherein the facial landmark detection model is trained based on sample images, the ground truth values ​​of facial landmarks corresponding to the sample images, and the ground truth values ​​of points within a preset range of facial landmarks; the facial landmark model is used to weight and output the facial landmark detection result based on the predicted facial landmarks from the face image and the points within the preset range. This invention guides the facial landmark detection model to learn facial landmarks by using points within the preset range of landmarks, thereby changing the model from predicting facial landmarks to predicting facial landmarks and points within the preset range, thus increasing the information learned by the model, improving the robustness of the model, further improving the stability of facial landmark detection, and reducing the false detection rate.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for detecting facial key points. Background Technology

[0002] Facial landmark detection is a crucial task in computer vision, aiming to locate facial landmarks such as the corners of the lips, eyes, and nose in images or videos. The facial landmark detection process is a key step in many downstream tasks, playing a vital role in face recognition and facial age estimation. Due to its wide range of applications, facial landmark detection involves challenging scenarios such as facial occlusion and lighting variations, thus placing higher demands on the accuracy and robustness of recognition methods.

[0003] In driver detection systems, facial landmark detection is essential, and the accuracy of landmark detection plays a decisive role in judging whether a driver is distracted or fatigued.

[0004] Currently, due to considerations such as cost and power consumption, facial landmark models are often required to operate with extremely low computing power. Therefore, regression methods are mostly used to directly predict facial landmarks. However, this method often introduces large errors in the landmarks due to small shifts, resulting in poor stability of facial landmarks and a tendency to miss or falsely detect them in complex environments. Summary of the Invention

[0005] This invention provides a method and apparatus for detecting facial landmarks, which addresses the shortcomings of existing technologies where the stability of facial landmark detection is poor due to low model computing power. This invention improves the stability of the model in detecting facial landmarks, reduces the false detection rate of the model, and enhances the robustness of the model.

[0006] This invention provides a method for detecting facial landmarks, comprising: acquiring a facial image; inputting the facial image into a facial landmark detection model to obtain a facial landmark detection result output by the facial landmark detection model; wherein, the facial landmark detection model is trained based on a sample image and the ground truth values ​​of facial landmarks corresponding to the sample image and the ground truth values ​​of points within a preset range of the facial landmarks; the facial landmark model is used to perform weighted output based on the facial landmarks predicted from the facial image and the points within the preset range to obtain the facial landmark detection result.

[0007] According to the present invention, a facial key point detection method is provided, wherein the facial key point detection model includes: a feature point extraction layer, which performs key point detection on an input facial image to obtain facial key points and points within a preset range, and obtains the confidence scores corresponding to the facial key points and points within the preset range respectively; and a facial key point detection layer, which obtains the facial key point detection result based on the facial key points and points within the preset range and the confidence scores corresponding to the facial key points and points within the preset range respectively.

[0008] According to a facial landmark detection method provided by the present invention, the step of detecting landmarks in an input facial image to obtain facial landmarks and points within a preset range, and obtaining the confidence scores corresponding to the facial landmarks and points within the preset range respectively, includes: extracting features from the input facial image to obtain facial landmarks and points within a preset range; and, based on the facial landmarks and points within the preset range, using a two-dimensional Gaussian distribution to obtain the confidence scores corresponding to the facial landmarks and points within the preset range respectively.

[0009] According to a method for detecting facial key points provided by the present invention, obtaining the detection result of the facial key points includes: weighting the output based on the facial key points and points within a preset range, combined with the confidence levels corresponding to the facial key points and points within the preset range, to obtain the detection result of the facial key points.

[0010] According to a facial landmark detection method provided by the present invention, training the facial landmark detection model includes: acquiring a sample image and ground truth values ​​of facial landmarks corresponding to the sample image and ground truth values ​​of points within a preset range of the facial landmarks; using the sample image as input data for the model to be trained, using the ground truth values ​​of facial landmarks and ground truth values ​​of points within a preset range of the facial landmarks as labels, and training the model to be trained to obtain a facial landmark detection model used to generate facial landmark detection results for facial images.

[0011] According to a facial landmark detection method provided by the present invention, training the model to be trained includes: inputting the sample image into a feature point extraction layer to obtain the predicted facial landmark points output by the feature point extraction layer and the predicted points within a preset range, and obtaining the confidence scores corresponding to the predicted facial landmark points output by the feature point extraction layer and the predicted points within the preset range; obtaining the facial landmark detection result by combining the confidence scores corresponding to the predicted facial landmark points and the predicted points within the preset range; constructing a keypoint loss function based on the predicted facial landmark points and the predicted points within the preset range, the ground truth values ​​of the facial landmark points, and the ground truth values ​​of the points within the preset range of the facial landmark points, and converging based on the keypoint loss function to end the training.

[0012] The present invention also provides a facial landmark detection device, comprising: an image acquisition module for acquiring a facial image; and a landmark detection module for inputting the facial image into a facial landmark detection model to obtain a facial landmark detection result output by the facial landmark detection model; wherein, the facial landmark detection model is trained based on a sample image and the ground truth values ​​of facial landmarks corresponding to the sample image and the ground truth values ​​of points within a preset range of the facial landmarks; the facial landmark model is used to perform weighted output based on the facial landmarks predicted from the facial image and the points within the preset range to obtain the facial landmark detection result.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the above-described facial landmark detection methods.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the facial landmark detection method as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described facial landmark detection methods.

[0016] The facial landmark detection method and apparatus provided by this invention guides the facial landmark detection model to learn facial landmarks by using points within a preset range of landmarks. This avoids interference from points within the preset range of landmarks on the model's learning process. Furthermore, the facial landmark detection model is transformed from predicting a single facial landmark to predicting both a single facial landmark and points within a preset range, thereby increasing the information learned by the model and improving its robustness. In addition, by weighting the confidence scores of the predicted facial landmark and several surrounding points to obtain the facial landmark detection result, the stability of facial landmark detection is improved, avoiding large errors caused by small shifts and reducing the false detection rate. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the facial landmark detection method provided by the present invention;

[0019] Figure 2 This is the second flowchart of the facial key point detection method provided by the present invention;

[0020] Figure 3 This is a flowchart illustrating the training process of the facial landmark detection model provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the facial key point detection device provided by the present invention;

[0022] Figure 5 This is a schematic diagram of the training module provided by the present invention;

[0023] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Figure 1The diagram illustrates a flowchart of a facial landmark detection method according to the present invention, the method comprising:

[0026] S11, acquire face image;

[0027] S12, input the face image into the face landmark detection model to obtain the face landmark detection result output by the face landmark detection model; wherein, the face landmark detection model is trained based on the sample image and the ground truth value of the face landmark corresponding to the sample image and the ground truth value of the points within the preset range of the face landmark; the face landmark model is used to perform weighted output based on the face landmark predicted from the face image and the points within the preset range to obtain the face landmark detection result.

[0028] It should be noted that S1N in this manual does not represent the order of the facial landmark detection methods. The following details will explain this in conjunction with... Figure 2 The present invention describes a facial landmark detection method.

[0029] Step S11: Obtain a face image.

[0030] In this embodiment, acquiring a face image includes: obtaining a face image to be detected based on a facial landmark detection method using an electronic device or application platform; or obtaining a face image based on a terminal device connected to the electronic device or application platform using the facial landmark detection method. It should be noted that the aforementioned terminal device can acquire a face image of a person within the recognition area through a connected visual sensor. It should also be noted that the aforementioned face image can be a single-frame image or a sequence of image frames captured by photography, or an image frame or sequence of image frames related to the face to be detected obtained by segmenting a video.

[0031] In one alternative embodiment, the acquired face image may be derived from images captured by millimeter-wave radar, lidar, detectors, cameras, and other imaging devices based on a specific face; the source of the face image is not further limited here.

[0032] Step S12: Input the face image into the face key point detection model to obtain the face key point detection result output by the face key point detection model; wherein, the face key point detection model is trained based on the sample image and the ground truth value of the face key point corresponding to the sample image and the ground truth value of the points within the preset range of the face key point; the face key point model is used to perform weighted output based on the face key points predicted from the face image and the points within the preset range to obtain the face key point detection result.

[0033] In this embodiment, the facial landmark detection model includes: a feature point extraction layer, which performs landmark detection on the input facial image to obtain facial landmarks and points within a preset range, and obtains the confidence scores corresponding to the facial landmarks and points within the preset range; and a facial landmark detection layer, which obtains the facial landmark detection result based on the facial landmarks and points within the preset range and the confidence scores corresponding to the facial landmarks and points within the preset range. Correspondingly, inputting a facial image into the facial landmark detection model to obtain the facial landmark detection result output by the model includes: inputting the facial image into the feature point extraction layer to obtain the facial landmarks and points within the preset range output by the feature point extraction layer, and obtaining the confidence scores corresponding to the facial landmarks and points within the preset range output by the feature point extraction layer; inputting the facial landmarks and points within the preset range and the confidence scores corresponding to the facial landmarks and points within the preset range into the facial landmark detection layer to obtain the facial landmark detection result output by the facial landmark detection layer. Specifically,

[0034] First, the face image is input into the feature point extraction layer to obtain facial key points and points within a preset range, as well as the confidence scores corresponding to each of the facial key points and points within the preset range. Figure 2 .

[0035] Furthermore, the feature point extraction layer performs key point detection on the input face image to obtain the face key points and points within a preset range, as well as the confidence scores corresponding to the face key points and points within the preset range. This includes: performing feature extraction on the input face image to obtain the face key points and points within a preset range; and, based on the face key points and points within the preset range, using a two-dimensional Gaussian distribution to obtain the confidence scores corresponding to the face key points and points within the preset range.

[0036] It should be noted that the feature point extraction layer can extract facial key points and points in the surrounding area based on the input face image, obtaining facial key points and points within a preset range. Here, facial key points can be represented as (x,y), and points within the preset range of facial key points can be represented as (xt,y), (x+t,y), (x,yt), and (x,y+t), where t represents the preset range and t is an adjustable learning parameter.

[0037] In addition, the feature point extraction layer can also obtain the confidence score of each point using a two-dimensional Gaussian distribution while extracting facial key points and their surrounding areas. It should be noted that in other embodiments, the confidence score of each point can also be obtained using a two-dimensional Gaussian distribution after extracting facial key points and their surrounding areas; this is not further limited here.

[0038] In this embodiment, the confidence level is represented as:

[0039]

[0040] Where, f(x) i ,y j ) represents the nearest points (x, y) of the facial landmark (x, y). i ,y j The corresponding confidence level, (x) i ,y j The expression can also be represented as (xt,y), (x+t,y), (x,yt), and (x,y+t) in this embodiment. These are then substituted into the aforementioned two-dimensional Gaussian heatmap to obtain the confidence level for each point. It should be noted that (xt,y), (x+t,y), (x,yt), and (x,y+t) are input into the aforementioned two-dimensional Gaussian heatmap to obtain the corresponding confidence levels. It should be understood that although... Figure 2 The diagram shows that points within a preset range of facial key points can be represented as (xt,y), (x+t,y), (x,yt), and (x,y+t). However, this is just an example. The number of points within the preset range of facial key points can be less than or more than four, such as two, six, eight, etc. Furthermore, the positions of points within the preset range of facial key points can also be other positions, such as (xt,yt), (x+t,y+t), (x+t,yt), and (xt,y+t), etc. The scope of this disclosure is not limited here.

[0041] Secondly, the confidence scores corresponding to the facial key points and their preset range points, as well as the confidence scores of the facial key points and their preset range points, are input into the facial key point detection layer to obtain the facial key point detection results output by the facial key point detection layer.

[0042] To elaborate further, the facial landmark detection layer obtains facial landmark detection results based on the facial landmarks and points within their preset range, as well as the confidence scores corresponding to the facial landmarks and points within their preset range. This includes: weighting the output based on the facial landmarks and points within their preset range, combined with the confidence scores corresponding to the facial landmarks and points within their preset range, to obtain the facial landmark detection results.

[0043] In this embodiment, the facial landmark detection results are represented as follows:

[0044] output_x=(xt)*f(xt,y)+x*f(x,y)+(x+t)*f(x+t,y)

[0045] output_y=(yt)*f(x,yt)+y*f(x,y)+(y+t)*f(x,y+t)

[0046] Wherein, (output_x, output_y) represents the face landmark detection result output by the face landmark detection model, output_x represents the coordinates output by the face landmark detection model along a preset first direction, output_y represents the coordinates output by the face landmark detection model along a preset second direction perpendicular to the first direction, (xt) and (x+t) represent the coordinates of a point within a preset range along the first direction, (yt) and (y+t) represent the coordinates of a point within a preset range along the second direction, f(xt,y) represents the confidence score corresponding to point (xt,y), f(x,y) represents the confidence score corresponding to point (x,y), f(x+t,y) represents the confidence score corresponding to point (x+t,y), f(x,yt) represents the confidence score corresponding to point (x,yt), and f(x,y+t) represents the confidence score corresponding to point (x,y+t).

[0047] In one alternative embodiment, reference Figure 3 The method further includes: training the facial landmark detection model before inputting the face image into the facial landmark detection model, specifically including:

[0048] S31, obtain the sample image and the ground truth value of the facial key points corresponding to the sample image and the ground truth value of the points within the preset range of the facial key points;

[0049] S32, the sample image is used as the input data of the model to be trained, and the ground truth values ​​of facial key points and the ground truth values ​​of points within the preset range of facial key points are used as labels to train the model to be trained, so as to obtain a facial key point detection model used to generate facial key point detection results for facial images.

[0050] It should be noted that S3N in this specification does not represent the order in which the face landmark detection model is trained. The training model for face landmark detection of this invention is described in detail below.

[0051] Step S31: Obtain the sample image and the ground truth values ​​of the facial key points corresponding to the sample image, as well as the ground truth values ​​of points within the preset range of the facial key points.

[0052] In this embodiment, obtaining a sample image and the ground truth values ​​of facial key points corresponding to the sample image and the ground truth values ​​of points within a preset range of facial key points includes: obtaining a sample image; annotating the sample image to obtain the ground truth values ​​of facial key points and the ground truth values ​​of points within a preset range of facial key points.

[0053] It should be noted that acquiring sample images includes: acquiring a video stream; capturing a certain number of video frame images at preset intervals as sample images; or, continuously capturing at least one frame image based on at least one face as a sample image. It should be noted that when acquiring video streams or capturing face images, different external factors such as pose angles, occlusions, and lighting conditions can be considered.

[0054] In addition to acquiring sample images, the process also includes performing face recognition on the acquired sample images to remove images that do not contain faces. It should be noted that sample images can be understood as a collection of image data. This image data can be images targeting at least one face, and each face can be located at different angles, with different pixel colors, etc. The amount of image data is usually quite large, reaching millions. These sample images are used to train the network to complete the model construction.

[0055] In an optional embodiment, after obtaining the sample image and the ground truth values ​​of the corresponding facial key points and the ground truth values ​​of points within a preset range of the facial key points, the method further includes: performing data augmentation on the sample image using a data augmentation strategy. Specifically, the data augmentation strategy includes at least one of flipping, rotating, cropping, deforming, and scaling; and / or, the data augmentation strategy includes at least one of noise reduction, blurring, color transformation, erasing, and filling. It should be noted that the data augmentation strategy selected based on this embodiment is suitable for performing data augmentation on the above-mentioned sample image, so as to increase the amount of training image data and significantly improve the model's key point detection accuracy in scenarios such as lighting, occlusion, incompleteness, large deflection angle, and facial expressions during subsequent model training.

[0056] Step S32: Use the sample image as input data for the model to be trained, and use the ground truth values ​​of facial key points and the ground truth values ​​of points within the preset range of facial key points as labels to train the model to be trained, so as to obtain a facial key point detection model used to generate facial key point detection results for facial images.

[0057] In this embodiment, the network to be trained typically includes a feature point extraction layer for detecting sample images, a facial key point prediction layer based on the feature point extraction layer and its predicted points within a preset range, a confidence level corresponding to the facial key point prediction and its predicted points within a preset range, a facial key point detection layer for detecting key points in facial images, and a loss function. According to a preset iteration rule, the above sample images or data-enhanced sample images are input into the model to be trained for training to obtain the trained facial key point detection model.

[0058] Specifically, training the model to be trained includes: inputting sample images into a feature point extraction layer to obtain the predicted facial key points output by the feature point extraction layer and the predicted points within a preset range, as well as the confidence scores corresponding to the predicted facial key points output by the feature point extraction layer and the predicted points within a preset range; obtaining the facial key point detection results by combining the predicted facial key points and the predicted points within a preset range with the confidence scores corresponding to the predicted facial key points and the predicted points within a preset range; constructing a key point loss function based on the predicted facial key points and the predicted points within a preset range, the ground truth values ​​of the facial key points, and the ground truth values ​​of the points within a preset range of the facial key points, and ending the training based on the convergence of the key point loss function. It should be noted that since the points surrounding facial landmarks greatly interfere with the learning of the facial landmark detection model, information about the vicinity of the landmarks is added to guide the model's learning without increasing the model's computational load. This transforms the model from predicting a single facial landmark to predicting a single facial landmark and several points around it, thereby increasing the information learned by the model and further improving its robustness. Based on the weighted averaging of the confidence scores of the predicted facial landmarks and several points around them, the facial landmark prediction results are output, thereby improving the stability of facial landmark detection and avoiding large errors caused by small offsets.

[0059] In summary, the embodiments of the present invention guide the facial landmark detection model to learn facial landmarks by using points within a preset range of landmarks, thereby avoiding interference from points within the preset range on the model's learning. Furthermore, the facial landmark detection model is transformed from predicting a single facial landmark to predicting a single facial landmark and points within a preset range, thus increasing the information learned by the model and improving its robustness. Additionally, by weighting the confidence scores of the predicted facial landmark and several surrounding points to obtain the facial landmark detection result, the stability of facial landmark detection is improved, avoiding large errors caused by small shifts and reducing the false detection rate.

[0060] The facial landmark detection device provided by the present invention will be described below. The facial landmark detection device described below can be referred to in correspondence with the facial landmark detection method described above.

[0061] Figure 4 A schematic diagram of a facial landmark detection device according to the present invention is shown. The device includes:

[0062] Image acquisition module 41 acquires face images;

[0063] The key point detection module 42 inputs the face image into the face key point detection model to obtain the face key point detection result output by the face key point detection model; wherein, the face key point detection model is trained based on the sample image and the ground truth value of the face key point corresponding to the sample image and the ground truth value of the points within the preset range of the face key point; the face key point model is used to perform weighted output based on the face key points predicted from the face image and the points within the preset range to obtain the face key point detection result.

[0064] Specifically, the image acquisition module 41 includes: a data acquisition unit, which acquires a face image to be detected based on an electronic device or application platform using a facial landmark detection method; or, a data acquisition unit, which acquires a face image based on a terminal device connected to the electronic device or application platform using the facial landmark detection method. It should be noted that the aforementioned terminal device can acquire a face image of a person within the recognition area through a connected visual sensor. It should also be noted that the aforementioned face image can be a single-frame image or a sequence of image frames captured by photography, or an image frame or sequence of image frames related to the face to be detected obtained by segmenting a video.

[0065] In an optional embodiment, the face image acquired by the data acquisition module 41 may be from images taken by millimeter-wave radar, lidar, detector, camera, and other imaging devices based on a specific face. The source of the face image is not further limited here.

[0066] In addition, the key point detection module 42 includes: a data input unit for inputting the face image into the face key point detection model; a face key point detection unit for detecting the face image based on the face key point detection model to obtain the face key point detection result; and a data output unit for outputting the face key point detection result.

[0067] Specifically, the facial landmark detection unit includes: a feature point extraction subunit, which performs landmark detection on the input facial image to obtain facial landmarks and points within a preset range, as well as the confidence scores corresponding to the facial landmarks and points within the preset range; and a facial landmark detection subunit, which obtains the facial landmark detection result based on the facial landmarks and points within the preset range and the confidence scores corresponding to the facial landmarks and points within the preset range. Correspondingly, inputting the facial image into the facial landmark detection unit to obtain the facial landmark detection result output by the facial landmark detection unit includes: inputting the facial image into the feature point extraction subunit to obtain the facial landmarks and points within the preset range output by the feature point extraction subunit, as well as the confidence scores corresponding to the facial landmarks and points within the preset range output by the feature point extraction subunit; and inputting the facial landmarks and points within the preset range and the confidence scores corresponding to the facial landmarks and points within the preset range into the facial landmark detection subunit to obtain the facial landmark detection result output by the facial landmark detection subunit.

[0068] In an optional embodiment, the feature point extraction subunit includes: a feature extraction subunit, which extracts features from the input face image to obtain facial key points and points within a preset range; and a confidence score acquisition subunit, which, based on the facial key points and points within a preset range, uses a two-dimensional Gaussian distribution to obtain the confidence scores corresponding to the facial key points and points within a preset range, respectively.

[0069] The facial landmark detection subunit includes: a detection subunit, which, based on the facial landmarks and points within a preset range, performs a weighted output by combining the confidence scores corresponding to the facial landmarks and points within the preset range, to obtain the facial landmark detection result.

[0070] In one alternative embodiment, reference Figure 5 The device also includes a training module for training the facial landmark detection model before inputting the face image into the model. Specifically, the training module includes:

[0071] The sample acquisition unit 51 acquires the sample image and the ground truth values ​​of the facial key points corresponding to the sample image, as well as the ground truth values ​​of points within a preset range of the facial key points.

[0072] Training unit 52 uses sample images as input data for the model to be trained, and uses ground truth values ​​of facial landmarks and ground truth values ​​of points within a preset range of facial landmarks as labels to train the model to be trained, thereby obtaining a facial landmark detection model used to generate facial landmark detection results for facial images.

[0073] In this embodiment, the sample acquisition unit 51 includes: a sample acquisition subunit for acquiring sample images; and an annotation subunit for annotating the sample images to obtain ground truth values ​​of facial key points and ground truth values ​​of points within a preset range of facial key points.

[0074] It should be noted that the sample acquisition subunit includes: a video acquisition subunit, which acquires the video stream; and an image acquisition subunit, which acquires a certain number of video frame images at preset intervals as sample images; or, an image acquisition subunit, which continuously captures at least one frame image of at least one face as a sample image. It should be noted that when acquiring the video stream or capturing face images, acquisition can be performed under different external factors such as pose angles, occlusions, and lighting conditions.

[0075] In addition, the sample acquisition subunit also includes a filtering subunit, which performs face recognition on the acquired sample images to remove face images that do not contain faces. It should be noted that sample images can be understood as a collection of image data. This image data can be images targeting at least one face, and each face can be located at different angles, with different pixel colors, etc. The amount of image data is usually quite large, reaching millions. These sample images are used to train the network to complete the model construction.

[0076] In an optional embodiment, the device further includes a data augmentation module, which performs data augmentation on the sample images using a data augmentation strategy. Specifically, the data augmentation strategy includes at least one of flipping, rotating, cropping, deforming, and scaling; and / or, the data augmentation strategy includes at least one of noise reduction, blurring, color transformation, erasing, and filling. It should be noted that the data augmentation strategy selected in this embodiment is suitable for performing data augmentation on the aforementioned sample images, thereby increasing the amount of training image data and significantly improving the model's keypoint detection accuracy in scenarios involving lighting, occlusion, incompleteness, large deflection angles, and facial expressions during subsequent model training.

[0077] The network to be trained typically includes a feature point extraction layer for detecting sample images, a facial key point prediction layer based on the feature point extraction layer and its predicted points within a preset range, a facial key point detection layer for detecting key points in facial images, and a loss function. According to a preset iteration rule, the above sample images or data-enhanced sample images are input into the training unit 52 for training to obtain the trained facial key point detection model.

[0078] Training unit 52 includes: a feature point extraction subunit, which inputs the sample image into the feature point extraction layer to obtain the predicted facial key points output by the feature point extraction layer and the predicted points within a preset range, as well as the confidence scores corresponding to the predicted facial key points output by the feature point extraction layer and the predicted points within a preset range; a facial key point detection subunit, which obtains the facial key point detection results by taking the predicted facial key points and the predicted points within a preset range and the confidence scores corresponding to the predicted facial key points and the predicted points within a preset range; and a loss function construction subunit, which constructs a key point loss function based on the predicted facial key points and the predicted points within a preset range, the ground truth values ​​of the facial key points, and the ground truth values ​​of the points within a preset range of the facial key points, and terminates training based on the convergence of the key point loss function.

[0079] In summary, this embodiment of the invention guides the facial keypoint detection model to learn facial keypoints by using keypoint detection modules within a preset range of keypoints. This avoids interference from points within the preset range of facial keypoints on the model's learning process. Furthermore, it transforms the facial keypoint detection model from predicting a single facial keypoint to predicting both a single facial keypoint and points within a preset range, thereby increasing the information learned by the model and improving its robustness. Additionally, by weighting the confidence scores of the predicted facial keypoint and several surrounding points to obtain the facial keypoint detection result, the stability of facial keypoint detection is improved, avoiding large errors caused by small shifts and reducing the false detection rate.

[0080] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 61, a communication interface 62, a memory 63, and a communication bus 64. The processor 61, communication interface 62, and memory 63 communicate with each other via the communication bus 64. The processor 61 can call logical instructions in the memory 63 to execute a facial landmark detection method. This method includes: acquiring a facial image; inputting the facial image into a facial landmark detection model to obtain the facial landmark detection result output by the model; wherein the facial landmark detection model is trained based on sample images and the ground truth values ​​of facial landmarks corresponding to the sample images, as well as the ground truth values ​​of points within a preset range of facial landmarks; the facial landmark model is used to weight and output the facial landmark detection result based on the predicted facial landmarks from the facial image and points within a preset range.

[0081] Furthermore, the logical instructions in the aforementioned memory 63 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the facial landmark detection method provided by the above methods. The method includes: acquiring a facial image; inputting the facial image into a facial landmark detection model to obtain a facial landmark detection result output by the facial landmark detection model; wherein, the facial landmark detection model is trained based on a sample image and the ground truth values ​​of facial landmarks corresponding to the sample image and the ground truth values ​​of points within a preset range of facial landmarks; the facial landmark model is used to perform weighted output based on the facial landmarks predicted from the facial image and the points within a preset range to obtain the facial landmark detection result.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the facial landmark detection method provided by the above methods. The method includes: acquiring a face image; inputting the face image into a facial landmark detection model to obtain a facial landmark detection result output by the facial landmark detection model; wherein the facial landmark detection model is trained based on a sample image and the ground truth values ​​of the facial landmarks corresponding to the sample image and the ground truth values ​​of points within a preset range of the facial landmarks; the facial landmark model is used to perform weighted output based on the facial landmarks predicted from the face image and the points within a preset range to obtain the facial landmark detection result.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0085] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting facial landmarks, characterized in that, include: Acquire facial images; The face image is input into the face landmark detection model to obtain the face landmark detection result output by the face landmark detection model; The facial landmark detection model is trained based on sample images, the ground truth values ​​of facial landmarks corresponding to the sample images, and the ground truth values ​​of points within a preset range of the facial landmarks. The facial landmark model is used to perform weighted output based on the facial landmarks predicted from the facial image and points within a preset range to obtain the facial landmark detection result; The facial landmark detection model includes: The feature point extraction layer performs key point detection on the input face image to obtain the face key points and points within a preset range, as well as the confidence scores corresponding to the face key points and points within the preset range. The facial landmark detection layer obtains the facial landmark detection result based on the facial landmarks and points within a preset range, and the confidence scores corresponding to the facial landmarks and points within a preset range.

2. The facial landmark detection method according to claim 1, characterized in that, The step of performing key point detection on the input face image to obtain face key points and points within a preset range, and obtaining the confidence scores corresponding to the face key points and points within the preset range, includes: Feature extraction is performed on the input face image to obtain facial key points and points within a preset range; Furthermore, based on the facial key points and points within a preset range, a two-dimensional Gaussian distribution is used to obtain the confidence levels corresponding to the facial key points and points within a preset range.

3. The facial landmark detection method according to claim 1, characterized in that, The process of obtaining the facial landmark detection results includes: Based on the facial key points and points within a preset range, the facial key points are weighted and output by combining the confidence levels corresponding to the facial key points and points within the preset range, thus obtaining the facial key point detection result.

4. The facial landmark detection method according to claim 1, characterized in that, Training the facial landmark detection model includes: Obtain a sample image and the ground truth values ​​of the facial key points corresponding to the sample image, as well as the ground truth values ​​of the points within a preset range of the facial key points; The sample image is used as input data for the model to be trained. The ground truth values ​​of the facial key points and the ground truth values ​​of points within a preset range of the facial key points are used as labels to train the model to be trained, thereby obtaining a facial key point detection model for generating facial key point detection results for facial images.

5. The facial landmark detection method according to claim 4, characterized in that, The training of the model to be trained includes: The sample image is input to the feature point extraction layer to obtain the predicted facial key points output by the feature point extraction layer and the predicted points within a preset range, as well as the confidence levels corresponding to the predicted facial key points output by the feature point extraction layer and the predicted points within a preset range. The facial key point detection result is obtained by calculating the confidence scores corresponding to the facial key points and their predicted points within a preset range, and the predicted points of the facial key points and their predicted points within a preset range. A keypoint loss function is constructed based on the predicted points of the facial keypoints and the predicted points within a preset range, the ground truth values ​​of the facial keypoints and the ground truth values ​​of the points within the preset range of the facial keypoints, and the training ends when the keypoint loss function converges.

6. A facial landmark detection device, characterized in that, include: The image acquisition module acquires facial images; The key point detection module inputs the face image into the face key point detection model and obtains the face key point detection result output by the face key point detection model; The facial landmark detection model is trained based on sample images, the ground truth values ​​of facial landmarks corresponding to the sample images, and the ground truth values ​​of points within a preset range of the facial landmarks. The facial landmark model is used to perform weighted output based on the facial landmarks predicted from the facial image and points within a preset range to obtain the facial landmark detection result; The facial landmark detection model includes: The feature point extraction layer performs key point detection on the input face image to obtain the face key points and points within a preset range, as well as the confidence scores corresponding to the face key points and points within the preset range. The facial landmark detection layer obtains the facial landmark detection result based on the facial landmarks and points within a preset range, and the confidence scores corresponding to the facial landmarks and points within a preset range.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the face key point detection method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the face key point detection method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the face key point detection method as described in any one of claims 1 to 5.

Images