A face key point detection method and device

Abstract

This application relates to the field of face detection technology, and in particular to a method and apparatus for facial landmark detection. The method includes: acquiring a test image; extracting features from the test image at different scales using a preset facial landmark detection model to obtain image feature data at different scales, and generating face bounding box data, landmark location data, and landmark occlusion attribute data at different scales based on the image feature data at different scales; merging the face bounding box data, landmark location data, and landmark occlusion attribute data at different scales to obtain a facial landmark detection result; and determining the facial landmark detection result corresponding to the face confidence level as the target facial landmark detection result, provided that the face confidence level meets a preset confidence value. This method can simultaneously predict face bounding boxes, landmark locations, and landmark occlusion attributes based on a single model, improving the accuracy of facial landmark detection results and reducing the computing power required for facial landmark detection.

Description

Technical Field

[0001] This application relates to the field of face detection technology, and in particular to a method and apparatus for detecting facial key points. Background Technology

[0002] Facial landmark detection is increasingly being applied in various terminal scenarios, such as in-vehicle facial recognition. Facial landmark detection requires predicting bounding boxes and facial landmarks based on the input image.

[0003] Typically, a face bounding box detection model is used to predict the face bounding box corresponding to a face. If a face bounding box is predicted, its coordinates are output. A keypoint detection model is used to identify key points (such as the nose) in the face image (i.e., the face image corresponding to the face bounding box). A keypoint occlusion attribute detection model is typically used to further predict the occlusion attributes (such as mask occlusion) of facial keypoints based on the face bounding box and facial keypoints. However, this approach ignores the relationship between face bounding box prediction, facial keypoint prediction, and keypoint occlusion attribute prediction, reducing the overall detection efficiency of face-related detection. Furthermore, using multiple models increases the computational burden on the terminal computing device, and the computing speed of existing terminal computing devices is usually unable to support multiple model predictions.

[0004] Therefore, there is a need to provide a method and apparatus for facial landmark detection that can simultaneously predict the face bounding box, landmark positions, and landmark occlusion attributes based on a single model, thereby ensuring the accuracy of facial landmark detection results and reducing the computing power required for facial landmark detection. Summary of the Invention

[0005] This application provides a method and apparatus for facial landmark detection, which can simultaneously predict the face bounding box, landmark position, and landmark occlusion attribute based on a single model, ensuring the accuracy of facial landmark detection results and reducing the computing power required for facial landmark detection.

[0006] This application provides a method for detecting facial landmarks, the method comprising:

[0007] Acquire the image to be tested;

[0008] By using a preset facial key point detection model, features at different scales are extracted from the image under test to obtain image feature data at different scales.

[0009] The preset facial landmark detection model generates facial bounding box data, landmark location data, and landmark occlusion attribute data at different scales based on the image feature data at different scales; the facial bounding box data includes the facial bounding box location data and facial confidence score.

[0010] By merging the face bounding box data, the key point location data, and the key point occlusion attribute data at different scales, the face key point detection result of the image under test is obtained.

[0011] If the face confidence score meets the preset confidence score value, the face key point detection result of the face corresponding to the face confidence score is determined as the target face key point detection result.

[0012] In some optional embodiments, before determining the facial landmark detection result of the face corresponding to the face confidence score as the target facial landmark detection result, the method further includes:

[0013] Based on nonmaximum suppression, the face bounding box position data is deduplicated.

[0014] In some optional embodiments, the method further includes:

[0015] The target facial landmark detection results are mapped to the corresponding image to be tested, and output through the preset facial landmark detection model.

[0016] In some optional embodiments, the method further includes: training the preset facial landmark detection model;

[0017] Training the preset facial landmark detection model includes:

[0018] Acquire multiple face sample images;

[0019] Data annotation is performed on each face sample image in the plurality of face sample images to obtain face bounding box location information, key point location information, and key point occlusion attribute information corresponding to each face sample image; the key point location information includes the location information of a preset number of key points;

[0020] The network is trained based on the multiple face sample images, the face bounding box position information of each face sample image, the key point position information, the key point occlusion attribute information, and the preset face key point detection model.

[0021] In some optional embodiments, the preset facial landmark detection model includes a data augmentation algorithm; the data augmentation algorithm is used to perform data augmentation processing on the multiple facial sample images; the data augmentation algorithm includes at least one data augmentation processing method selected from mosaic enhancement and horizontal flipping.

[0022] In some optional embodiments, the network training includes calculating a first loss function for the face bounding box location information, a second loss function for the keypoint location information, a third loss function for the keypoint occlusion attribute information, a fourth loss function for the face confidence, and a fifth loss function for the object category within the face bounding box; and

[0023] The parameters of the preset facial landmark detection model are updated based on the first loss function, the second loss function, the third loss function, the fourth loss function, and the fifth loss function.

[0024] In some optional embodiments, the second loss function for calculating the key point location information includes occlusion attribute weights and feature weights for determining the key point location; the occlusion attribute weights are determined based on the key point occlusion attribute information corresponding to the key point location; and the feature weights are determined based on the facial features corresponding to the key point location.

[0025] In some alternative embodiments, the keypoint occlusion attribute information includes at least one of exceeding image boundaries, occlusion, and visibility.

[0026] In some optional embodiments, the keypoint occlusion attribute data includes a visibility score and keypoint occlusion attribute information; the visibility score is used to characterize the visibility degree of the keypoint; the keypoint occlusion attribute information is determined based on the visibility score.

[0027] This application provides a facial landmark detection device, which includes:

[0028] The acquisition module is used to acquire the image to be tested;

[0029] The extraction module is used to extract features from the image under test at different scales using a preset facial key point detection model, so as to obtain image feature data at different scales.

[0030] The detection module is used to generate face bounding box data, key point location data, and key point occlusion attribute data at different scales based on the image feature data at different scales using the preset face key point detection model; the face bounding box data includes face bounding box location data and face confidence.

[0031] The merging module is used to merge the face bounding box data, the key point location data, and the key point occlusion attribute data at different scales to obtain the face key point detection result corresponding to the image under test;

[0032] The determination module is used to determine the facial landmark detection result of the face corresponding to the face confidence as the target facial landmark detection result when the face confidence meets the preset confidence value.

[0033] In some optional embodiments, the determining module further includes:

[0034] Based on nonmaximum suppression, the face bounding box position data is deduplicated.

[0035] In some optional embodiments, the method further includes:

[0036] The target facial landmark detection results are mapped to the corresponding image to be tested, and output through the preset facial landmark detection model.

[0037] In some optional embodiments, the apparatus further includes a training module for training the preset facial landmark detection model;

[0038] Training the preset facial landmark detection model includes:

[0039] Acquire multiple face sample images;

[0040] Data annotation is performed on each face sample image in the plurality of face sample images to obtain face bounding box location information, key point location information, and key point occlusion attribute information corresponding to each face sample image; the key point location information includes the location information of a preset number of key points;

[0041] The network is trained based on the multiple face sample images, the face bounding box position information of each face sample image, the key point position information, the key point occlusion attribute information, and the preset face key point detection model.

[0042] In some optional embodiments, the preset facial landmark detection model includes a data augmentation algorithm; the data augmentation algorithm is used to perform data augmentation processing on the multiple facial sample images; the data augmentation algorithm includes at least one data augmentation processing method selected from mosaic enhancement and horizontal flipping.

[0043] In some optional embodiments, the network training includes calculating a first loss function for the face bounding box location information, a second loss function for the keypoint location information, a third loss function for the keypoint occlusion attribute information, a fourth loss function for the face confidence, and a fifth loss function for the object category within the face bounding box; and

[0044] The parameters of the preset facial landmark detection model are updated based on the first loss function, the second loss function, the third loss function, the fourth loss function, and the fifth loss function.

[0045] In some optional embodiments, the second loss function for calculating the key point location information includes occlusion attribute weights and feature weights for determining the key point location; the occlusion attribute weights are determined based on the key point occlusion attribute information corresponding to the key point location; and the feature weights are determined based on the facial features corresponding to the key point location.

[0046] In some alternative embodiments, the keypoint occlusion attribute information includes at least one of exceeding image boundaries, occlusion, and visibility.

[0047] In some optional embodiments, the keypoint occlusion attribute data includes a visibility score and keypoint occlusion attribute information; the visibility score is used to characterize the visibility degree of the keypoint; the keypoint occlusion attribute information is determined based on the visibility score.

[0048] This application provides a method and apparatus for facial landmark detection, which has the following advantages: It acquires a test image; extracts features from the test image at different scales using a preset facial landmark detection model, obtaining image feature data at different scales; generates face bounding box data, landmark location data, and landmark occlusion attribute data at different scales based on the image feature data at different scales using the preset facial landmark detection model; the face bounding box data includes face bounding box location data and face confidence; by merging face bounding box data, landmark location data, and landmark occlusion attribute data at different scales, it obtains the facial landmark detection result corresponding to the test image; and by determining the facial landmark detection result of the face corresponding to the face confidence as the target facial landmark detection result when the face confidence meets a preset confidence value. It can simultaneously predict face bounding boxes, landmark locations, and landmark occlusion attributes based on a single model, ensuring the accuracy of the facial landmark detection results; and reducing the computing power required for facial landmark detection. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram of facial landmark detection;

[0051] Figure 2 This is a flowchart illustrating a facial landmark detection method provided in an embodiment of this application.

[0052] Figure 3This is a schematic diagram of a process for outputting facial landmark detection results provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of a process for training a preset facial landmark detection model according to an embodiment of this application;

[0054] Figure 5 This application provides a schematic diagram of a preset facial landmark detection model for facial landmark detection;

[0055] Figure 6 This is a schematic diagram of the structure of a facial landmark detection device provided in an embodiment of this application;

[0056] Figure 7 This is a hardware structure block diagram of a server for a face key point detection method provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0058] It should be noted that the terms "first," "", etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0059] Figure 1 This is a schematic diagram of facial landmark detection, such as... Figure 1 As shown, the image under test is used to predict the corresponding face bounding boxes of the face using a face bounding box prediction model, thus obtaining face bounding box data. For example, the face bounding box data includes... Figure 1 The identifier 'a' and the coordinates of the face bounding box are shown in the figure (not shown).

[0060] When predicting keypoint occlusion attributes, if the number of keypoints exceeds 5, it is usually necessary to first obtain a face image using a face bounding box prediction model, and then perform keypoint occlusion attribute prediction on the face image. For example... Figure 1 As shown, the face image is obtained from the image to be tested based on the face bounding box coordinates. The key point detection model is used to predict the key point positions of the face image to obtain key point position data. The key point position data includes key point identifiers and key point coordinates, such as the coordinates of key point identifier B and key point B (not shown in the figure).

[0061] like Figure 1 As shown, a keypoint occlusion attribute detection model is used to predict the occlusion attributes of keypoints obtained by the keypoint detection model. For example, keypoint occlusion attributes include occlusion (…). Figure 1 The keypoint B obtained by the keypoint occlusion attribute detection model is gray (representing occlusion) and visible (…). Figure 1 The keypoint C obtained by the keypoint occlusion attribute detection model is colored black (representing visibility). Keypoint occlusion attribute prediction can provide additional effective keypoint information for pose assessment, fatigue detection, and face recognition.

[0062] As mentioned earlier, face bounding box prediction, keypoint location prediction, and keypoint occlusion attribute prediction are currently implemented using different models. This results in two problems: firstly, face keypoint detection ignores the relationship between these three processes, reducing overall detection efficiency; secondly, face keypoint detection requires multiple models, consuming significant computational resources and time, making it difficult to deploy on mobile devices (such as vehicles).

[0063] To address the aforementioned problems, this application provides a method for facial landmark detection, comprising: acquiring a test image; extracting features from the test image at different scales using a preset facial landmark detection model to obtain image feature data at different scales; generating face bounding box data, landmark location data, and landmark occlusion attribute data at different scales based on the image feature data at different scales using the preset facial landmark detection model; the face bounding box data including face bounding box location data and face confidence; merging face bounding box data, landmark location data, and landmark occlusion attribute data at different scales to obtain the facial landmark detection result corresponding to the test image; and determining the facial landmark detection result of the face corresponding to the face confidence as the target facial landmark detection result when the face confidence meets a preset confidence value. This method can simultaneously predict face bounding boxes, landmark locations, and landmark occlusion attributes based on a single model, ensuring the accuracy of the facial landmark detection results and reducing the computing power required for facial landmark detection.

[0064] The following describes a specific embodiment of a facial landmark detection method according to this application. Figure 2 This is a flowchart illustrating a facial landmark detection method provided in an embodiment of this application. This specification provides the method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual system or server products, the method can be executed in the order shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Figure 2 This is a schematic diagram illustrating the application of a facial landmark detection method provided in an embodiment of this application. The following is a description of... Figure 2 The method shown will be described in detail, specifically as follows: Figure 2 As shown, the method may include:

[0065] S202: Acquire the image to be tested.

[0066] For example, the image to be tested is based on a video stream inside the vehicle; or an infrared facial image of the person inside the vehicle.

[0067] S204: By using a preset facial landmark detection model, feature extraction is performed on the image under test at different scales to obtain image feature data at different scales.

[0068] Figure 5 This application provides a schematic diagram illustrating the use of a preset facial landmark detection model for facial landmark detection. For example... Figure 5 As shown, the preset facial landmark detection model extracts features from the input image, that is, it extracts features at different scales to obtain image feature data at different scales. An input image to be tested is fed into the feature extraction network to obtain image feature data at different scales (i.e.,... Figure 5 The symbols shown are “Stride_8_out”, “Stride_16_out”, and “Stride_32_out”.

[0069] In some optional embodiments, the mobile-friendly ShuffleNetV2 network structure is adopted, reducing the number of backbone network layers in terms of network depth and the number of network channels in terms of network width. A structured global pruning method is then used to prune and train the compressed network again. This improves the running speed on chips with lower computing power while maintaining the effectiveness of facial landmark detection, reducing computational resource consumption and latency.

[0070] In this way, the three tasks share a single feature extraction network. Simultaneously, face bounding box prediction, keypoint location prediction, and keypoint occlusion attribute prediction are performed based on the image feature data obtained from the feature extraction network, eliminating the need to load multiple models and avoiding preprocessing operations for each model. The ShuffleNetV2-based network structure, accelerated by the NCNN framework, achieves real-time performance on CPU. For a 1920*1080 input image, the CPU achieves over 50 FPS (test environment: Ubuntu 18.04 system, Intel(R) Xeon(R) Gold 6248R CPU@3.00GHz).

[0071] S206: Generate face bounding box data, key point location data, and key point occlusion attribute data at different scales based on image feature data of different scales using a pre-set face key point detection model. The face bounding box data includes the face bounding box location data and face confidence score.

[0072] In some optional embodiments, the preset facial landmark detection model is obtained based on the YOLOv5Face face detection algorithm. A landmark occlusion attribute (classifier head) is added to the YOLOv5Face structure, along with a loss calculation method corresponding to the landmark occlusion attribute. For other physical occlusions such as closed eyes, pose occlusion, and masks, a corresponding visibility score can be given based on the degree of occlusion. The landmark visibility score is used to characterize the visibility degree of the landmarks.

[0073] The pre-defined facial landmark detection model is based on three anchors of different sizes for image feature data at three different scales. Figure 5 The prediction is performed using the Anchor shown, resulting in three different scales of prediction results (i.e., "Stride_8_pred", "Stride_16_pred", and "Stride_32_pred", which are arrays of size 19200*C, 4800*C, and 1200*C, respectively, where C contains information such as the face bounding box data to be predicted, key point location data, and key point occlusion attribute data). For example, C includes the face bounding box coordinates and the face bounding box score, i.e., the face confidence score, which indicates the probability that complete face data exists within the predicted face bounding box region. Figure 1 The face confidence value of the face bounding box shown is 0.9.

[0074] S208: Merge face bounding box data, key point location data, and key point occlusion attribute data of different scales to obtain the face key point detection results corresponding to the image under test.

[0075] For example, such as Figure 5As shown, the results of merging (sorting) different scales are "Stride_8_pred", "Stride_16_pred" and "Stride_32_pred" mentioned above.

[0076] S210: If the face confidence score meets the preset confidence score value, determine the face landmark detection result of the face corresponding to the face confidence score as the target face landmark detection result.

[0077] Specifically, calculate the face confidence score corresponding to each face bounding box, sort them from high to low confidence scores, and remove the face key point results with confidence scores below the threshold.

[0078] For example, the output of facial landmark results includes an array of [n×(6+3N)], where n represents the number of faces detected in the image to be tested, and N represents the number of landmarks.

[0079] To improve the accuracy of the output facial landmark results, it is necessary to filter the facial landmark detection results. Figure 3 This is a flowchart illustrating the output of facial landmark detection results provided in an embodiment of this application, such as... Figure 3 As shown, the steps for outputting the facial landmark detection results are as follows:

[0080] S2101: The face confidence score meets the preset confidence score value. Specifically, the face confidence score corresponding to the face bounding box meets the preset confidence score value.

[0081] S2102: Based on nonmaximum suppression, deduplication is performed on the face bounding box location data.

[0082] For example, execute Figure 5 The NMS (Non-maximum Suppression) shown is used to remove duplicates from the face bounding box data.

[0083] S2103: Map the target face key point detection results to the corresponding image to be tested.

[0084] For example, the face bounding boxes and keypoint detection results of the network input size are mapped back to the original test image and labeled. The final output results for n faces corresponding to the original test image are obtained. The output data may include face bounding box location data (e.g., coordinates), face confidence score, coordinates of N keypoints, visibility scores of N keypoints (which can be used to characterize keypoint occlusion attributes), and the category. The category is "face".

[0085] For example, keypoint occlusion attributes can be categorized based on a keypoint visibility score threshold (e.g., 0.5). A score less than 0.5 indicates low keypoint visibility, meaning the keypoint is occluded. Figure 1The visibility score of key point B shown is 0.3. The corresponding label color for key point B is gray, which is used to indicate that key point B is occluded. Figure 1 The eyes of the face within the frame are closed, which is one type of occlusion. Other types of occlusion include those caused by hands, posture, or masks.

[0086] Figure 1 As shown, we can see the face bounding box a, the face confidence (the face confidence corresponding to face bounding box a is 0.9), the key point location (such as key point B), and the corresponding key point visibility score (such as the visibility score of key point B is 0.3).

[0087] In order to obtain a preset facial landmark detection model with good facial landmark detection performance, some embodiments of this application provide a method for training the preset facial landmark detection model. Figure 4 This is a flowchart illustrating a process for training a preset facial landmark detection model, as provided in an embodiment of this application; Figure 4 As shown, the method for training the preset facial landmark detection model is as follows:

[0088] S401: Acquire multiple face sample images.

[0089] In some optional embodiments, multiple face sample images can be selected based on the application scenario of a preset face landmark detection model.

[0090] S402: Perform data annotation on each face sample image in multiple face sample images to obtain the face bounding box location information, key point location information, and key point occlusion attribute information corresponding to each face sample image; the key point location information includes the location information of a preset number of key points.

[0091] The aforementioned key point location information includes the location information of a preset number of key points, avoiding the cumbersome data processing process caused by different key point definitions.

[0092] S403: The network is trained based on multiple face sample images, the face bounding box location information, key point location information, key point occlusion attribute information of each face sample image, and a preset face key point detection model.

[0093] In the pre-defined face landmark detection model, the number of landmarks is used as a variable parameter when creating the detection network. For example, the network structure is based on YOLOv5-face, with the number of landmarks as an input parameter, and occlusion attribute prediction is added for N landmarks.

[0094] In some optional embodiments, the preset facial landmark detection model includes a data augmentation algorithm; the data augmentation algorithm is used to perform data augmentation processing on multiple facial sample images; the data augmentation algorithm includes at least one data augmentation processing method selected from mosaic enhancement and horizontal flipping.

[0095] For example, the data augmentation algorithm uses the Mosaic data augmentation algorithm; this algorithm randomly crops multiple images and stitches them together into a single image as training data; the left-right flipping algorithm is used to improve the accuracy of key point localization in face detection. Specifically, based on variable key point parameters (such as the number of key points), corresponding label information (such as eyes) is selected, and the Mosaic and left-right flipping algorithms are modified to support input of arbitrary key point coordinates. For images captured inside a vehicle, Mosaic augmentation is effective for both face recall and key point enhancement.

[0096] It should be noted that key points and face bounding boxes are strongly dependent on each other; however, as the number of key points increases, the accuracy of face detection is almost unaffected.

[0097] In some optional embodiments, the network training includes calculating a first loss function for face bounding box location information, a second loss function for key point location information, a third loss function for key point occlusion attribute information, a fourth loss function for confidence level, and a fifth loss function for object category within the face bounding box; and updating relevant parameters of the preset face key point detection model based on the first, second, third, fourth, and fifth loss functions.

[0098] In some optional embodiments, the second loss function used to calculate the key point location information includes occlusion attribute weights and feature weights for determining the key point location; the occlusion attribute weights are determined based on the key point occlusion attribute information corresponding to the key point location; and the feature weights are determined based on the facial features corresponding to the key point location.

[0099] In this embodiment, the keypoint detection method determines that the keypoint attribute is occluded based on the face occlusion attribute. Data augmentation is then applied to the localization of keypoints in the corresponding occluded area to improve the accuracy of keypoint location prediction for large poses and occluded faces. This avoids the significant errors that can occur with manual annotation of invisible keypoints, making it difficult to obtain the true coordinates of the keypoints and resulting in unreliable keypoint location prediction results.

[0100] In some alternative embodiments, the keypoint occlusion attribute includes at least one of exceeding image boundaries, occlusion, and visibility.

[0101] For example, different occlusion attribute weights are defined for three different types: exceeding image boundaries, occlusion, and no occlusion. Specifically, the YOLOv5Face keypoint location prediction loss calculation method is modified to add a keypoint occlusion attribute weight parameter. This suppresses the prediction loss of occluded keypoints, making keypoint location prediction more accurate.

[0102] In some optional embodiments, the keypoint occlusion attribute data includes a visibility score and keypoint occlusion attribute information; the visibility score is used to characterize the visibility degree of the keypoint; the keypoint occlusion attribute information is determined based on the visibility score.

[0103] For example, keypoint occlusion attributes include occlusion, beyond image boundaries (outside the image), and visible.

[0104] The following is an example of loss calculation in the network training of the aforementioned preset facial landmark detection model.

[0105] In some optional embodiments, in order to perform keypoint location prediction and keypoint occlusion attribute prediction, the keypoint loss function is defined as follows:

[0106] Based on the original YOLOv5Face keypoint loss function (the second loss function mentioned above), supervised training is performed by incorporating keypoint occlusion attributes, i.e., keypoint location prediction is based on keypoint occlusion attributes. Using YOLOv5Face as the keypoint location prediction method makes the loss function more sensitive to small errors and results in more accurate localization.

[0107] The Wing loss for YOLO5Face keypoint location prediction is calculated as shown in formula (1):

[0108]

[0109] Specifically, when the error between the predicted and true values ​​is less than the parameter W (w = 10), it is represented by a logarithmic function with an offset (where the parameter e = 2 is fixed); when the error x is greater than w, it is represented by L1 regression loss, with the constant C = ww * ln(1 + w / e). L1 regression loss (Mean Absolute Error, MAE) is used to characterize the average error magnitude in the predicted values.

[0110] The keypoint loss function (i.e., the second loss function) is as shown in the following formula (2):

[0111]

[0112] Among them, s i ′ represents the true value of the i-th coordinate, and s represents the predicted value of the i-th coordinate.

[0113] In the above formula (2), α is added to the original formula (1) for calculating the loss at key points. i and β i Two weighting parameters. Where α... i For the occlusion attribute weight of the i-th keypoint, different occlusion attribute weight parameters are given for keypoints with different attributes such as being outside the image boundary, occluded, and visible. This reduces the problem of excessive keypoint loss caused by occlusion and other reasons, and focuses more on the position prediction accuracy of visible keypoints. β i The additional weight parameter (i.e. the feature weight mentioned above) for the i-th key point is based on facial feature points (such as the lowest point of the chin, the highest point of the nose, and the edge points of the left and right cheeks).

[0114] The aforementioned additional weighting parameters are used to adjust for keypoint position errors caused by indistinct texture features. Specifically, because keypoints with indistinct texture features have larger position errors during manual keypoint marking, parameter β... i The loss weights of specific key points corresponding to texture features can be adjusted.

[0115] For example, in driver monitoring scenarios, pupil coordinates are particularly important for eye-tracking, which can be achieved by increasing β. i To improve the accuracy of pupil localization (i.e., key point location prediction).

[0116] The third loss function mentioned above occ The visibility loss calculation includes keypoint occlusion attribute classification, as shown in formula (3):

[0117]

[0118] Where, p occ Let t be the predicted occlusion score (i.e., occlusion classification, such as occlusion, outside the image, or visible) for the i-th keypoint. occ For the true label (0,1) of the i-th keypoint, mask occ σ indicates whether the keypoint has a true value (0 indicates no true value, 1 indicates true value), σ represents the Sigmoid (an activation function used for variable mapping) applied to the prediction result, and BCE is the binary cross-entropy loss.

[0119] The total loss of the above-mentioned preset face key point detection model is calculated as follows (4):

[0120] loss = λ box loss box +λ obj loss obj +λ cls losscls +λ lmk loss lmk +λ occ loss occ (4)

[0121] Wherein, the total loss is based on the first loss function. box (Regarding the face bounding box position), the second loss function is... occ (For key point locations), the third loss function (loss) lmk (Regarding key point occlusion attributes), the fourth loss function obj (Regarding face confidence) and the fifth loss function cls (For the detection category), and the total loss is calculated based on the self-weighting factors corresponding to the five loss functions, such as the fourth loss function loss. obj The corresponding weighting factor λ obj .

[0122] This application provides a method for facial landmark detection, including acquiring a test image; extracting features from the test image at different scales using a preset facial landmark detection model to obtain image feature data at different scales; generating face bounding box data, landmark position data, and landmark occlusion attribute data at different scales based on the image feature data at different scales using the preset facial landmark detection model; and obtaining the facial landmark detection result corresponding to the test image by merging the face bounding box data, landmark position data, and landmark occlusion attribute data at different scales. The face bounding box data includes face bounding box position data and face confidence; and determining the facial landmark detection result of the face corresponding to the face confidence as the target facial landmark detection result when the face confidence meets a preset confidence value. This method can simultaneously predict face bounding boxes, landmark positions, and landmark occlusion attributes based on a single model, ensuring the accuracy of the facial landmark detection results and reducing the computing power required for facial landmark detection.

[0123] Some embodiments of this application provide a facial landmark detection device. Figure 6 This is a schematic diagram of the structure of a facial landmark detection device provided in an embodiment of this application; as shown... Figure 6 The device shown includes:

[0124] The acquisition module 501 is used to acquire the image to be tested;

[0125] The extraction module 502 is used to extract features at different scales from the image under test using a preset face key point detection model, so as to obtain image feature data at different scales.

[0126] The detection module 503 is used to generate face bounding box data, key point location data, and key point occlusion attribute data at different scales based on image feature data at different scales using a preset face key point detection model; the face bounding box data includes face bounding box location data and face confidence.

[0127] The merging module 504 is used to merge face bounding box data, key point location data and key point occlusion attribute data of different scales to obtain the face key point detection results corresponding to the image under test;

[0128] The determination module 505 is used to determine the facial key point detection result of the face corresponding to the face confidence as the target facial key point detection result when the face confidence meets the preset confidence value.

[0129] In some optional embodiments, the determining module 505 further includes:

[0130] Based on nonmaximum suppression, the face bounding box position data is deduplicated.

[0131] In some optional embodiments, the method further includes:

[0132] The target facial landmark detection results are mapped to the corresponding image to be tested.

[0133] In some optional embodiments, the apparatus further includes a training module for training the preset facial landmark detection model;

[0134] Training the preset facial landmark detection model includes:

[0135] Acquire multiple face sample images;

[0136] Data annotation is performed on each face sample image in the plurality of face sample images to obtain face bounding box location information, key point location information, and key point occlusion attribute information corresponding to each face sample image; the key point location information includes the location information of a preset number of key points;

[0137] The network is trained based on the multiple face sample images, the face bounding box position information of each face sample image, the key point position information, the key point occlusion attribute information, and the preset face key point detection model.

[0138] In some optional embodiments, the preset facial landmark detection model includes a data augmentation algorithm; the data augmentation algorithm is used to perform data augmentation processing on the multiple facial sample images; the data augmentation algorithm includes at least one data augmentation processing method selected from mosaic enhancement and horizontal flipping.

[0139] In some optional embodiments, the network training includes calculating a first loss function for the face bounding box location information, a second loss function for the keypoint location information, a third loss function for the keypoint occlusion attribute information, a fourth loss function for the face confidence, and a fifth loss function for the object category within the face bounding box; and

[0140] The parameters of the preset facial landmark detection model are updated based on the first loss function, the second loss function, the third loss function, the fourth loss function, and the fifth loss function.

[0141] In some optional embodiments, the second loss function for calculating the key point location information includes occlusion attribute weights and feature weights for determining the key point location; the occlusion attribute weights are determined based on the key point occlusion attribute information corresponding to the key point location; and the feature weights are determined based on the facial features corresponding to the key point location.

[0142] In some alternative embodiments, the keypoint occlusion attribute information includes at least one of exceeding image boundaries, occlusion, and visibility.

[0143] In some optional embodiments, the keypoint occlusion attribute data includes a visibility score and keypoint occlusion attribute information; the visibility score is used to characterize the visibility degree of the keypoint; the keypoint occlusion attribute information is determined based on the visibility score.

[0144] The apparatus and method embodiments in this application are based on the same application concept.

[0145] The methods and embodiments provided in this application can be executed on a computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 7 This is a hardware structure block diagram of a server for a facial landmark detection method provided in an embodiment of this application. For example... Figure 7As shown, the server 700 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 710 (CPUs 710 may include, but are not limited to, microprocessors NCUs or programmable logic devices FPGAs), a memory 730 for storing data, and one or more storage media 720 (e.g., one or more mass storage devices) for storing application programs 723 or data 722. The memory 730 and storage media 720 may be temporary or persistent storage. The program stored in the storage media 720 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 710 may be configured to communicate with the storage media 720 and execute the series of instruction operations stored in the storage media 720 on the server 700. Server 700 may also include one or more power supplies 760, one or more wired or wireless network interfaces 750, one or more input / output interfaces 740, and / or one or more operating systems 721, such as Windows, Mac OS, Unix, Linux, FreeBSD, etc.

[0146] The input / output interface 740 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 700. In one example, the input / output interface 740 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 740 may be a radio frequency (RF) module for wireless communication with the Internet.

[0147] Those skilled in the art will understand that Figure 7 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 700 may also include... Figure 7 The more or fewer components shown, or having the same Figure 7 The different configurations shown.

[0148] This application provides a method and apparatus for facial landmark detection. The method includes acquiring a test image; extracting features from the test image at different scales using a preset facial landmark detection model to obtain image feature data at different scales; generating face bounding box data, landmark location data, and landmark occlusion attribute data at different scales based on the image feature data at different scales using the preset facial landmark detection model; the face bounding box data includes face bounding box location data and face confidence; merging face bounding box data, landmark location data, and landmark occlusion attribute data at different scales to obtain the facial landmark detection result corresponding to the test image; and determining the facial landmark detection result of the face corresponding to the face confidence as the target facial landmark detection result when the face confidence meets a preset confidence value. This method can simultaneously predict face bounding boxes, landmark locations, and landmark occlusion attributes based on a single model, ensuring the accuracy of the facial landmark detection results and reducing the computing power required for facial landmark detection.

[0149] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0150] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0151] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0152] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for detecting facial landmarks, characterized in that, The method includes: Acquire the image to be tested; By using a preset facial key point detection model, features at different scales are extracted from the image under test to obtain image feature data at different scales. The preset facial landmark detection model generates facial bounding box data, landmark location data, and landmark occlusion attribute data at different scales based on the image feature data at different scales; the facial bounding box data includes the facial bounding box location data and facial confidence score. By merging the face bounding box data, the key point location data, and the key point occlusion attribute data at different scales, the face key point detection result of the image under test is obtained. When the face confidence score meets a preset confidence value, the face key point detection result corresponding to the face confidence score is determined as the target face key point detection result; the preset face key point detection model is obtained in the following way: acquiring multiple face sample images; performing data annotation on each face sample image in the multiple face sample images to obtain the face bounding box position information, key point position information, and key point occlusion attribute information corresponding to each face sample image; calculating the first loss function of the face bounding box position information, the second loss function of the key point position information, the third loss function of the key point occlusion attribute information, the fourth loss function of the face confidence score, and the fifth loss function of the object category within the face bounding box; based on the... The first loss function, the second loss function, the third loss function, the fourth loss function, and the fifth loss function update the relevant parameters of the preset facial landmark detection model. The second loss function is the Wing-loss landmark loss function multiplied by the occlusion attribute weight and feature weight of each landmark. The occlusion attribute weight is determined based on the landmark occlusion attribute information corresponding to the landmark position. The landmark occlusion attribute information is determined based on the visibility score, which is used to characterize the visibility degree of the landmark. The feature weight is determined based on the importance of the facial features corresponding to the landmark position. The more important the facial features corresponding to the landmark position, the greater the feature weight of the landmark.

2. The method according to claim 1, characterized in that, Before determining the facial landmark detection result of the face corresponding to the facial confidence score as the target facial landmark detection result, the method further includes: Based on nonmaximum suppression, the face bounding box position data is deduplicated.

3. The method according to claim 1, characterized in that, The method further includes: The target facial landmark detection results are mapped to the corresponding image to be tested.

4. The method according to any one of claims 1 to 3, characterized in that, The key point location information includes the location information of a preset number of key points.

5. The method according to claim 4, characterized in that, The preset facial landmark detection model includes a data augmentation algorithm; the data augmentation algorithm is used to perform data augmentation processing on the multiple facial sample images; the data augmentation algorithm includes at least one data augmentation processing method among mosaic enhancement and left-right flipping.

6. The method according to claim 1, characterized in that, The keypoint occlusion attribute information includes at least one of the following: beyond image boundary, occlusion, and visibility.

7. A facial landmark detection device, characterized in that, The device includes: The acquisition module is used to acquire the image to be tested; The extraction module is used to extract features from the image under test at different scales using a preset facial key point detection model, so as to obtain image feature data at different scales. The detection module is used to generate face bounding box data, key point location data, and key point occlusion attribute data at different scales based on the image feature data at different scales using the preset face key point detection model; the face bounding box data includes face bounding box location data and face confidence. The merging module is used to merge the face bounding box data, the key point location data, and the key point occlusion attribute data at different scales to obtain the face key point detection result of the image under test; The determination module is used to determine the facial landmark detection result of the face corresponding to the face confidence as the target facial landmark detection result when the face confidence meets a preset confidence value. The preset facial landmark detection model is obtained using the following training module, which is used to acquire multiple face sample images; perform data annotation on each face sample image in the multiple face sample images to obtain the face bounding box position information, landmark position information, and landmark occlusion attribute information corresponding to each face sample image; calculate the first loss function of the face bounding box position information, the second loss function of the landmark position information, the third loss function of the landmark occlusion attribute information, the fourth loss function of the face confidence, and the fifth loss function of the object category within the face bounding box. The loss function is used to update the relevant parameters of the preset facial landmark detection model based on the first loss function, the second loss function, the third loss function, the fourth loss function, and the fifth loss function. The second loss function is the Wing-loss landmark loss function multiplied by the occlusion attribute weight and feature weight of each landmark. The occlusion attribute weight is determined based on the occlusion attribute information of the landmark corresponding to the landmark position. The occlusion attribute information is determined based on the visibility score, which is used to characterize the visibility degree of the landmark. The feature weight is determined based on the importance of the facial features corresponding to the landmark position. The more important the facial features corresponding to the landmark position, the greater the feature weight of the landmark.

Images