Face key point detection method and device, electronic equipment and storage medium

Abstract

This invention relates to a method, apparatus, electronic device, and storage medium for detecting facial key points. The method includes: acquiring a face image to be detected; inputting the face image into a preset dense point prediction model to obtain dense points corresponding to the face image; generating a dense point heatmap corresponding to the face image based on the dense points; and inputting the dense point heatmap into a preset sparse point prediction model to obtain key points corresponding to the face image. This improves the accuracy of facial key point detection and enhances the user experience.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for detecting facial key points. Background Technology

[0002] Facial landmark detection refers to the precise localization of facial regions, including facial features and contours such as cheeks, eyebrows, eyes, mouth, and nose. Since the first application of CNN (Convolutional Neural Network) technology to facial landmark detection in 2013, this technology has undergone rapid development, and the technical methods have become relatively mature. Both fast detection technologies applied to mobile devices and high-precision detection technologies applied to the cloud have achieved rapid progress.

[0003] With the development of facial landmark technology, it has been applied to various scenarios. For example, the landmark technology can be used to divide the face into areas such as eyebrows, eyes, mouth, and cheeks to achieve facial beautification effects. The outer contour area of ​​the face can be used for face slimming. In addition, there are many other application scenarios, which are widely distributed in the fields of short videos, long videos, and photography.

[0004] However, existing facial landmark detection technologies still suffer from inaccuracies and jitter in some points during practical applications, especially in edge-test models. To meet the speed requirements of edge testing, the models are often lightweight. For example, with 106 points, although the points are marked on the edge of the face contour, the discrete points mean that the gaps between the points are large. This makes it difficult for the key points extracted by the facial landmark detection model to express accurate semantic information, resulting in low accuracy and a tendency to make errors. Summary of the Invention

[0005] In view of this, in order to solve the technical problem that the key points extracted by the existing facial key point detection models often cannot express accurate semantic information, have low accuracy, and are prone to deviation, the present invention provides a method, device, electronic device and storage medium for detecting facial key points.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting facial key points, the method comprising:

[0007] Acquire the face image to be detected;

[0008] The face image is input into a preset dense point prediction model to obtain the dense points corresponding to the face image;

[0009] Based on the dense points, a dense point heatmap corresponding to the face image is generated;

[0010] The dense point heatmap is input into a preset sparse point prediction model to obtain the key points corresponding to the face image.

[0011] As one possible implementation, the dense point prediction model includes a base network layer, a fully connected layer, and a dense point prediction layer. The step of inputting the face image into the preset dense point prediction model to obtain the dense points corresponding to the face image includes:

[0012] The face image is input into the base network layer to obtain the base network features output by the base network layer;

[0013] The basic network features are input into the fully connected layer to obtain the dense point features output by the fully connected layer;

[0014] The dense point features are input into the dense point prediction layer to obtain the dense points corresponding to the face image.

[0015] As one possible implementation, generating a dense point heatmap corresponding to the face image based on the dense point locations includes:

[0016] Create a single-channel image, wherein the dimensions of the single-channel image are the same as the dimensions of the base network features;

[0017] The dense points are marked on the single-channel image, and the pixel values ​​at the corresponding positions are filled with preset values ​​to obtain the basic dense point heat map corresponding to the face image.

[0018] A preset fuzzing algorithm is used to blur the basic dense point heatmap to obtain the dense point heatmap corresponding to the face image.

[0019] As one possible implementation, the sparse point prediction model includes at least one sparse point feature prediction layer, a fully connected layer, and a sparse point prediction layer. The step of inputting the dense point heatmap into the preset sparse point prediction model to obtain the key points corresponding to the face image includes:

[0020] The dense point heatmap and the basic network features are passed through at least one sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image;

[0021] The first sparse point feature is input into the fully connected layer to obtain the second sparse point feature corresponding to the face image;

[0022] The second sparse point feature is input into the sparse point prediction layer to obtain the key points corresponding to the face image.

[0023] As a possible implementation, the sparse point feature prediction layer includes a channel connection layer and at least one convolutional layer. When the sparse point prediction model includes two sparse point feature prediction layers, the step of passing the dense point heatmap and the basic network features through at least one of the sparse point feature prediction layers to obtain the first sparse point feature corresponding to the face image includes:

[0024] The dense point heatmap and the basic network features are input into the channel connection layer of the first sparse point feature prediction layer for channel connection to obtain the first connection feature corresponding to the dense point heatmap;

[0025] The first connection feature is sequentially processed by convolutional layer at least one of the convolutional layers of the first sparse point feature prediction layer to obtain the third sparse point feature corresponding to the face image.

[0026] The third sparse point feature and the basic network feature are input into the channel connection layer of the second sparse point feature prediction layer to perform channel connection, thereby obtaining the second connection feature corresponding to the dense point heatmap.

[0027] The second connection feature is sequentially processed by convolutional layer at least one of the convolutional layers of the second sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image.

[0028] As one possible implementation, the dense point prediction model is trained in the following way:

[0029] Obtain a first face image training sample, and the first face image training sample is labeled with dense sample points corresponding to the face image;

[0030] Input the first face image training sample into the first preset base model to obtain the training dense points corresponding to the first face image training sample output by the first preset base model.

[0031] Using a preset loss function, the training dense points and the sample dense points are calculated to obtain the first loss value for this training.

[0032] If the first loss value meets the preset conditions, training is stopped, and the dense point prediction model is obtained.

[0033] As one possible implementation, the first preset base model includes a training base network layer, a training fully connected layer, and a training dense point prediction layer. The step of inputting the first face image training sample into the first preset base model to obtain the training dense points corresponding to the first face image training sample output by the first preset base model includes:

[0034] The first face image training sample is input into the training base network layer to obtain the training base network features output by the training base network layer.

[0035] The training base network features are input into the training fully connected layer to obtain the training dense point features output by the training fully connected layer.

[0036] The training dense point features are input into the training dense point prediction layer to obtain the training dense points corresponding to the training samples of the first face image.

[0037] As one possible implementation, the sparse point prediction model is trained in the following way:

[0038] Obtain a second face image training sample, and the second face image training sample is labeled with sparse points corresponding to the face image;

[0039] The training sample of the second face image is input into the dense point prediction model to obtain the training dense point corresponding to the training sample of the second face image.

[0040] Based on the training dense points, a heatmap of the training dense points corresponding to the training samples of the second face image is generated;

[0041] The training dense point heatmap is input into the second preset base model to obtain the training sparse points corresponding to the training samples of the second face image output by the second preset base model.

[0042] Using a preset loss function, the training sparse points and the sample sparse points are calculated to obtain the second loss value for this training.

[0043] If the second loss value meets the preset conditions, training is stopped, and the sparse point prediction model is obtained.

[0044] As one possible implementation, generating a heatmap of training dense points corresponding to the training samples of the second face image based on the training dense points includes:

[0045] Create a single-channel image, wherein the length and width of the single-channel image are the same as the length and width of the training base network features, wherein the training base network features are obtained by processing the training samples of the second face image by the base network layer in the dense point prediction model;

[0046] The training dense points are marked on the single-channel image, and the pixel values ​​at the corresponding positions are filled with preset values ​​to obtain the basic training dense point heatmap corresponding to the training sample of the second face image.

[0047] A preset fuzzing algorithm is used to fuzz the base training dense point heatmap to obtain the training dense point heatmap corresponding to the second face image training sample.

[0048] As one possible implementation, the second preset base model includes at least one training sparse point feature prediction layer, a training fully connected layer, and a training sparse point prediction layer. The step of inputting the training dense point heatmap into the second preset base model to obtain the training sparse points corresponding to the training samples of the second face image output by the second preset base model includes:

[0049] The training dense point heatmap and training basic network features are passed through at least one training sparse point feature prediction layer to obtain the first training sparse point feature corresponding to the training sample of the second face image.

[0050] The first training sparse point feature is input into the training fully connected layer to obtain the second training sparse point feature corresponding to the training sample of the second face image.

[0051] The second training sparse point feature is input into the training sparse point prediction layer to obtain the training sparse points corresponding to the training samples of the second face image.

[0052] As a possible implementation, the training sparse point feature prediction layer includes a training channel connection layer and at least one training convolutional layer. When the second preset base model includes two training sparse point feature prediction layers, the step of passing the training dense point heatmap and the training base network features through at least one training sparse point feature prediction layer to obtain the first training sparse point feature corresponding to the second face image training sample includes:

[0053] The training dense point heatmap and the training basic network features are input into the training channel connection layer of the first training sparse point feature prediction layer to perform channel connection, thereby obtaining the first training connection feature corresponding to the dense point heatmap.

[0054] The first training connection feature is sequentially processed by at least one of the training convolutional layers of the first training sparse point feature prediction layer to obtain the third training sparse point feature corresponding to the training sample of the second face image.

[0055] The third training sparse point feature and the basic network feature are input into the training channel connection layer of the second training sparse point feature prediction layer to perform channel connection, thereby obtaining the second training connection feature corresponding to the dense point heatmap.

[0056] The second training connection features are sequentially processed by passing them through at least one of the training convolutional layers of the second training sparse point feature prediction layer to obtain the first training sparse point features corresponding to the training samples of the second face image.

[0057] As a possible implementation, the dense point prediction model and the sparse point prediction model are trained in the following manner:

[0058] Obtain training samples of third-person face images, wherein the training samples of third-person face images are labeled with dense and sparse points corresponding to the face images;

[0059] The training sample of the third face image is input into the third preset base model to obtain the training dense points corresponding to the training sample of the third face image output by the third preset base model.

[0060] Based on the training dense points, a heatmap of the training dense points corresponding to the training samples of the third face image is generated;

[0061] The training dense point heatmap is input into the fourth preset base model to obtain the training sparse points corresponding to the training samples of the third face image output by the fourth preset base model.

[0062] Using a preset loss function, the third loss value for this training is calculated for the dense training points and the dense sample points, and the fourth loss value for this training is calculated for the sparse training points and the sparse sample points.

[0063] If both the third and fourth loss values ​​meet the preset conditions, training is stopped, and the dense point prediction model and the sparse point prediction model are obtained.

[0064] Secondly, embodiments of the present invention provide a facial key point detection device, the device comprising:

[0065] The acquisition module is used to acquire the face image to be detected;

[0066] The first input module is used to input the face image into a preset dense point prediction model to obtain the dense points corresponding to the face image;

[0067] The generation module is used to generate a dense point heatmap corresponding to the face image based on the dense point locations.

[0068] The second input module is used to input the dense point heatmap into a preset sparse point prediction model to obtain the key points corresponding to the face image.

[0069] As one possible implementation, the dense point prediction model includes a base network layer, a fully connected layer, and a dense point prediction layer. The first input module is specifically used for:

[0070] The face image is input into the base network layer to obtain the base network features output by the base network layer;

[0071] The basic network features are input into the fully connected layer to obtain the dense point features output by the fully connected layer;

[0072] The dense point features are input into the dense point prediction layer to obtain the dense points corresponding to the face image.

[0073] As one possible implementation, the generation module is specifically used for:

[0074] Create a single-channel image, wherein the dimensions of the single-channel image are the same as the dimensions of the base network features;

[0075] The dense points are marked on the single-channel image, and the pixel values ​​at the corresponding positions are filled with preset values ​​to obtain the basic dense point heat map corresponding to the face image.

[0076] A preset fuzzing algorithm is used to blur the basic dense point heatmap to obtain the dense point heatmap corresponding to the face image.

[0077] As one possible implementation, the sparse point prediction model includes at least one sparse point feature prediction layer, a fully connected layer, and a sparse point prediction layer. The second input module includes:

[0078] The first determining submodule is used to pass the dense point heatmap and the basic network features through at least one sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image.

[0079] The first input submodule is used to input the first sparse point feature into the fully connected layer to obtain the second sparse point feature corresponding to the face image;

[0080] The second input submodule is used to input the second sparse point feature into the sparse point prediction layer to obtain the key points corresponding to the face image.

[0081] As one possible implementation, the sparse point feature prediction layer includes a channel connection layer and at least one convolutional layer. In the case where the sparse point prediction model includes two sparse point feature prediction layers, the first determining submodule is specifically used for:

[0082] The dense point heatmap and the basic network features are input into the channel connection layer of the first sparse point feature prediction layer for channel connection to obtain the first connection feature corresponding to the dense point heatmap;

[0083] The first connection feature is sequentially processed by convolutional layer at least one of the convolutional layers of the first sparse point feature prediction layer to obtain the third sparse point feature corresponding to the face image.

[0084] The third sparse point feature and the basic network feature are input into the channel connection layer of the second sparse point feature prediction layer to perform channel connection, thereby obtaining the second connection feature corresponding to the dense point heatmap.

[0085] The second connection feature is sequentially processed by convolutional layer at least one of the convolutional layers of the second sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image.

[0086] As one possible implementation, the device further includes:

[0087] The first sample acquisition module is used to acquire a first face image training sample, wherein the first face image training sample is labeled with dense sample points corresponding to the face image;

[0088] The first sample input module is used to input the first face image training sample into the first preset base model to obtain the training dense points corresponding to the first face image training sample output by the first preset base model.

[0089] The first loss value calculation module is used to calculate the first loss value of the training dense points and the sample dense points using a preset loss function.

[0090] The first model determination module is used to stop training and obtain the dense point prediction model when the first loss value meets the preset conditions.

[0091] As one possible implementation, the first preset base model includes training a base network layer, training a fully connected layer, and training a dense point prediction layer. The first sample input module is specifically used for:

[0092] The first face image training sample is input into the training base network layer to obtain the training base network features output by the training base network layer.

[0093] The training base network features are input into the training fully connected layer to obtain the training dense point features output by the training fully connected layer.

[0094] The training dense point features are input into the training dense point prediction layer to obtain the training dense points corresponding to the training samples of the first face image.

[0095] As one possible implementation, the device further includes:

[0096] The second sample acquisition module is used to acquire a second face image training sample, wherein the second face image training sample is labeled with sparse points corresponding to the face image.

[0097] The second sample input module is used to input the second face image training sample into the dense point prediction model to obtain the training dense point corresponding to the second face image training sample.

[0098] The heatmap generation module is used to generate a heatmap of the training dense points corresponding to the training samples of the second face image based on the training dense points.

[0099] The third sample input module is used to input the training dense point heatmap into the second preset base model to obtain the training sparse points corresponding to the training samples of the second face image output by the second preset base model.

[0100] The second loss value calculation module is used to calculate the second loss value of the training sparse points and the sample sparse points using a preset loss function.

[0101] The second model determination module is used to stop training when the second loss value meets the preset conditions, and obtain the sparse point prediction model.

[0102] As one possible implementation, the heatmap generation module is specifically used for:

[0103] Create a single-channel image, wherein the length and width of the single-channel image are the same as the length and width of the training base network features, wherein the training base network features are obtained by processing the training samples of the second face image by the base network layer in the dense point prediction model;

[0104] The training dense points are marked on the single-channel image, and the pixel values ​​at the corresponding positions are filled with preset values ​​to obtain the basic training dense point heatmap corresponding to the training sample of the second face image.

[0105] A preset fuzzing algorithm is used to fuzz the base training dense point heatmap to obtain the training dense point heatmap corresponding to the second face image training sample.

[0106] As one possible implementation, the second preset base model includes at least one training sparse point feature prediction layer, a training fully connected layer, and a training sparse point prediction layer; the second sample input module includes:

[0107] The determination submodule is used to pass the training dense point heatmap and training basic network features through at least one training sparse point feature prediction layer to obtain the first training sparse point feature corresponding to the training sample of the second face image.

[0108] The first sample input submodule is used to input the first training sparse point feature into the training fully connected layer to obtain the second training sparse point feature corresponding to the second face image training sample.

[0109] The second sample input submodule is used to input the second training sparse point feature into the training sparse point prediction layer to obtain the training sparse points corresponding to the training samples of the second face image.

[0110] As one possible implementation, the training sparse point feature prediction layer includes a training channel connection layer and at least one training convolutional layer. In the case where the second preset base model includes two training sparse point feature prediction layers, the determining submodule is specifically used for:

[0111] The training dense point heatmap and the training basic network features are input into the training channel connection layer of the first training sparse point feature prediction layer to perform channel connection, thereby obtaining the first training connection feature corresponding to the dense point heatmap.

[0112] The first training connection feature is sequentially processed by at least one of the training convolutional layers of the first training sparse point feature prediction layer to obtain the third training sparse point feature corresponding to the training sample of the second face image.

[0113] The third training sparse point feature and the basic network feature are input into the training channel connection layer of the second training sparse point feature prediction layer to perform channel connection, thereby obtaining the second training connection feature corresponding to the dense point heatmap.

[0114] The second training connection features are sequentially processed by passing them through at least one of the training convolutional layers of the second training sparse point feature prediction layer to obtain the first training sparse point features corresponding to the training samples of the second face image.

[0115] As one possible implementation, the device further includes:

[0116] The third sample acquisition module is used to acquire third face image training samples, wherein the third face image training samples are labeled with dense and sparse points corresponding to the face images.

[0117] The fourth sample input module is used to input the third face image training sample into the third preset base model to obtain the training dense points corresponding to the third face image training sample output by the third preset base model.

[0118] The point heatmap generation module is used to generate a training dense point heatmap corresponding to the training sample of the third face image based on the training dense points.

[0119] The fifth sample input module is used to input the training dense point heatmap into the fourth preset base model to obtain the training sparse points corresponding to the training samples of the third face image output by the fourth preset base model.

[0120] The third loss value calculation module is used to calculate the third loss value of the training dense points and the sample dense points using a preset loss function, and to calculate the fourth loss value of the training sparse points and the sample sparse points.

[0121] The training stop module is used to stop training when both the third loss value and the fourth loss value meet preset conditions, thereby obtaining the dense point prediction model and the sparse point prediction model.

[0122] Thirdly, embodiments of the present invention provide an electronic device, including: a processor and a memory, wherein the processor is configured to execute a facial key point detection program stored in the memory to implement the facial key point detection method described in any one of the first aspects.

[0123] Fourthly, embodiments of the present invention provide a storage medium storing one or more programs, which can be executed by one or more processors to implement the facial key point detection method described in any one aspect.

[0124] The technical solution provided by this invention acquires a face image to be detected, inputs the face image into a preset dense point prediction model to obtain the dense points corresponding to the face image, generates a dense point heatmap corresponding to the face image based on the dense points, and inputs the dense heatmap into a preset sparse point prediction model to obtain the key points corresponding to the face image. This technical solution predicts the dense points of the face image, generates a dense point heatmap based on the dense points, and then generates the key points corresponding to the face image based on the dense point heatmap. Since the dense point heatmap already contains relatively accurate positional information of the face edge contour, the detection of key points in the face image based on the dense point heatmap is more accurate, thereby improving the accuracy of face key point detection and enhancing the user experience. Attached Figure Description

[0125] Figure 1 A flowchart illustrating an embodiment of a method for detecting facial key points provided by this invention;

[0126] Figure 2A A schematic diagram of a basic dense point heatmap corresponding to a face image provided in an embodiment of the present invention;

[0127] Figure 2B This is a schematic diagram of a dense point heatmap corresponding to a face image provided in an embodiment of the present invention;

[0128] Figure 3A This is a schematic diagram of the structure of a first sparse point feature prediction layer provided in an embodiment of the present invention;

[0129] Figure 3B This is a schematic diagram of the structure of a second sparse point feature prediction layer provided in an embodiment of the present invention;

[0130] Figure 4 A flowchart illustrating an embodiment of a training method for a dense point prediction model provided by the present invention;

[0131] Figure 5 A flowchart illustrating an embodiment of a training method for a sparse point prediction model provided by the present invention;

[0132] Figure 6 A flowchart illustrating an embodiment of a training method for a dense point prediction model and a sparse point prediction model provided by an embodiment of the present invention;

[0133] Figure 7 This is a schematic diagram of a facial key point detection model structure provided in an embodiment of the present invention;

[0134] Figure 8 This is a block diagram illustrating an embodiment of a facial key point detection device provided by the present invention.

[0135] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0136] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0137] The method for detecting facial key points provided by the present invention will be further explained and described below with reference to the accompanying drawings and specific embodiments. The embodiments do not constitute a limitation on the embodiments of the present invention.

[0138] See Figure 1 This is a flowchart illustrating an embodiment of a method for detecting facial key points provided by this invention. Figure 1 As shown, the process may include the following steps:

[0139] Step 101: Obtain the face image to be detected.

[0140] The aforementioned face images refer to the face images collected for keypoint detection.

[0141] In one embodiment, the executing entity of this invention may be a terminal, which can be a hardware device or software that supports network connectivity to provide various network services. When the terminal is hardware, it can be various electronic devices that support a display screen, including but not limited to smartphones, tablets, laptops, desktop computers, etc. When the terminal is software, it can be installed in the electronic devices listed above.

[0142] In one embodiment, the executing entity of this invention can acquire the aforementioned face image through a preset image acquisition device.

[0143] In another embodiment, the executing entity of this invention may obtain the aforementioned face image input by the user.

[0144] Step 102: Input the above face image into the preset dense point prediction model to obtain the dense point corresponding to the face image.

[0145] The dense point prediction model described above is used to predict the dense points of the face in the above face image, such as the 826 points of the face.

[0146] In this embodiment of the invention, in order to more accurately predict sparse keypoints in a face image, dense keypoints in the face image can be predicted first, and then sparse keypoints can be predicted based on the dense keypoints. Specifically, the face image can be input into a preset dense keypoint prediction model to obtain the dense keypoints corresponding to the face image output by the dense keypoint prediction model.

[0147] Optionally, the above dense location prediction model may include a base network layer, a fully connected layer, and a dense location prediction layer.

[0148] Based on this, the execution subject of this embodiment of the invention can first input the above-mentioned face image into the basic network layer to obtain the basic network features output by the basic network layer.

[0149] Then, the basic network features can be input into the fully connected layer to obtain the dense point features output by the fully connected layer.

[0150] Finally, the dense point features can be input into the dense point prediction layer to obtain the dense points corresponding to the face image.

[0151] The specific methods for training the aforementioned dense point prediction model will be explained below. Figure 4 or Figure 6 The process shown will be explained in detail here.

[0152] Step 103: Generate a heatmap of dense points corresponding to the above-mentioned face image based on the above-mentioned dense points.

[0153] A heatmap refers to a page area that visitors frequently view, highlighted in a special way. Similarly, the aforementioned dense point heatmap refers to a dense point in a face image that is highlighted in a special way.

[0154] In this embodiment of the invention, a dense point heatmap corresponding to a face image can be generated based on the aforementioned dense point locations.

[0155] Specifically, a single-channel image can be created first, in which the single-channel image is completely black and its length and width are consistent with the length and width of the basic network features of the aforementioned face image.

[0156] Then, the aforementioned dense points can be labeled on the single-channel image, and the pixel values ​​at the corresponding positions can be filled with preset values ​​to obtain a basic dense point heatmap corresponding to the face image. For example, see [link to relevant documentation]. Figure 2A This is a schematic diagram of a basic dense point heatmap corresponding to a face image provided in an embodiment of the present invention. Optionally, the above preset value can be 1.

[0157] Finally, a preset blurring algorithm can be used to blur the aforementioned basic dense point heatmap to obtain the dense point heatmap corresponding to the face image. Specifically, a Gaussian blur algorithm can be applied to each dense point in the aforementioned basic dense point heatmap, with a blur radius of r. This process expands each dense point into a circular region of radius r. The pixel value at the center of the circle is the brightest, decreasing outwards, with values ​​outside the radius region being 0. This yields the aforementioned dense point heatmap corresponding to the face image. For example, see [link to example]. Figure 2B This is a schematic diagram of a dense point heatmap corresponding to a face image provided in an embodiment of the present invention.

[0158] Step 104: Input the above dense point heatmap into the preset sparse point prediction model to obtain the key points corresponding to the above face image.

[0159] The key points mentioned above are sparse key points corresponding to the face image.

[0160] The sparse point prediction model described above is used to predict sparse keypoints corresponding to face images.

[0161] In this embodiment of the invention, the key points corresponding to the above-mentioned face image can be obtained by inputting the above-mentioned dense point heat map into a preset sparse point prediction model.

[0162] Optionally, the above sparse point prediction model may include at least one sparse point feature prediction layer, a fully connected layer, and a sparse point prediction layer.

[0163] In this embodiment of the invention, the dense point heatmap already contains relatively accurate positional information of the face edge contour. However, this dense point heatmap is expanded from discrete points and has no information features from the input image. If the point is slightly off or inaccurate, the subsequent feature extraction process will not have any dependent information for position correction. Therefore, in this embodiment of the invention, based on the input dense point heatmap, the basic network features of the face image are added as information input to further correct the position of the points and ensure the accuracy of sparse point prediction.

[0164] Based on this, the execution subject of this embodiment of the invention can process the dense point heatmap corresponding to the generated face image and the basic network features of the face image through at least one sparse point feature prediction layer to obtain the sparse point features corresponding to the face image (hereinafter referred to as the first sparse point feature for ease of description).

[0165] Then, the first sparse point features can be input into the fully connected layer to obtain the sparse point features corresponding to the face image (hereinafter referred to as the second sparse point features for ease of description).

[0166] Finally, the second sparse point feature can be input into the sparse point prediction layer to obtain the key points corresponding to the face image.

[0167] Furthermore, the aforementioned sparse point feature prediction layer may include a channel connection layer and at least one convolutional layer.

[0168] In one embodiment, the sparse point prediction model may include a sparse point feature prediction layer. Based on this, the execution entity of this embodiment can input the dense point heatmap and basic network features into the channel connection layer of the sparse point feature prediction layer for channel connection, obtaining the connection features corresponding to the dense point heatmap. These connection features are then sequentially processed through at least one convolutional layer to obtain the first sparse point feature corresponding to the face image.

[0169] In another embodiment, the sparse point prediction model described above may include two sparse point feature prediction layers. Based on this, the execution entity of this embodiment can input the dense point heatmap and the basic network features into the channel connection layer of the first sparse point feature prediction layer (hereinafter referred to as the first sparse point feature prediction layer for ease of description) for channel connection to obtain the connection features corresponding to the dense point heatmap (hereinafter referred to as the first connection features for ease of description), and then pass the first connection features sequentially through at least one convolutional layer of the first sparse point feature prediction layer for convolution processing to obtain the sparse point features corresponding to the face image (hereinafter referred to as the third sparse point features for ease of description).

[0170] Subsequently, the execution subject of this embodiment of the invention can input the above-mentioned third sparse point feature and the above-mentioned basic network features of the face image into the channel connection layer of the second sparse point feature prediction layer (hereinafter referred to as the second sparse point feature prediction layer for ease of description) to perform channel connection, and obtain the connection features corresponding to the dense point heatmap (hereinafter referred to as the second connection features for ease of description).

[0171] Then, the second connection feature is sequentially passed through at least one convolutional layer of the second sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image.

[0172] For example, see Figure 3A This is a schematic diagram of the structure of a first sparse point feature prediction layer provided in an embodiment of the present invention. Figure 3A As shown, the first sparse point feature prediction layer can be a block1 (module 1), which may include an input module for the basic network features f and the dense point heatmap, a channel connection module for image features, and three convolutional layers: convolution 1, convolution 2, and convolution 3.

[0173] like Figure 3BThe diagram shown is a schematic representation of the structure of a second sparse point feature prediction layer provided in an embodiment of the present invention. Figure 3B As shown, the second sparse point feature prediction layer can be used as block2 (module 2), which may include an input module for the basic network features f and the output features of block1, a channel connection module for image features, and three convolutional layers: convolution 1, convolution 2, and convolution 3.

[0174] In block 1, the inputs are the basic network features f and the dense point heatmap M. First, the two are connected by feature channels (⊕ in the figure indicates that the image feature maps are connected in the channel dimension. For example, the shape of I1 is (B, C1, H, W), the shape of I2 is (B, C2, H, W), and the output after channel connection is O, with the shape (B, C1+C2, H, W)). Then, the connected features are used as new inputs and passed through convolution 1, convolution 2 and convolution 3 in sequence.

[0175] The input to block2 is the base network feature f and the output feature of block1. Similarly, the feature channels of the two are first concatenated, and then the concatenated feature is used as the new input, sequentially passing through convolution 1, convolution 2, and convolution 3. The structures of convolutions 1, 2, and 3 are completely identical, all being 3x3 convolutions, and the output feature dimension can be D = 256.

[0176] It should be noted that the above sparse point prediction model may also include two or more sparse point feature prediction layers, and the embodiments of the present invention do not impose any restrictions on this.

[0177] To facilitate understanding of the facial landmark detection method provided in the embodiments of the present invention, the following examples are given:

[0178] See Figure 7 This is a schematic diagram of a facial key point detection model structure provided in an embodiment of the present invention. Figure 7 As shown, the facial landmark detection model may include: a base network layer, a fully connected layer, a dense landmark prediction layer, a dense landmark heatmap generation module, block1 (sparse landmark feature prediction layer 1), block2 (sparse landmark feature prediction layer 2), a fully connected layer, and a 106-point prediction layer (i.e., a sparse landmark prediction layer). Specifically, this facial landmark detection model detects facial landmarks in the following ways:

[0179] First, the system receives the input face image. Then, the base network layer extracts basic features, and the output base network feature f has dimensions (B, C, H, W), where B represents the number of images (i.e., the training batch size), C represents the number of feature channels, and H and W represent the feature height and width, respectively. Next, the model first passes through a fully connected layer to predict dense points, and then generates a dense point heatmap from the predicted dense points. This is then passed through block1 and block2 sequentially. After block2, another fully connected layer is connected to predict 106 points.

[0180] Here, dense points N2 = 826 * 2, meaning 826 dense points are predicted, each containing x and y coordinate parameters; 106 points N1 = 106 * 2, meaning 106 points are predicted with x and y coordinates. The feature dimension D is typically 256. The base network used here can be any existing model architecture, such as MobileNetV3.

[0181] The technical solution provided by this invention acquires a face image to be detected, inputs the face image into a preset dense point prediction model to obtain the dense points corresponding to the face image, generates a dense point heatmap corresponding to the face image based on the dense points, and inputs the dense heatmap into a preset sparse point prediction model to obtain the key points corresponding to the face image. This technical solution predicts the dense points of the face image, generates a dense point heatmap based on the dense points, and then generates the key points corresponding to the face image based on the dense point heatmap. Since the dense point heatmap already contains relatively accurate positional information of the face edge contour, the detection of key points in the face image based on the dense point heatmap is more accurate, thereby improving the accuracy of face key point detection and enhancing the user experience.

[0182] See Figure 4 The above is a flowchart of an embodiment of a training method for a dense point prediction model provided by the present invention. Figure 4 The process shown is in Figure 1 Based on the illustrated process, a detailed description is provided of how the aforementioned dense point prediction model was trained. For example... Figure 4 As shown, the process may include the following steps:

[0183] Step 401: Obtain the first face image training sample. The first face image training sample is labeled with dense sample points corresponding to the face image.

[0184] The first face image training sample mentioned above is a face image with densely labeled points (hereinafter referred to as densely labeled points for ease of description).

[0185] In one embodiment, the executing entity of this invention may acquire the aforementioned first face image training sample through a preset image acquisition device.

[0186] In another embodiment, the executing entity of this invention may obtain the aforementioned first face image training sample input by the user.

[0187] Step 402: Input the first face image training sample into the first preset base model to obtain the training dense points corresponding to the first face image training sample output by the first preset base model.

[0188] The aforementioned first preset basic model is the basic model for predicting the dense points of the training samples of the first face image.

[0189] Accordingly, the aforementioned dense training points are the dense points of the first face image training samples predicted by the first preset basic model.

[0190] In this embodiment of the invention, a first face image training sample can be input into the first preset basic model to obtain the training dense points corresponding to the first face image training sample output by the first preset basic model.

[0191] Specifically, the aforementioned first preset basic model may include training a basic network layer, training a fully connected layer, and training a dense point prediction layer.

[0192] Based on this, the execution subject of this embodiment of the invention can input the first face image training sample into the training base network layer to obtain the training base network features output by the training base network layer.

[0193] Then, the above-mentioned training base network features are input into the above-mentioned training fully connected layer to obtain the training dense point features output by the above-mentioned training fully connected layer, and the training dense point features are input into the above-mentioned training dense point prediction layer to obtain the training dense points corresponding to the above-mentioned first face image training sample.

[0194] Step 403: Using the preset loss function, calculate the first loss value for the training dense points and the sample dense points to obtain the first loss value for this training.

[0195] Step 404: If the first loss value meets the preset conditions, stop training and obtain the dense point prediction model.

[0196] The following provides a unified explanation of steps 403 and 404:

[0197] The first loss value mentioned above is used to characterize the loss between the dense training points and the dense sample points mentioned above.

[0198] In this embodiment of the invention, after training the first preset basic model to obtain the training dense points of the first face image training samples, the preset loss function can be used to calculate the above-mentioned training dense points and sample dense points to obtain the first loss value of this training.

[0199] Optionally, if the first loss value meets a preset condition, training is stopped to obtain the dense point prediction model. The preset condition may be that the first loss value is less than or equal to a preset loss value threshold, which may be 0.2 or 0.1. This embodiment of the invention does not limit this.

[0200] Conversely, if the first loss value does not meet the preset conditions, the first face image training sample can be input into the first preset base model to continue training the first preset base model.

[0201] The technical solution provided by this invention involves acquiring a first face image training sample, which is labeled with dense point locations corresponding to the face image. This first face image training sample is then input into a first preset base model to obtain the training dense point locations corresponding to the first face image training sample output by the first preset base model. A preset loss function is used to calculate the training dense point locations and the sample dense point locations to obtain a first loss value for this training. Training is stopped when the first loss value meets a preset condition, resulting in a dense point location prediction model. This technical solution improves the accuracy of dense point location prediction by training the first preset base model with first face image training samples labeled with dense point locations, thereby increasing the accuracy of face keypoint detection and enhancing the user experience.

[0202] See Figure 5 The above is a flowchart of an embodiment of a training method for a sparse point prediction model provided by an embodiment of the present invention. Figure 5 The process shown is in Figure 1 Based on the illustrated process, a detailed description is provided of how the aforementioned sparse point prediction model was trained. For example... Figure 5 As shown, the process may include the following steps:

[0203] Step 501: Obtain the second face image training sample. The second face image training sample is labeled with sparse points corresponding to the face image.

[0204] The second face image training sample mentioned above is a face image with sparsely labeled points (hereinafter referred to as sparse points for ease of description).

[0205] In one embodiment, the executing entity of this invention can acquire the aforementioned second face image training samples through a preset image acquisition device.

[0206] In another embodiment, the executing entity of this invention may obtain the aforementioned second face image training sample input by the user.

[0207] Step 502: Input the training sample of the second face image into the dense point prediction model to obtain the training dense points corresponding to the face image.

[0208] Step 503: Generate a heatmap of training dense points corresponding to the training samples of the second face image based on the training dense points.

[0209] The following provides a unified explanation of steps 502 and 503:

[0210] The above dense point prediction model can be the above Figure 4 The dense point prediction model is trained using the process shown.

[0211] In this embodiment of the invention, by inputting the above-mentioned second face image training sample into the above-mentioned dense point prediction model, the training dense points of the face image in the second face image training sample can be obtained.

[0212] Then, the dense point locations corresponding to the training samples of the second face image can be used to generate a dense point heatmap (hereinafter referred to as the training dense point heatmap for ease of description).

[0213] Specifically, a single-channel image can be created first. This single-channel image is completely black, and its dimensions are consistent with the dimensions of the basic network features (hereinafter referred to as the training basic network features) of the training samples of the second face image. The training basic network features of the training samples of the second face image can be obtained by the dense point prediction model during the process of processing the second face image to obtain the training dense points.

[0214] Then, the aforementioned dense training points can be labeled on the single-channel image, and the pixel values ​​at the corresponding positions can be filled with preset values ​​to obtain the basic training dense point heatmap corresponding to the training sample of the second face image. Optionally, the aforementioned preset value can be 1.

[0215] Finally, a preset blurring algorithm can be used to blur the aforementioned basic training dense point heatmap to obtain the training dense point heatmap corresponding to the second face image training sample. Specifically, for each dense point in the aforementioned basic training dense point heatmap, a Gaussian blur algorithm can be applied, setting the blur radius to r. This process expands each dense point into a circular region of radius r. The pixel value at the center of the circle is the brightest, decreasing sequentially outwards, with values ​​outside the radius region being 0. This yields the aforementioned training dense point heatmap corresponding to the second face image training sample.

[0216] Step 504: Input the above-mentioned dense training point heatmap into the second preset base model to obtain the sparse training points corresponding to the second face image training samples output by the second preset base model.

[0217] The aforementioned second preset basic model is a basic model used to predict the sparse points of the training samples of the second face image.

[0218] Accordingly, the above-mentioned training sparse points are the sparse points obtained by processing the above-mentioned dense point heatmap by the above-mentioned second preset base model (hereinafter referred to as training sparse points for ease of description).

[0219] In this embodiment of the invention, the above-mentioned dense training point heatmap can be input into the above-mentioned second preset base model to obtain the training sparse points corresponding to the second face image training sample output by the above-mentioned second preset base model.

[0220] Specifically, the aforementioned second preset basic model may include at least one training sparse point feature prediction layer, a training fully connected layer, and a training sparse point prediction layer.

[0221] Based on this, the execution subject of this embodiment of the invention can pass the above-mentioned training dense point heatmap and the training basic network features of the second face image training sample through at least one training sparse point feature prediction layer to obtain the sparse point features corresponding to the second face image training sample (hereinafter referred to as the first training sparse point features for ease of description).

[0222] Then, the first training sparse point features can be input into the training fully connected layer to obtain the sparse point features corresponding to the training sample of the second face image (hereinafter referred to as the second training sparse point features for ease of description), and the second sparse point features can be input into the training sparse point prediction layer to obtain the training sparse points corresponding to the training sample of the second face image.

[0223] Furthermore, the aforementioned training sparse point feature prediction layer may include a training channel connection layer and at least one training convolutional layer.

[0224] In one embodiment, the aforementioned second preset base model may include a training sparse point feature prediction layer. Based on this, the execution subject of this embodiment of the invention may input the aforementioned training dense point heatmap and training base network features into the training channel connection layer of the training sparse point feature prediction layer for channel connection to obtain the connection features corresponding to the training dense point heatmap (hereinafter referred to as training connection features for ease of description), and then input the training connection features sequentially through at least one training convolutional layer for convolution processing to obtain the first training sparse point features corresponding to the second face image training sample.

[0225] In another embodiment, the aforementioned second preset base model may include two training sparse point feature prediction layers. Based on this, the execution subject of this embodiment can input the training dense point heatmap and the training base network features into the training channel connection layer of the first training sparse point feature prediction layer (hereinafter referred to as the first training sparse point feature prediction layer for ease of description) to perform channel connection, thereby obtaining the connection features corresponding to the training dense point heatmap (hereinafter referred to as the first training connection features for ease of description), and then sequentially pass the first training connection features through at least one training convolutional layer of the aforementioned first training sparse point feature prediction layer for convolution processing to obtain the sparse point features corresponding to the second face image training sample (hereinafter referred to as the third training sparse point features for ease of description).

[0226] Subsequently, the execution subject of this embodiment of the invention can input the above-mentioned third training sparse point feature and the training base network feature of the above-mentioned second face image training sample into the training channel connection layer of the second training sparse point feature prediction layer (hereinafter referred to as the second training sparse point feature prediction layer for ease of description) to perform channel connection, and obtain the connection feature corresponding to the training dense point heatmap (hereinafter referred to as the second training connection feature for ease of description).

[0227] Then, the second training connection features are sequentially processed through at least one training convolutional layer of the second training sparse point feature prediction layer to obtain the first training sparse point features corresponding to the training samples of the second face image.

[0228] It should be noted that the aforementioned second preset basic model may also include two or more training sparse point feature prediction layers, and the embodiments of the present invention do not impose any restrictions on this.

[0229] Step 505: Using the preset loss function, calculate the training sparse points and the sample sparse points to obtain the second loss value for this training.

[0230] Step 506: If the second loss value meets the preset conditions, stop training and obtain the sparse point prediction model.

[0231] The following provides a unified explanation of steps 505 and 506:

[0232] The second loss value is used to characterize the loss between the training sparse points and the sample sparse points.

[0233] In this embodiment of the invention, after training the second preset base model to obtain the training sparse points of the second face image training samples, the preset loss function can be used to calculate the above-mentioned training sparse points and sample sparse points to obtain the second loss value of this training.

[0234] Optionally, if the second loss value meets a preset condition, training is stopped to obtain the sparse point prediction model. The preset condition can be less than or equal to a preset loss value threshold, which can be 0.2 or 0.1. This embodiment of the invention does not limit this.

[0235] Conversely, if the second loss value does not meet the preset conditions, the second face image training sample can be input into the dense point prediction model to continue training the second preset base model.

[0236] The technical solution provided by this invention involves acquiring a second face image training sample, which is labeled with sparse points corresponding to the face image. This second face image training sample is then input into a dense point prediction model to obtain training dense points corresponding to the face image. Based on these dense points, a training dense point heatmap is generated. This heatmap is then input into a second preset base model to obtain training sparse points corresponding to the second face image training sample, output by the second preset base model. A preset loss function is used to calculate the training sparse points and sample sparse points to obtain a second loss value for this training. Training stops when the second loss value meets a preset condition, resulting in a sparse point prediction model. This technical solution improves the accuracy of the sparse point prediction model by using second face image training samples labeled with sparse points to train a second preset base model, thereby increasing the accuracy of face keypoint detection and enhancing the user experience.

[0237] See Figure 6 The above is a flowchart illustrating an embodiment of a training method for a dense point prediction model and a sparse point prediction model provided by the present invention. Figure 6 The process shown is in Figure 1 Based on the illustrated process, a detailed description is provided of how the aforementioned dense point prediction model and sparse point prediction model were trained. For example... Figure 6 As shown, the process may include the following steps:

[0238] Step 601: Obtain training samples of third-person face images. The training samples of the third-person face images are labeled with dense and sparse points corresponding to the face images.

[0239] The aforementioned third face image training samples are face images labeled with dense points (hereinafter referred to as dense points for ease of description) and sparse points (hereinafter referred to as sparse points for ease of description).

[0240] In one embodiment, the executing entity of this invention may acquire the aforementioned third face image training samples through a preset image acquisition device.

[0241] In another embodiment, the executing entity of this invention may obtain the aforementioned third face image training sample input by the user.

[0242] Step 602: Input the training sample of the third face image into the third preset base model to obtain the training dense points corresponding to the training sample of the third face image output by the third preset base model.

[0243] The aforementioned third preset basic model is the basic model for predicting the dense points of the training samples of the third face image.

[0244] Accordingly, the aforementioned dense training points are the dense points of the third face image training samples predicted by the aforementioned third preset basic model.

[0245] In this embodiment of the invention, a third face image training sample can be input into the third preset basic model to obtain the training dense points corresponding to the third face image training sample output by the third preset basic model.

[0246] Specifically, the aforementioned third preset basic model may include training a basic network layer, training a fully connected layer, and training a dense point prediction layer.

[0247] Based on this, the execution subject of this embodiment of the invention can input the training sample of the third face image into the training base network layer to obtain the training base network features output by the training base network layer.

[0248] Then, the above-mentioned training base network features are input into the above-mentioned training fully connected layer to obtain the training dense point features output by the above-mentioned training fully connected layer, and the training dense point features are input into the above-mentioned training dense point prediction layer to obtain the training dense points corresponding to the above-mentioned first face image training sample.

[0249] Step 603: Based on the above training dense points, generate a heatmap of the training dense points corresponding to the training samples of the third face image.

[0250] In one embodiment, the executing entity of this invention can generate a dense point heatmap (hereinafter referred to as the training dense point heatmap) corresponding to the training dense points of the third face image training sample.

[0251] Specifically, a single-channel image can be created first. This single-channel image is completely black, and its dimensions are consistent with the dimensions of the basic network features (hereinafter referred to as the training basic network features) of the training samples of the second face image. The training basic network features of the training samples of the second face image can be obtained by the dense point prediction model during the process of processing the second face image to obtain the training dense points.

[0252] Then, the aforementioned dense training points can be labeled on the single-channel image, and the pixel values ​​at the corresponding positions can be filled with preset values ​​to obtain the basic training dense point heatmap corresponding to the training sample of the second face image. Optionally, the aforementioned preset value can be 1.

[0253] Finally, a preset blurring algorithm can be used to blur the aforementioned basic training dense point heatmap to obtain the training dense point heatmap corresponding to the second face image training sample. Specifically, for each dense point in the aforementioned basic training dense point heatmap, a Gaussian blur algorithm can be applied, setting the blur radius to r. This process expands each dense point into a circular region of radius r. The pixel value at the center of the circle is the brightest, decreasing sequentially outwards, with values ​​outside the radius region being 0. This yields the aforementioned training dense point heatmap corresponding to the second face image training sample.

[0254] Step 604: Input the training dense point heatmap into the fourth preset base model to obtain the training sparse points corresponding to the training samples of the third face image output by the fourth preset base model.

[0255] The aforementioned fourth preset basic model is a basic model used to predict sparse points in training samples of third-person face images.

[0256] Accordingly, the above-mentioned training sparse points are the sparse points obtained by processing the above-mentioned dense point heatmap by the above-mentioned fourth preset basic model (hereinafter referred to as training sparse points for ease of description).

[0257] In this embodiment of the invention, the above-mentioned dense training point heatmap can be input into the above-mentioned fourth preset basic model to obtain the sparse training points corresponding to the third face image training sample output by the above-mentioned fourth preset basic model.

[0258] Specifically, the aforementioned fourth preset basic model may include at least one training sparse point feature prediction layer, a training fully connected layer, and a training sparse point prediction layer.

[0259] Based on this, the execution subject of this embodiment of the invention can pass the training base network features of the above-mentioned training dense point heatmap and the training sample of the third face image through at least one training sparse point feature prediction layer to obtain the sparse point features corresponding to the training sample of the third face image (hereinafter referred to as the first training sparse point feature for ease of description).

[0260] Then, the first training sparse point features can be input into the training fully connected layer to obtain the sparse point features corresponding to the training sample of the second face image (hereinafter referred to as the second training sparse point features for ease of description), and the second sparse point features can be input into the training sparse point prediction layer to obtain the training sparse points corresponding to the training sample of the second face image.

[0261] Furthermore, the aforementioned training sparse point feature prediction layer may include a training channel connection layer and at least one training convolutional layer.

[0262] In one embodiment, the aforementioned fourth preset base model may include a training sparse point feature prediction layer. Based on this, the execution entity of this embodiment can input the aforementioned training dense point heatmap and training base network features into the training channel connection layer of the training sparse point feature prediction layer for channel connection to obtain the connection features corresponding to the training dense point heatmap (hereinafter referred to as training connection features for ease of description), and then input the training connection features sequentially through at least one training convolutional layer for convolution processing to obtain the first training sparse point features corresponding to the training sample of the third face image.

[0263] In another embodiment, the aforementioned fourth preset base model may include two training sparse point feature prediction layers. Based on this, the execution entity of this embodiment can input the training dense point heatmap and the training base network features into the training channel connection layer of the first training sparse point feature prediction layer (hereinafter referred to as the first training sparse point feature prediction layer for ease of description) to perform channel connection, thereby obtaining the connection features corresponding to the training dense point heatmap (hereinafter referred to as the first training connection features for ease of description), and then sequentially pass the first training connection features through at least one training convolutional layer of the aforementioned first training sparse point feature prediction layer for convolution processing, thereby obtaining the sparse point features corresponding to the third face image training sample (hereinafter referred to as the third training sparse point features for ease of description).

[0264] Subsequently, the execution subject of this embodiment of the invention can input the above-mentioned third training sparse point feature and the training base network feature of the above-mentioned third face image training sample into the training channel connection layer of the second training sparse point feature prediction layer (hereinafter referred to as the second training sparse point feature prediction layer for ease of description) to perform channel connection, and obtain the connection feature corresponding to the training dense point heatmap (hereinafter referred to as the second training connection feature for ease of description).

[0265] Then, the second training connection features are sequentially processed through at least one training convolutional layer of the second training sparse point feature prediction layer to obtain the first training sparse point features corresponding to the third face image training sample.

[0266] It should be noted that the aforementioned fourth preset basic model may also include two or more training sparse point feature prediction layers, and the embodiments of the present invention do not impose any restrictions on this.

[0267] Step 605: Using the preset loss function, calculate the third loss value for the training dense points and the sample dense points to obtain the third loss value for this training, and calculate the fourth loss value for the training sparse points and the sample sparse points to obtain the fourth loss value for this training.

[0268] Step 606: If both the third and fourth loss values ​​meet the preset conditions, stop training to obtain the dense point prediction model and the sparse point prediction model.

[0269] The following provides a unified explanation of steps 605 and 606:

[0270] The third loss value mentioned above is used to characterize the loss between the training dense points and the sample dense points.

[0271] The fourth loss value mentioned above is used to characterize the loss between the training sparse points and the sample sparse points.

[0272] In this embodiment of the invention, after obtaining the training dense points and training sparse points, a preset loss function can be used to calculate the training dense points and sample dense points, as well as the training sparse points and sample sparse points, to obtain the third loss value and the fourth loss value for this training.

[0273] Optionally, if the third and fourth loss values ​​meet preset conditions, training is stopped to obtain the dense point prediction model and the sparse point prediction model. The preset conditions can be less than or equal to a preset loss value threshold, which can be 0.2 or 0.1. This embodiment of the invention does not limit this.

[0274] Conversely, if either the third or fourth loss value does not meet the aforementioned preset conditions, the third face image training sample can be input into the third preset base model to continue training the third and fourth preset base models.

[0275] See Figure 8 This is a block diagram illustrating an embodiment of a facial key point detection device provided by an embodiment of the present invention. Figure 8As shown, the device includes:

[0276] The acquisition module 81 is used to acquire the face image to be detected;

[0277] The first input module 82 is used to input the face image into a preset dense point prediction model to obtain the dense point corresponding to the face image;

[0278] The generation module 83 is used to generate a dense point heatmap corresponding to the face image based on the dense point locations.

[0279] The second input module 84 is used to input the dense point heat map into a preset sparse point prediction model to obtain the key points corresponding to the face image.

[0280] As one possible implementation, the dense point prediction model includes a base network layer, a fully connected layer, and a dense point prediction layer. The first input module 82 is specifically used for:

[0281] The face image is input into the base network layer to obtain the base network features output by the base network layer;

[0282] The basic network features are input into the fully connected layer to obtain the dense point features output by the fully connected layer;

[0283] The dense point features are input into the dense point prediction layer to obtain the dense points corresponding to the face image.

[0284] As one possible implementation, the generation module 83 is specifically used for:

[0285] Create a single-channel image, wherein the dimensions of the single-channel image are the same as the dimensions of the base network features;

[0286] The dense points are marked on the single-channel image, and the pixel values ​​at the corresponding positions are filled with preset values ​​to obtain the basic dense point heat map corresponding to the face image.

[0287] A preset fuzzing algorithm is used to blur the basic dense point heatmap to obtain the dense point heatmap corresponding to the face image.

[0288] As one possible implementation, the sparse point prediction model includes at least one sparse point feature prediction layer, a fully connected layer, and a sparse point prediction layer. The second input module 84 includes:

[0289] The first determining submodule is used to pass the dense point heatmap and the basic network features through at least one sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image.

[0290] The first input submodule is used to input the first sparse point feature into the fully connected layer to obtain the second sparse point feature corresponding to the face image;

[0291] The second input submodule is used to input the second sparse point feature into the sparse point prediction layer to obtain the key points corresponding to the face image.

[0292] As one possible implementation, the sparse point feature prediction layer includes a channel connection layer and at least one convolutional layer. In the case where the sparse point prediction model includes two sparse point feature prediction layers, the first determining submodule is specifically used for:

[0293] The dense point heatmap and the basic network features are input into the channel connection layer of the first sparse point feature prediction layer for channel connection to obtain the first connection feature corresponding to the dense point heatmap;

[0294] The first connection feature is sequentially processed by convolutional layer at least one of the convolutional layers of the first sparse point feature prediction layer to obtain the third sparse point feature corresponding to the face image.

[0295] The third sparse point feature and the basic network feature are input into the channel connection layer of the second sparse point feature prediction layer to perform channel connection, thereby obtaining the second connection feature corresponding to the dense point heatmap.

[0296] The second connection feature is sequentially processed by convolutional layer at least one of the convolutional layers of the second sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image.

[0297] As one possible implementation, the device further includes (not shown in the figure):

[0298] The first sample acquisition module is used to acquire a first face image training sample, wherein the first face image training sample is labeled with dense sample points corresponding to the face image;

[0299] The first sample input module is used to input the first face image training sample into the first preset base model to obtain the training dense points corresponding to the first face image training sample output by the first preset base model.

[0300] The first loss value calculation module is used to calculate the first loss value of the training dense points and the sample dense points using a preset loss function.

[0301] The first model determination module is used to stop training and obtain the dense point prediction model when the first loss value meets the preset conditions.

[0302] As one possible implementation, the first preset base model includes training a base network layer, training a fully connected layer, and training a dense point prediction layer. The first sample input module is specifically used for:

[0303] The first face image training sample is input into the training base network layer to obtain the training base network features output by the training base network layer.

[0304] The training base network features are input into the training fully connected layer to obtain the training dense point features output by the training fully connected layer.

[0305] The training dense point features are input into the training dense point prediction layer to obtain the training dense points corresponding to the training samples of the first face image.

[0306] As one possible implementation, the device further includes (not shown in the figure):

[0307] The second sample acquisition module is used to acquire a second face image training sample, wherein the second face image training sample is labeled with sparse points corresponding to the face image.

[0308] The second sample input module is used to input the second face image training sample into the dense point prediction model to obtain the training dense point corresponding to the second face image training sample.

[0309] The heatmap generation module is used to generate a heatmap of the training dense points corresponding to the training samples of the second face image based on the training dense points.

[0310] The third sample input module is used to input the training dense point heatmap into the second preset base model to obtain the training sparse points corresponding to the training samples of the second face image output by the second preset base model.

[0311] The second loss value calculation module is used to calculate the second loss value of the training sparse points and the sample sparse points using a preset loss function.

[0312] The second model determination module is used to stop training when the second loss value meets the preset conditions, and obtain the sparse point prediction model.

[0313] As one possible implementation, the heatmap generation module is specifically used for:

[0314] Create a single-channel image, wherein the length and width of the single-channel image are the same as the length and width of the training base network features, wherein the training base network features are obtained by processing the training samples of the second face image by the base network layer in the dense point prediction model;

[0315] The training dense points are marked on the single-channel image, and the pixel values ​​at the corresponding positions are filled with preset values ​​to obtain the basic training dense point heatmap corresponding to the training sample of the second face image.

[0316] A preset fuzzing algorithm is used to fuzz the base training dense point heatmap to obtain the training dense point heatmap corresponding to the second face image training sample.

[0317] As one possible implementation, the second preset base model includes at least one training sparse point feature prediction layer, a training fully connected layer, and a training sparse point prediction layer; the second sample input module includes:

[0318] The determination submodule is used to pass the training dense point heatmap and training basic network features through at least one training sparse point feature prediction layer to obtain the first training sparse point feature corresponding to the training sample of the second face image.

[0319] The first sample input submodule is used to input the first training sparse point feature into the training fully connected layer to obtain the second training sparse point feature corresponding to the second face image training sample.

[0320] The second sample input submodule is used to input the second training sparse point feature into the training sparse point prediction layer to obtain the training sparse points corresponding to the training samples of the second face image.

[0321] As one possible implementation, the training sparse point feature prediction layer includes a training channel connection layer and at least one training convolutional layer. In the case where the second preset base model includes two training sparse point feature prediction layers, the determining submodule is specifically used for:

[0322] The training dense point heatmap and the training basic network features are input into the training channel connection layer of the first training sparse point feature prediction layer to perform channel connection, thereby obtaining the first training connection feature corresponding to the dense point heatmap.

[0323] The first training connection feature is sequentially processed by at least one of the training convolutional layers of the first training sparse point feature prediction layer to obtain the third training sparse point feature corresponding to the training sample of the second face image.

[0324] The third training sparse point feature and the basic network feature are input into the training channel connection layer of the second training sparse point feature prediction layer to perform channel connection, thereby obtaining the second training connection feature corresponding to the dense point heatmap.

[0325] The second training connection features are sequentially processed by passing them through at least one of the training convolutional layers of the second training sparse point feature prediction layer to obtain the first training sparse point features corresponding to the training samples of the second face image.

[0326] As one possible implementation, the device further includes (not shown in the figure):

[0327] The third sample acquisition module is used to acquire third face image training samples, wherein the third face image training samples are labeled with dense and sparse points corresponding to the face images.

[0328] The fourth sample input module is used to input the third face image training sample into the third preset base model to obtain the training dense points corresponding to the third face image training sample output by the third preset base model.

[0329] The point heatmap generation module is used to generate a training dense point heatmap corresponding to the training sample of the third face image based on the training dense points.

[0330] The fifth sample input module is used to input the training dense point heatmap into the fourth preset base model to obtain the training sparse points corresponding to the training samples of the third face image output by the fourth preset base model.

[0331] The third loss value calculation module is used to calculate the third loss value of the training dense points and the sample dense points using a preset loss function, and to calculate the fourth loss value of the training sparse points and the sample sparse points.

[0332] The training stop module is used to stop training when both the third loss value and the fourth loss value meet preset conditions, thereby obtaining the dense point prediction model and the sparse point prediction model.

[0333] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 9The illustrated electronic device 900 includes at least one processor 901, a memory 902, at least one network interface 904, and a user interface 903. The various components in the electronic device 900 are coupled together via a bus system 905. It is understood that the bus system 905 is used to implement communication between these components. In addition to a data bus, the bus system 905 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 9 The general labeled all buses as Bus System 905.

[0334] The user interface 903 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0335] It is understood that the memory 902 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate Synchronous DRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 902 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0336] In some implementations, memory 902 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 9021 and application program 9022.

[0337] The operating system 9021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 9022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 9022.

[0338] In this embodiment of the invention, by calling the program or instructions stored in the memory 902, specifically the program or instructions stored in the application program 9022, the processor 901 executes the method steps provided in each method embodiment, including, for example:

[0339] Acquire the face image to be detected;

[0340] The face image is input into a preset dense point prediction model to obtain the dense points corresponding to the face image;

[0341] Based on the dense points, a dense point heatmap corresponding to the face image is generated;

[0342] The dense point heatmap is input into a preset sparse point prediction model to obtain the key points corresponding to the face image.

[0343] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 901. Processor 901 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware or by instructions in the form of software in processor 901. The processor 901 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 902. Processor 901 reads the information in memory 902 and, in conjunction with its hardware, completes the steps of the above method.

[0344] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0345] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0346] The electronic device provided in this embodiment may be as follows: Figure 9 The electronic device shown can perform the following: Figure 1 All steps of the method for detecting facial key points in Chinese, thereby achieving... Figure 1 For details on the technical effectiveness of the facial landmark detection method, please refer to [link / reference needed]. Figure 1 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0347] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0348] One or more programs in the storage medium can be executed by one or more processors to implement the above-mentioned method for detecting facial key points executed on the electronic device side.

[0349] The processor is used to execute a facial landmark detection method stored in the memory to implement the following steps of the facial landmark detection method executed on the electronic device side:

[0350] Acquire the face image to be detected;

[0351] The face image is input into a preset dense point prediction model to obtain the dense points corresponding to the face image;

[0352] Based on the dense points, a dense point heatmap corresponding to the face image is generated;

[0353] The dense point heatmap is input into a preset sparse point prediction model to obtain the key points corresponding to the face image.

[0354] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0355] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0356] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting human face key points, characterized in that, The method includes: Acquire the face image to be detected; The face image is input into a preset dense point prediction model to obtain the dense points corresponding to the face image; wherein, the dense point prediction model includes at least a base network layer for extracting the basic network features of the face image; Based on the dense points, a dense point heatmap corresponding to the face image is generated; The dense point heatmap is input into a preset sparse point prediction model to obtain the key points corresponding to the face image; wherein, the sparse point prediction model includes at least one sparse point feature prediction layer for processing the dense point heatmap and the basic network features.

2. The method according to claim 1, characterized in that, The dense point prediction model includes a base network layer, a fully connected layer, and a dense point prediction layer. The step of inputting the face image into the preset dense point prediction model to obtain the dense points corresponding to the face image includes: The face image is input into the base network layer to obtain the base network features output by the base network layer; The basic network features are input into the fully connected layer to obtain the dense point features output by the fully connected layer; The dense point features are input into the dense point prediction layer to obtain the dense points corresponding to the face image.

3. The method according to claim 2, characterized in that, The step of generating a dense point heatmap corresponding to the face image based on the dense point locations includes: Create a single-channel image, wherein the dimensions of the single-channel image are the same as the dimensions of the base network features; The dense points are marked on the single-channel image, and the pixel values ​​at the corresponding positions are filled with preset values ​​to obtain the basic dense point heat map corresponding to the face image. A preset fuzzing algorithm is used to blur the basic dense point heatmap to obtain the dense point heatmap corresponding to the face image.

4. The method according to claim 2, characterized in that, The sparse point prediction model includes at least one sparse point feature prediction layer, a fully connected layer, and a sparse point prediction layer. The step of inputting the dense point heatmap into the preset sparse point prediction model to obtain the key points corresponding to the face image includes: The dense point heatmap and the basic network features are passed through at least one sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image; The first sparse point feature is input into the fully connected layer to obtain the second sparse point feature corresponding to the face image; The second sparse point feature is input into the sparse point prediction layer to obtain the key points corresponding to the face image.

5. The method according to claim 4, characterized in that, The sparse point feature prediction layer includes a channel connection layer and at least one convolutional layer. When the sparse point prediction model includes two sparse point feature prediction layers, the step of passing the dense point heatmap and the basic network features through at least one of the sparse point feature prediction layers to obtain the first sparse point feature corresponding to the face image includes: The dense point heatmap and the basic network features are input into the channel connection layer of the first sparse point feature prediction layer for channel connection to obtain the first connection feature corresponding to the dense point heatmap; The first connection feature is sequentially processed by convolutional layer at least one of the convolutional layers of the first sparse point feature prediction layer to obtain the third sparse point feature corresponding to the face image. The third sparse point feature and the basic network feature are input into the channel connection layer of the second sparse point feature prediction layer to perform channel connection, thereby obtaining the second connection feature corresponding to the dense point heatmap. The second connection feature is sequentially processed by convolutional layer at least one of the convolutional layers of the second sparse point feature prediction layer to obtain the first sparse point feature corresponding to the face image.

6. The method according to claim 1, characterized in that, The dense point prediction model is trained using the following method: Obtain a first face image training sample, and the first face image training sample is labeled with dense sample points corresponding to the face image; Input the first face image training sample into the first preset base model to obtain the training dense points corresponding to the first face image training sample output by the first preset base model. Using a preset loss function, the training dense points and the sample dense points are calculated to obtain the first loss value for this training. If the first loss value meets the preset conditions, training is stopped, and the dense point prediction model is obtained.

7. The method according to claim 6, characterized in that, The first preset base model includes a training base network layer, a training fully connected layer, and a training dense point prediction layer. The step of inputting the first face image training sample into the first preset base model to obtain the training dense points corresponding to the first face image training sample output by the first preset base model includes: The first face image training sample is input into the training base network layer to obtain the training base network features output by the training base network layer. The training base network features are input into the training fully connected layer to obtain the training dense point features output by the training fully connected layer. The training dense point features are input into the training dense point prediction layer to obtain the training dense points corresponding to the training samples of the first face image.

8. The method according to claim 1, characterized in that, The sparse point prediction model is trained using the following method: Obtain a second face image training sample, and the second face image training sample is labeled with sparse points corresponding to the face image; The training sample of the second face image is input into the dense point prediction model to obtain the training dense point corresponding to the training sample of the second face image. Based on the training dense points, a heatmap of the training dense points corresponding to the training samples of the second face image is generated; The training dense point heatmap is input into the second preset base model to obtain the training sparse points corresponding to the training samples of the second face image output by the second preset base model. Using a preset loss function, the training sparse points and the sample sparse points are calculated to obtain the second loss value for this training. If the second loss value meets the preset conditions, training is stopped, and the sparse point prediction model is obtained.

9. The method according to claim 8, characterized in that, The step of generating a heatmap of training dense points corresponding to the training samples of the second face image based on the training dense points includes: Create a single-channel image, wherein the length and width of the single-channel image are the same as the length and width of the training base network features, wherein the training base network features are obtained by processing the training samples of the second face image by the base network layer in the dense point prediction model; The training dense points are marked on the single-channel image, and the pixel values ​​at the corresponding positions are filled with preset values ​​to obtain the basic training dense point heatmap corresponding to the training sample of the second face image. A preset fuzzing algorithm is used to blur the basic training dense point heatmap to obtain the training dense point heatmap corresponding to the training sample of the second face image.

10. The method according to claim 8, characterized in that, The second preset base model includes at least one training sparse point feature prediction layer, a training fully connected layer, and a training sparse point prediction layer. The step of inputting the training dense point heatmap into the second preset base model to obtain the training sparse points corresponding to the training samples of the second face image output by the second preset base model includes: The training dense point heatmap and training basic network features are passed through at least one training sparse point feature prediction layer to obtain the first training sparse point feature corresponding to the training sample of the second face image. The first training sparse point feature is input into the training fully connected layer to obtain the second training sparse point feature corresponding to the training sample of the second face image. The second training sparse point feature is input into the training sparse point prediction layer to obtain the training sparse points corresponding to the training samples of the second face image.

11. The method according to claim 10, characterized in that, The training sparse point feature prediction layer includes a training channel connection layer and at least one training convolutional layer. When the second preset base model includes two training sparse point feature prediction layers, the step of passing the training dense point heatmap and training base network features through at least one training sparse point feature prediction layer to obtain the first training sparse point feature corresponding to the second face image training sample includes: The training dense point heatmap and the training basic network features are input into the training channel connection layer of the first training sparse point feature prediction layer to perform channel connection, thereby obtaining the first training connection feature corresponding to the dense point heatmap. The first training connection feature is sequentially processed by at least one of the training convolutional layers of the first training sparse point feature prediction layer to obtain the third training sparse point feature corresponding to the training sample of the second face image. The third training sparse point feature and the basic network feature are input into the training channel connection layer of the second training sparse point feature prediction layer to perform channel connection, thereby obtaining the second training connection feature corresponding to the dense point heatmap. The second training connection features are sequentially processed by convolutional processing through at least one of the training convolutional layers of the second training sparse point feature prediction layer to obtain the first training sparse point features corresponding to the training samples of the second face image.

12. The method according to claim 1, characterized in that, The dense point prediction model and the sparse point prediction model are trained in the following manner: Obtain training samples of third-person face images, wherein the training samples of third-person face images are labeled with dense and sparse points corresponding to the face images; The training sample of the third face image is input into the third preset base model to obtain the training dense points corresponding to the training sample of the third face image output by the third preset base model. Based on the training dense points, a heatmap of the training dense points corresponding to the training samples of the third face image is generated; The training dense point heatmap is input into the fourth preset base model to obtain the training sparse points corresponding to the training samples of the third face image output by the fourth preset base model. Using a preset loss function, the third loss value for this training is calculated for the dense training points and the dense sample points, and the fourth loss value for this training is calculated for the sparse training points and the sparse sample points. If both the third and fourth loss values ​​meet the preset conditions, training is stopped, and the dense point prediction model and the sparse point prediction model are obtained.

13. A device for detecting facial key points, characterized in that, The device includes: The acquisition module is used to acquire the face image to be detected; The first input module is used to input the face image into a preset dense point prediction model to obtain the dense points corresponding to the face image; wherein, the dense point prediction model includes at least a basic network layer for extracting the basic network features of the face image; The generation module is used to generate a dense point heatmap corresponding to the face image based on the dense point locations. The second input module is used to input the dense point heatmap into a preset sparse point prediction model to obtain the key points corresponding to the face image; wherein, the sparse point prediction model includes at least one sparse point feature prediction layer for processing the dense point heatmap and the basic network features.

14. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a facial landmark detection program stored in the memory to implement the facial landmark detection method according to any one of claims 1 to 12.

15. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the facial key point detection method according to any one of claims 1 to 12.

Images