Methods and apparatus for predicting key point coordinates of face images

Abstract

A method and apparatus for predicting the coordinates of keypoints in a face image are provided. The method includes: obtaining multi-level feature maps of the face image through convolutional neural network layers; fully connecting the last-level feature maps in the multi-level feature maps through fully connected layers to obtain an initial query matrix, wherein the initial query matrix represents the initial features of the keypoints in the face image, and the number of elements in the initial query matrix is ​​equal to the number of keypoints in the face image; obtaining a memory feature matrix by flattening and connecting the multi-level feature maps; and inputting the memory feature matrix and the initial query matrix into at least one cascaded decoder layer to determine the coordinates of the keypoints in the face image.

Description

Technical Field

[0001] This application relates to the field of facial landmark detection technology, and more specifically, to a method and apparatus for predicting the coordinates of key points in a facial image. Background Technology

[0002] Facial landmark detection, also known as face alignment, aims to automatically locate key landmarks on a face. It is an important component of face analysis tasks such as face recognition, facial expression analysis, face frontalization, and 3D face reconstruction.

[0003] In recent years, due to the rapid development of deep neural network technology, facial landmark detection has been significantly improved. Existing facial landmark detection methods are mainly divided into two categories: coordinate regression-based methods and heatmap regression-based methods.

[0004] Coordinate regression-based methods directly map the input image to keypoint coordinates. In models utilizing deep learning frameworks, the input image is fed into a CNN model to obtain image features. These features are then directly mapped to coordinates via a fully connected prediction layer. To improve detection accuracy, coordinate regression modules are typically cascaded or integrated with heatmap regression modules.

[0005] Heatmap-based regression methods first generate heatmaps based on given keypoint coordinates, which serve as surrogates for ground truth. Each heatmap represents the probability of a keypoint location. Heatmap-based regression models typically predict heatmaps using fully convolutional networks and then derive keypoints based on the peak probability locations on the heatmaps. Because heatmap-based models preserve the spatial structure of image features, they generally outperform coordinate regression-based models.

[0006] Although heatmap-based regression methods have relatively high accuracy, they suffer from the following inherent problems: 1) A post-processing step is required to map the heatmap probability output to coordinates; however, the post-processing step is non-differentiable, which disables end-to-end training of the entire framework; 2) Considering computational complexity, the resolution of the heatmap is usually lower than that of the input image, which inevitably leads to quantization errors and limits further improvements; 3) Manually designing the generation of ground truth heatmaps introduces heuristic hyperparameters that need to be tuned.

[0007] In contrast, coordinate regression-based methods can circumvent the aforementioned drawbacks and achieve end-to-end model training. However, using fully connected prediction layers disrupts the spatial structure of image features. The mapping from global features to keypoint coordinates becomes like a black box, where features and predictions are misaligned, which significantly degrades localization performance.

[0008] Therefore, there is an urgent need for a method and apparatus that can more accurately predict the coordinates of facial key points. On the one hand, such a method and apparatus can overcome the aforementioned shortcomings of heatmap-based methods and can be well trained end-to-end without heuristic post-processing. On the other hand, it can extract most of the relevant image features around the key points for coordinate prediction. Summary of the Invention

[0009] The purpose of this invention is to provide a method and apparatus for predicting the coordinates of key points in a face image, so as to at least solve the problems in the above-mentioned related technologies, or it may not solve any of the above-mentioned problems.

[0010] According to one aspect of an exemplary embodiment of the present disclosure, a method for predicting keypoint coordinates of a face image is provided, comprising: obtaining a multi-level feature map of the face image through a convolutional neural network layer; performing a fully connected layer on the last level feature map of the multi-level feature map to obtain an initial query matrix, wherein the initial query matrix represents initial features of keypoints of the face image, and wherein the number of elements in the initial query matrix is ​​equal to the number of keypoints of the face image; obtaining a memory feature matrix by performing flattening and concatenation operations on the multi-level feature map; and inputting the memory feature matrix and the initial query matrix into at least one cascaded decoder layer to determine the keypoint coordinates of the face image.

[0011] The exemplary embodiments of this disclosure extract N (the number of key points in a face image) features from the last layer of the backbone neural network using a simple fully connected layer and use them as the initial query matrix, instead of using a randomly initialized query matrix. This design speeds up the convergence of the model and further improves the accuracy of predicting the coordinates of key points in face images.

[0012] Optionally, each decoder layer includes a cascaded self-attention module layer, a deformable attention module layer, and a keypoint coordinate prediction layer.

[0013] Optionally, the step of determining the key point coordinates of the face image includes: inputting an initial query matrix embedded with location information, an initial query matrix embedded with location information, and an initial query matrix as the query matrix, key matrix, and value matrix of the self-attention module layer of the first decoder layer into the self-attention module layer of the first decoder layer; inputting the output matrix, memory feature matrix, and key point coordinates predicted by the previous decoder layer of the current decoder layer into the deformable attention module layer of the current decoder layer to obtain the output matrix of the deformable attention module layer, wherein the output matrix and memory feature matrix of the self-attention module layer are the query matrix and value matrix of the deformable attention module layer, and the key point coordinates predicted by the previous decoder layer into the deformable attention module layer of the current decoder layer. The keypoint coordinates predicted by the previous decoder layer are the initial keypoint coordinates obtained based on the initial query moments. The output matrix of the deformable attention module layer of the current decoder layer, the output matrix of the deformable attention module layer with embedded position information, and the output matrix of the deformable attention module layer with embedded position information are used as the value matrix, query matrix, and key matrix of the self-attention module layer of the cascaded next decoder layer and input to the self-attention module layer of the next decoder layer. The output matrix of the deformable attention module layer of the current decoder layer is input to the keypoint coordinate prediction layer of the current decoder layer to obtain the keypoint coordinates of the face image predicted by each decoder layer. The keypoint coordinates of the face image predicted by the last decoder layer are used as the keypoint coordinates of the final face image.

[0014] By cascading the layers, end-to-end training of the model can be achieved. At the same time, the attention mechanism can be used to focus more on the relationship between feature points and surrounding features, thereby improving the accuracy of prediction.

[0015] Optionally, the output matrix of the self-attention module layer of each decoder layer is obtained by the following formula:

[0016]

[0017] in, Let α represent the i-th row vector in the output matrix. ij The attention weight q represents the normalized result of the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer. j The j-th row vector in the output matrix of the initial query matrix or the deformed attention layer of the previous decoder layer represents the number of key points.

[0018] The self-attention module layer learns the structural dependencies between keypoints. This information is important for keypoint localization, thereby improving the accuracy of keypoint coordinate prediction.

[0019] Optionally, the output matrix of the deformable attention module layer of each decoder layer is obtained by the following formula:

[0020]

[0021] Among them, f i β represents the updated feature of the i-th key point. ik x represents the attention weights obtained by performing fully connected and softmax operations on the query matrix input to the deformable attention module layer. ik The feature in the memory feature matrix corresponding to the coordinates of the k-th reference point is represented by the positional offset between the coordinates of the k-th reference point and the coordinates of the i-th keypoint in the keypoint coordinates predicted by the previous decoder layer. This offset is obtained by performing a fully connected operation on the query matrix input to the deformable attention module layer. Here, K is a preset value.

[0022] Instead of calculating the relationship between each element in the output matrix of the self-attention module layer and the memory feature matrix, the deformable attention module layer focuses only on a small set of features obtained by sampling the memory features based on the keypoint coordinates predicted by the previous decoder layer. This greatly reduces the area of ​​feature search, thereby speeding up the convergence of the model.

[0023] Optionally, the keypoint coordinates predicted by each decoder layer are obtained using the following formula:

[0024] y = σ(y O +σ -1 (y R ))

[0025] Where y represents the keypoint coordinates predicted by the current decoder layer, y R This represents the keypoint coordinates predicted by the previous decoder layer or the initial keypoint coordinates, y O This represents the output of the keypoint coordinate prediction layer, indicating y relative to y R The offset. Optionally, the convolutional network layers, fully connected layers, and at least one decoder layer are obtained by training based on the following regression loss function:

[0026]

[0027] Among them, L reg Let y represent the regression loss function. l This represents the keypoint coordinates of the training samples predicted by each decoder layer. L represents the actual keypoint coordinates of the training samples. d is the number of decoder layers, and l is the index of the decoder layer.

[0028] This loss function enables end-to-end training of the entire framework, thereby enabling more accurate prediction of key point coordinates in face images.

[0029] According to one aspect of an exemplary embodiment of the present disclosure, an apparatus for predicting keypoint coordinates of a face image is provided, comprising: an encoder configured to: acquire a multi-level feature map of the face image through a convolutional neural network layer; fully connect the last level feature map in the multi-level feature map through a fully connected layer to obtain an initial query matrix, wherein the initial query matrix represents initial features of keypoints in the face image, wherein the number of elements in the initial query matrix is ​​equal to the number of keypoints in the face image; and acquire a memory feature matrix by flattening and connecting the multi-level feature maps; and a decoder including at least one cascaded decoder layer, wherein the at least one decoder layer is configured to: determine the keypoint coordinates of the face image based on the memory feature matrix received from the encoder and the initial query matrix.

[0030] The exemplary embodiments of this disclosure extract N (i.e., the number of key points in a face image) features from the last layer of the backbone neural network using a simple fully connected layer and use them as the initial query matrix, instead of using a randomly initialized query matrix. This simple design speeds up the convergence of the model and further improves the accuracy of predicting the coordinates of key points in face images.

[0031] Optionally, each decoder layer includes a cascaded self-attention module layer, a deformable attention module layer, and a keypoint coordinate prediction layer.

[0032] Optionally, the self-attention module layer of the first decoder layer is configured to: obtain the output matrix of the self-attention module layer of the first decoder layer based on the received initial query matrix embedded with location information, the initial query matrix embedded with location information, and the initial query matrix, wherein the received initial query matrix embedded with location information, the initial query matrix embedded with location information, and the initial query matrix are respectively the query matrix, key matrix, and value matrix input to the self-attention module layer of the first layer. The self-attention module layer of each decoder layer other than the first decoder layer is configured to: obtain the output matrix of the deformable attention module of the current decoder layer based on the output matrix of the deformable attention module layer of the cascaded previous decoder layer, the output matrix of the deformable attention module layer of the previous decoder layer embedded with location information, and the output matrix of the deformable attention module layer of the previous decoder layer embedded with location information. The output matrix of the deformable attention module layer of the previous decoder layer, which embeds location information, is the value matrix, query matrix, and key matrix of the self-attention module layer of the current decoder layer. The deformable attention module layer of each decoder layer is configured to obtain the output matrix of the deformable attention module layer of each decoder layer based on the output matrix of the self-attention module layer of the current decoder layer, the memory feature matrix, and the key point coordinates predicted by the cascaded previous decoder layer. The output matrix and memory feature matrix of the self-attention module layer of the current decoder layer are the query matrix and value matrix of the deformable attention module layer of the current decoder layer. The key point coordinates predicted by the previous decoder layer of the first decoder layer are the initial key point coordinates obtained based on the initial query matrix. Each key point coordinate prediction layer is configured to obtain the key point coordinates of the face image predicted by the current decoder layer based on the output matrix of the deformable attention module layer of the current decoder layer. The key point coordinates of the face image predicted by the last decoder layer are used as the key point coordinates of the final face image.

[0033] By cascading the layers, end-to-end training of the model can be achieved. At the same time, by utilizing the attention mechanism, more attention can be paid to the relationship between feature points and surrounding features, thereby improving the accuracy of predicting the coordinates of key points in face images.

[0034] Optionally, the self-attention module layer of each decoder layer obtains the output matrix based on the following formula:

[0035]

[0036] in, Let α represent the i-th row vector in the output matrix. ijThe attention weight q represents the normalized result of the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer. j This represents the j-th row vector in the initial query matrix or the output matrix of the deformed attention layer of the previous decoder layer, and N represents the number of key points in the face image.

[0037] The self-attention module layer learns the structural dependencies between keypoints. This information is important for keypoint localization, thereby improving the accuracy of keypoint coordinate prediction.

[0038] Optionally, the deformable attention module layer of each decoder layer obtains its output matrix based on the following formula:

[0039]

[0040] Among them, f i β represents the updated feature of the i-th key point. ik x represents the attention weights obtained by performing fully connected and softmax operations on the query matrix input to the deformable attention module layer. ik The feature in the memory feature matrix corresponding to the coordinates of the k-th reference point is represented by the positional offset between the coordinates of the k-th reference point and the coordinates of the i-th keypoint in the keypoint coordinates predicted by the previous decoder layer. This offset is obtained by performing a fully connected operation on the query matrix input to the deformable attention module layer. Here, K is a preset value.

[0041] Instead of calculating the relationship between each element in the output matrix of the self-attention module layer and the memory feature matrix, the deformable attention module layer focuses only on a small set of features obtained by sampling the memory features based on the keypoint coordinates predicted by the previous decoder layer. This greatly reduces the area of ​​feature search, thereby speeding up the convergence of the model.

[0042] Optionally, the keypoint coordinate prediction layer of each decoder layer predicts keypoint coordinates based on the following formula:

[0043] y = σ(y O +σ -1 (y R ))

[0044] Where y represents the keypoint coordinates predicted by the current decoder layer, y R This represents the keypoint coordinates predicted by the previous decoder layer or the initial keypoint coordinates, y O This represents the output of the keypoint coordinate prediction layer, indicating y relative to y R The offset.

[0045] Optionally, the encoder and decoder are trained using training image samples based on the following regression loss function:

[0046]

[0047] Among them, L reg Representing regression loss, y l This represents the keypoint coordinates of the training image samples predicted by each decoder layer. L represents the actual keypoint coordinates of the training image samples. d is the number of decoder layers, and l is the index of the decoder layer.

[0048] This loss function enables end-to-end training of the entire framework, thereby enabling more accurate prediction of key point coordinates in face images.

[0049] According to embodiments of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, it implements the method for predicting key point coordinates of a face image as described above. Attached Figure Description

[0050] The above and other objects and features of the invention will become clearer from the following description, taken in conjunction with the accompanying drawings, which exemplarily illustrate one example, wherein:

[0051] Figure 1 A flowchart of a method for predicting facial landmark coordinates according to an embodiment of the present disclosure is shown;

[0052] Figure 2 This is a flowchart illustrating a method for predicting facial landmark coordinates according to an embodiment of the present disclosure;

[0053] Figure 3 This is a diagram illustrating an example of the structure of the first decoder layer.

[0054] Figure 4 This is a block diagram illustrating the structure of an apparatus for predicting key point coordinates of a face image according to an embodiment of the present disclosure;

[0055] Figure 5 This is a block diagram illustrating the structure of an apparatus for predicting key point coordinates of a face image according to another embodiment of the present disclosure. Detailed Implementation

[0056] In the following description, various embodiments of the present disclosure are illustrated with reference to the accompanying drawings, wherein the same reference numerals are used to denote the same or similar elements, features, and structures. However, the present disclosure is not intended to be limited to the specific embodiments described herein, and it is intended that the present disclosure cover all modifications, equivalents, and / or substitutions of the present disclosure, provided they fall within the scope of the appended claims and their equivalents. The terms and words used in the following description and claims are not limited to their dictionary meanings, but are used only to enable a clear and consistent understanding of the present disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is for illustrative purposes only and is not intended to limit the purpose of the present disclosure as defined by the appended claims and their equivalents.

[0057] It should be understood that, unless the context clearly indicates otherwise, the singular form includes the plural form. The terms “comprising,” “including,” and “having” as used herein indicate the presence of a disclosed function, operation, or element, but do not exclude other functions, operations, or elements.

[0058] For example, the expression “A or B” or “at least one of A and / or B” can indicate A and B, or A or B. For example, the expression “A or B” or “at least one of A and / or B” can indicate (1) A, (2) B or (3) both A and B.

[0059] In various embodiments of this disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to, or being "coupled" or "connected" to, another component (e.g., a second component), the component may be directly connected to, or may be connected via, another component (e.g., a third component). Conversely, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to, or being directly coupled to or directly connected to, another component (e.g., a second component), there is no other component (e.g., a third component) between the component and the other component.

[0060] The expression “configured as” used in describing the various embodiments of this disclosure may be used interchangeably, for example, with expressions such as “suitable for,” “capable of,” “designed to,” “suitable for,” “manufactured as,” and “capable,” depending on the context. The term “configured as” may not necessarily indicate that the hardware is “specifically designed for.” Rather, in some cases, the expression “a device configured as…” may indicate that the device and another device or portion are “capable of….” For example, the expression “a processor configured to perform A, B, and C” may indicate a dedicated processor (e.g., an embedded processor) for performing the respective operations or a general-purpose processor (e.g., a central processing unit CPU or application processor (AP)) for performing the respective operations by executing at least one software program stored in a memory device.

[0061] The terminology used herein is intended to describe certain embodiments of this disclosure but is not intended to limit the scope of other embodiments. Unless otherwise indicated herein, all terms used herein (including technical or scientific terms) are to have the same meaning as commonly understood by one of ordinary skill in the art. Generally, terms as defined in dictionaries should be considered to have the same meaning as in the context of the relevant art and should not be interpreted differently or as having an overly formal meaning unless expressly defined herein. In no event should the terminology defined in this disclosure be construed as excluding embodiments of this disclosure.

[0062] The terminology used herein is intended to describe certain embodiments of this disclosure but is not intended to limit the scope of other embodiments. Unless otherwise indicated herein, all terms used herein (including technical or scientific terms) are to have the same meaning as commonly understood by one of ordinary skill in the art. Generally, terms as defined in dictionaries should be considered to have the same meaning as in the context of the relevant art and should not be interpreted differently or as having an overly formal meaning unless expressly defined herein. In no event should the terminology defined in this disclosure be construed as excluding embodiments of this disclosure.

[0063] According to the facial keypoint coordinate prediction method and apparatus of the present disclosure, keypoint detection is planned as an N-coordinate prediction problem, where N is the number of facial keypoints, such as 6 facial keypoints including the corners of the eyes and mouth, as well as common 68-point and 98-point models. First, a self-attention module is employed to latently learn the structural dependencies between keypoint coordinates, outputting a feature vector represented by each keypoint. Then, a deformable attention module based on image features focuses on a small number of image features sampled around a reference point, rather than searching the entire image (which would slow down model convergence). Information related to the keypoint coordinates predicted in the previous layer or initially (i.e., the reference point) is used as guidance for extracting the most relevant image features around the keypoints and refining the coordinates. Furthermore, exemplary embodiments of the present disclosure extract N features from the last layer of the backbone network in a fully connected manner and use them as the initial query matrix, instead of using a randomly initialized query matrix. This design accelerates model convergence and further improves the accuracy of predicting keypoint coordinates of facial images.

[0064] Figure 1 A flowchart of a method for predicting facial landmark coordinates according to an embodiment of the present disclosure is shown.

[0065] Compared to the deformable DETR model, the method disclosed herein uses fully connected layers to extract more meaningful features based on image feature maps and uses the extracted features as the initial query embedding instead of using randomly generated numbers as the initial query embedding. The number of queries is precisely set to the number of keypoints to be predicted. The initial query embedding based on the extracted features provides image-related coarse reference point localization instead of standard keypoint templates that are image-independent.

[0066] Furthermore, compared to the deformable DETR model, the method of this disclosure removes the encoder layer, retains only the convolutional and fully connected layers, reduces the number of decoder layers (e.g., the deformable DETR includes 6 decoder layers, while this disclosure may include only 3 or fewer decoder layers), and uses only the L1 norm loss (representing the absolute value of the difference between the predicted and the true values) function for model training, thereby reducing the model size and saving computational costs.

[0067] Reference Figure 1 The initial query matrix Q is obtained by performing a fully connected operation on the last-level feature map. init The method obtains a memory feature matrix by performing flattening and concatenation operations on multi-level feature maps. The initial query matrix and the memory feature matrix are then used as input to the decoder layer to predict facial landmark coordinates. The individual steps of the method will be explained below.

[0068] Figure 2This is a flowchart illustrating a method for predicting facial landmark coordinates according to embodiments of the present disclosure. (Refer to...) Figure 2 In step S201, multi-level feature maps of the face image are obtained through convolutional neural network layers (e.g., ...). Figure 1 (The four-level feature maps in the image). Those skilled in the art should understand that the extracted multi-level feature maps are pyramid features, where low-level feature maps represent local features of the image and high-level feature maps represent global features of the image.

[0069] As an example, pyramid features of a face image can be obtained through a backbone network. This patent uses ResNet18, but is not limited to ResNet18. Given an input image size of 256×256×3, the sizes of the feature maps at each level are 64×64×64, 32×32×128, 16×16×256, and 8×8×512, respectively.

[0070] In step S202, the last level feature map in the multi-level feature map is fully connected through a fully connected layer to obtain the initial query matrix Q. init The initial query matrix represents the initial features of the keypoints in the face image, where the number of elements in the initial query matrix is ​​equal to the number of keypoints in the face image. Q init It can be obtained as follows:

[0071] Q init =FC(F),

[0072] Here, F represents the feature map from the last layer of the backbone, denoted as (H×W)×C, where H and W are the width and height of the feature map in space, and C is the feature dimension of the feature map. FC represents a fully connected layer, which maps the spatial scale of each feature channel (H×W) to a vector of size N.

[0073] Query matrix Q init It has a size of N×C, where N is the number of keypoints in the face image and C is the feature dimension. It is essentially a learnable matrix that extracts keypoint-related features and transforms them into coordinates.

[0074] In step S203, a memory feature matrix is ​​obtained by performing flatten and contate operations on the multi-level feature map.

[0075] As an example, a typical encoder-decoder framework can be used. Specifically, a backbone network can be used as the encoder to extract pyramid features from the input image. Then, 1×1 convolutions are applied to these feature maps to obtain feature maps with the same number of output channels. These feature maps are then flattened and concatenated together as a memory feature matrix M.

[0076] In step S204, the initial query matrix and the memory feature matrix are input into at least one cascaded decoder layer to determine the key point coordinates of the face image.

[0077] According to exemplary embodiments of this disclosure, each decoder layer has the same structure, except that the input matrix and output matrix of each decoder layer are different.

[0078] As an example, each decoder layer includes a cascaded self-attention module layer, a deformable attention module layer, and a keypoint coordinate prediction layer. Those skilled in the art will understand that each decoder layer may also include additional layers as needed.

[0079] As an example, the step of determining the key point coordinates of a face image includes: inputting an initial query matrix embedded with location information, the initial query matrix embedded with location information, and the initial query matrix as the query matrix, key matrix, and value matrix of the self-attention module layer of the first decoder layer into the self-attention module layer of the first decoder layer.

[0080] The output matrix, memory feature matrix, and keypoint coordinates predicted by the previous decoder layer of the current decoder layer are input into the deformable attention module layer of the current decoder layer to obtain the output matrix of the deformable attention module layer. The output matrix and memory feature matrix of the self-attention module layer are the query matrix and value matrix of the deformable attention module layer, respectively. The keypoint coordinates predicted by the previous decoder layer of the deformable attention module layer of the first decoder layer are the initial keypoint coordinates obtained based on the initial query matrix.

[0081] As an example, a full join operation can be performed on the query matrix to obtain the initial keypoint coordinates.

[0082] The output matrix of the deformable attention module layer of the current decoder layer, the output matrix of the deformable attention module layer with embedded positional information, and the output matrix of the deformable attention module layer with embedded positional information are used as the value matrix, query matrix, and key matrix of the self-attention module layer of the cascaded next decoder layer and input to the self-attention module layer of the next decoder layer; the output matrix of the deformable attention module layer of the current decoder layer is input to the key point coordinate prediction layer of the current decoder layer to obtain the key point coordinates of the face image predicted by the current decoder layer, wherein the key point coordinates of the face image predicted by the last decoder layer are used as the key point coordinates of the final face image.

[0083] Figure 3 This is a diagram illustrating an example of the structure of the first decoder layer; the other decoder layers have the same structure.

[0084] Reference Figure 3 Each decoder layer includes a self-attention module layer and a deformable attention module layer. The self-attention module layer of each decoder layer may include a self-attention layer and a residual summation and normalization layer (Add&Norm).

[0085] The query matrix, key matrix, and value matrix of the self-attention module layer of the first decoder layer are Q, which are embedded with position information. init Q, which embeds location information init and Q init The query matrix, key matrix, and value matrix of the self-attention module layer of the other decoder layers are respectively the output matrix of the deformable attention module layer of the previous decoder layer with embedded position information, the output matrix of the deformable attention module layer of the previous decoder layer with embedded position information, and the output matrix of the deformable attention layer of the previous decoder layer.

[0086] Each decoder layer's deformable attention module layer may include a deformable attention layer, a residual summation and normalization layer (Add&Norm), and a feedforward layer (FFN).

[0087] The output matrix of the self-attention module layer is used as the query matrix input to the deformable attention module layer (deformable attention layer), and the memory feature matrix is ​​the value matrix input to the deformable attention module layer.

[0088] The self-attention module layer only uses the query matrix Q (Q is used for the first decoder layer). init The input is QP(QP)(QP) for the first decoder layer, which is an initial query matrix QP embedded with location information. This information is actually image-independent and captures pose, expression, etc., from the keypoint locations, which has proven important for keypoint localization. The self-attention module layer takes QP as input. init For other decoder layers, the output matrix of the deformable attention module layer that embeds positional information from the previous decoder layer is used; QP (for the first decoder layer, the initial query matrix Q embeds positional information is used) is used. init For other decoder layers, the output matrix of the deformable attention module layer that embeds positional information in the previous decoder layer is Q (for the first decoder layer, Q is Q). init For other decoder layers, the output matrix of the deformable attention module layer of the previous decoder layer is used as the query matrix, key matrix, and value matrix, respectively, where QP = Q + P, and P is the learnable position embedding. The output from the self-attention module layer is represented as QE, which can be obtained as follows:

[0089]

[0090] in, Let α represent the i-th row vector in the output matrix. ij q represents the normalized result of the dot product of the i-th row vector of the input query matrix of the self-attention module layer and the j-th row vector of the input key matrix of the self-attention module layer. j Let represent the j-th row vector in the initial query matrix or the output matrix of the deformable attention module layer of the previous decoder layer, where N represents the number of keypoints. If the input query matrix Q, key matrix K, and value matrix V are all N×C matrices, then:

[0091]

[0092] α ij Then d represents the value of the (i,j)th element in the L×L matrix α. k It is the dimension of the row vectors of the key matrix.

[0093] The sofmax operation is an existing technology, and its meaning is as follows:

[0094]

[0095] The softmax operation normalizes all input values ​​to the range (0,1) and ensures that the sum of all inputs is 1. The denominator in the formula represents the sum of the exponents of all inputs, and the numerator represents the exponent of a certain value.

[0096] Each deformable attention module layer of the decoder layer obtains updated features of the keypoints based on the output matrix of the self-attention module layer, the memory feature matrix, and the keypoint coordinate matrix predicted by the previous decoder layer (or the initial keypoint coordinate matrix for the first decoder layer), which is the output matrix of the deformable attention module layer. The initial keypoint coordinate matrix is ​​obtained by performing a fully connected operation on the initial query matrix. Those skilled in the art should understand that the keypoint coordinates and keypoint coordinate matrix described herein have the same or similar meanings.

[0097] As an example, the output matrix of the deformable attention module layer can be obtained as follows:

[0098]

[0099] Among them, f i β represents the updated feature of the i-th key point. ik x represents the attention weights obtained by performing fully connected and softmax operations on the query matrix input to the deformable attention module layer. ikThis represents the feature in the memory feature matrix corresponding to the coordinates of the k-th reference point. The positional offset between the coordinates of the k-th reference point and the coordinates of the i-th keypoint predicted by the previous decoder layer is obtained by performing a fully connected operation on the query matrix input to the deformable attention module layer. Here, K is a preset value. In other words, the coordinates of the k-th reference point are obtained by adding the positional offset to the coordinates of the i-th keypoint, and the parameter matrix of the fully connected operation is related to K. Specifically, β... ik Specifically, it can be obtained as follows:

[0100] β i =Softmax(W k QE i (5)

[0101] β ik For β i The k-th element in, where QE i W represents the C-th dimension row vector of the input query matrix of the deformable attention module layer. K A matrix of size K×C, representing the expression for QE i The fully connected parameter matrix for performing the fully connected operation is a learnable matrix. As mentioned above, W is first obtained. K With QE i The attention weight β is obtained by normalizing the dot product of the two products using a softmax operation. ik .

[0102] x ik This indicates that the initial keypoint coordinates (for the first decoder layer) or the element p in the keypoint coordinate matrix predicted by the previous decoder layer are used. i (i.e., the coordinates of the i-th keypoint predicted by the previous layer decoder) and the position offset Δp ik The coordinates obtained by addition are used as coordinate indices in the value matrix M to obtain features (for example, based on the obtained coordinates, the features corresponding to the coordinates in M ​​are determined). Since k takes values ​​of 1, ..., K, K features can be indexed in M ​​for each key point.

[0103] Wherein, the position offset Δp ik This represents the k-th reference point among the K reference points obtained by fully connecting the input query matrix (its coordinates are the coordinates of the i-th reference point plus the position offset Δp). ik The relative offset between the position of the i-th keypoint and the position of the i-th keypoint can be obtained by the following formula:

[0104] Δp i =W′ K QE i (6)

[0105] Δp ik Indicates Δp i The k-th element in the array, where k = 1, ..., K.

[0106] QE i Let W′ represent the C-th dimension row vector of the input query matrix. K Represents a 2K×C matrix indicating the QE i The parameter matrix of the fully connected operation is a learnable matrix, where 2 indicates that each position consists of two values: an x-coordinate and a y-coordinate.

[0107] K represents the number of reference points required for each key point, and its size can be preset.

[0108] To better understand this disclosure, we will take the third element in the coordinates predicted by the previous decoder layer as an example, and set K=4.

[0109] The third element is the coordinate p3 of the third keypoint predicted by the previous decoder layer (when the current decoder layer is the first decoder layer, p3 indicates the initial coordinates of the third keypoint, which are obtained by querying the initial query matrix Q). init (Obtained by performing a fully connected operation). Based on a preset K, through QE... i Performing a full connection operation yields Δp3, which consists of four elements: Δp 31 Δp 32 Δp 33 Δp 34 p3+Δp 31 Get x 31 The coordinates of the first reference point are used to determine the element x in the memory feature matrix. 31 That is, the coordinates in the memory feature matrix are p3+Δp 31 The elements, similarly, p3+Δp 32 p3+Δp 33 p3+Δp 34 Get x 32 x 33 x 34 The coordinates of the corresponding second, third, and fourth reference points are used to determine the element x in the memory feature matrix. 32 x 33 and x i34 Ultimately, f3, the updated feature of the third keypoint, can be obtained. The updated features of other keypoints can be obtained in the same way. In other words, the output matrix of the deformable attention module layer represents the updated features of the keypoints in the face image.

[0110] As described above, the deformable attention module layer uses QE as the query matrix and the memory feature matrix M as the value matrix. Instead of calculating the relationship between each element of QE and M, it only focuses on a small set of features obtained by sampling M based on reference points (i.e., the initial keypoint coordinate matrix or the keypoint coordinate matrix predicted by the previous decoder layer). (For example, when calculating the features of the i-th keypoint, the features of the K points near the i-th keypoint are used.)

[0111] As an example, after obtaining the output matrix of the deformable module layer, the offset y of the keypoint coordinates predicted by the current decoder layer relative to the keypoint coordinates predicted by the previous decoder layer can be obtained through the keypoint coordinate prediction layer of each decoder layer. O In other words, by using QD as the input to the perception layer, the output is y. O .

[0112] As an example, a three-layer fully connected network with ReLU activation functions can be used. The first two layers consist of a linear fully connected layer followed by a ReLU activation function, and the last layer directly outputs the coordinate offset information through a fully connected layer without adding a ReLU activation function. For example... Figure 3 The Multilayer Perceptron (MLP) in this model takes QD as input and outputs coordinate-related information. The formula for the ReLU activation function is:

[0113] ReLU(x) = max(0,x) (7)

[0114] In obtaining y O Then, the keypoint coordinates predicted by the current decoder layer can be obtained as follows:

[0115] y = σ(y O +σ -1 (y R (8)

[0116] Where y represents the keypoint coordinates predicted by the current decoder layer, y R This represents the initial keypoint coordinate matrix (for the first decoder layer) or the keypoint coordinates predicted by the previous decoder layer, y O This represents the output of the perception layer of the keypoint coordinate prediction layer, indicating y relative to y R The relative offset, where the input of the perception layer is the output matrix of the deformable attention module layer.

[0117] Wherein, the σ function is existing technology, and the specific formula for the σ function is:

[0118]

[0119] Finally, the keypoint coordinates predicted by the last decoder layer are determined as the final predicted keypoint coordinates.

[0120] To better understand this disclosure, a model with three decoder layers will be used for illustration.

[0121] For the first decoder layer, the query matrix, key matrix, and value matrix of the self-attention module layer are Q, which are embedded with positional information, respectively. init Q, which embeds location information init and Q init .

[0122] For the second decoder layer, the query matrix, key matrix, and value matrix of the self-attention module layer are respectively the QE of the first decoder layer with embedded location information, the QE of the first decoder layer with embedded location information, and the QE of the first decoder layer.

[0123] For the third decoder layer, the query matrix, key matrix, and value matrix of the self-attention module layer are respectively the QE of the second decoder layer with embedded position information, the QE of the second decoder layer with embedded position information, and the QE of the second decoder layer.

[0124] The first decoder layer predicts keypoint coordinates using the initial keypoint coordinate matrix; the second decoder layer predicts keypoint coordinates using the keypoint coordinates predicted by the first decoder layer; and the third decoder layer predicts keypoint coordinates using the keypoint coordinates predicted by the second decoder layer. Those skilled in the art should understand that a model with three decoder layers is merely an example; a model may have only one decoder layer or more decoder layers, and the other decoder layers besides the first decoder layer may have similar input-output operations.

[0125] As an example, a model can be trained using the L1 norm loss function (representing the absolute value of the difference between the predicted and true keypoint coordinates of the training image samples) between the predicted keypoint coordinates and the true keypoint coordinates of the training image samples. The regression loss function used in training is L... reg as follows:

[0126]

[0127] Among them, y l This represents the keypoint coordinates of the training image samples predicted by each decoder layer. L represents the actual coordinates of the keypoints in the training image samples. d is the number of decoder layers, and l is the index of the decoder layer.

[0128] The method for predicting facial landmark coordinates according to embodiments of the present disclosure can train the entire prediction model end-to-end.

[0129] Reference above Figures 1-3The method for predicting facial landmarks has been explained in the following instructions. Figures 4-5 An apparatus for predicting facial landmarks according to an embodiment of the present disclosure will be described.

[0130] Reference Figure 4 The apparatus 400 for predicting facial landmarks according to embodiments of the present disclosure may include an encoder 401 and a decoder 402. Those skilled in the art will understand that the apparatus 400 may additionally include other components, and the components included in the apparatus 400 may be separated or combined.

[0131] As an example, encoder 401 can be configured to: obtain multi-level feature maps of a face image through a convolutional neural network layer; fully connect the last level feature map in the multi-level feature maps through a fully connected layer to obtain an initial query matrix, wherein the initial query matrix represents the initial features of key points of the face image, wherein the number of elements in the initial query matrix is ​​equal to the number of key points of the face image; and obtain a memory feature matrix by flattening and connecting the multi-level feature maps.

[0132] As an example, the encoder can be implemented as a convolutional module, a fully connected module, and a flattening & connecting module. The convolutional module is configured to acquire multi-level feature maps of a face image, the fully connected module is configured to perform a full connection on the last level feature map in the multi-level feature maps to obtain an initial query matrix, and the flattening & connecting module is configured to obtain a memory feature matrix by performing flattening and connecting operations on the multi-level feature maps.

[0133] As an example, decoder 402 may be configured as at least one cascaded decoder layer, wherein at least one decoder layer is configured to determine key point coordinates of a face image based on a memory feature matrix and an initial query matrix received from the encoder.

[0134] As an example, each decoder layer in at least one decoder layer includes a cascaded self-attention module layer, a deformable attention module layer, and a keypoint coordinate prediction layer.

[0135] As an example, the self-attention module layer of the first decoder layer is configured to: obtain the output matrix of the self-attention module layer of the first decoder layer based on the received initial query matrix embedded with position information, the initial query matrix embedded with position information, and the initial query matrix, wherein the received initial query matrix embedded with position information, the initial query matrix embedded with position information, and the initial query matrix are respectively the query matrix, the key matrix, and the value matrix input to the self-attention module layer of the first layer.

[0136] The self-attention module layer of each decoder layer other than the first decoder layer is configured to: obtain the output matrix of the deformable attention module layer of the current decoder layer based on the output matrix of the deformable attention module layer of the cascaded previous decoder layer, the output matrix of the deformable attention module layer of the previous decoder layer with embedded position information, and the output matrix of the deformable attention module layer of the previous decoder layer with embedded position information. The output matrix of the deformable attention module layer of the previous decoder layer, the output matrix of the deformable attention module layer of the previous decoder layer with embedded position information, and the output matrix of the deformable attention module layer of the previous decoder layer with embedded position information are the value matrix, query matrix, and key matrix of the self-attention module layer of the current decoder layer.

[0137] Each deformable attention module layer of the decoder layer is configured to: obtain the output matrix of the deformable attention module layer of the current decoder layer based on the output matrix of the self-attention module layer of the current decoder layer, the memory feature matrix, and the key point coordinates predicted by the cascaded previous decoder layer. The output matrix and memory feature matrix of the self-attention module layer of the current decoder layer are the query matrix and value matrix of the deformable attention module layer of the current decoder layer, respectively. The key point coordinates predicted by the previous decoder layer of the first decoder layer are the initial key point coordinates based on the initial query matrix.

[0138] Each keypoint coordinate prediction layer is configured to obtain the keypoint coordinates of the face image predicted by the current decoder layer based on the output matrix of the deformable attention module layer of the current decoder layer, wherein the keypoint coordinates of the face image predicted by the last decoder layer are used as the keypoint coordinates of the final face image.

[0139] As an example, the self-attention module layer of each decoder layer obtains the output matrix based on formula (1).

[0140] As an example, the deformable attention module layer of each decoder layer obtains the output matrix based on formula (4).

[0141] As an example, the keypoint coordinate prediction layer of each decoder layer predicts keypoint coordinates based on formula (8).

[0142] As an example, the encoder and decoder are trained using training image samples based on the regression loss function in Equation (10).

[0143] As an example, device 400 may additionally include a fully connected layer to perform a fully connected operation on the initial query matrix to obtain the initial coordinates of key points of the face image.

[0144] Figure 5This is a block diagram illustrating the structure of an electronic device 500 for predicting key point coordinates of a face image according to another exemplary embodiment of the present disclosure.

[0145] Typically, electronic device 500 includes a processor 501 and a memory 502.

[0146] Processor 501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). Processor 501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0147] Memory 502 may include one or more computer-readable storage media, which may be non-transitory. Memory 502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 502 is used to store at least one instruction, which is executed by processor 501 to implement the method of training an echo cancellation model and / or eliminating echo delay of this disclosure.

[0148] In some embodiments, the electronic device 500 may also optionally include a peripheral device interface 503 and at least one peripheral device. The processor 501, memory 502, and peripheral device interface 503 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 503 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: a radio frequency circuit 504, a touch display screen 505, a camera 506, an audio circuit 507, a positioning component 508, and a power supply 509.

[0149] Peripheral device interface 503 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 501 and memory 502. In some embodiments, processor 501, memory 502 and peripheral device interface 503 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 501, memory 502 and peripheral device interface 503 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0150] The radio frequency (RF) circuit 504 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 504 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 504 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 504 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 504 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 504 may also include circuitry related to NFC (Near Field Communication), which is not limited in this disclosure.

[0151] Display screen 505 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 505 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 501 for processing. In this case, display screen 505 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 505, disposed on the front panel of electronic device 500; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1000 or in a folded design; in still other embodiments, display screen 505 may be a flexible display screen, disposed on a curved or folded surface of terminal 1000. Furthermore, display screen 505 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 505 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0152] The camera assembly 506 is used to acquire images or videos. Optionally, the camera assembly 506 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 506 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0153] The audio circuit 507 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 501 for processing, or input to the radio frequency circuit 504 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal 500. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 501 or the radio frequency circuit 504 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 507 may also include a headphone jack.

[0154] Positioning component 508 is used to locate the current geographic location of electronic device 500 for navigation or LBS (Location Based Service). Positioning component 508 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, Russia's Granas system, or the European Union's Galileo system.

[0155] Power source 509 is used to supply power to the various components in electronic device 500. Power source 509 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power source 509 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0156] In some embodiments, the electronic device 500 further includes one or more sensors 510. The one or more sensors 510 include, but are not limited to: an accelerometer 511, a gyroscope 512, a pressure sensor 513, a fingerprint sensor 514, an optical sensor 515, and a proximity sensor 516.

[0157] Accelerometer 511 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established by terminal 500. For example, accelerometer 511 can be used to detect the components of gravitational acceleration on the three coordinate axes. Processor 501 can control touch screen 505 to display the user interface in landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 511. Accelerometer 511 can also be used for games or for acquiring user motion data.

[0158] The gyroscope sensor 512 can detect the orientation and rotation angle of the terminal 500. The gyroscope sensor 512, in conjunction with the accelerometer sensor 511, can collect 3D motion data from the user on the terminal 500. Based on the data collected by the gyroscope sensor 512, the processor 501 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0159] The pressure sensor 513 can be disposed on the side bezel of the terminal 500 and / or on the lower layer of the touch display screen 505. When the pressure sensor 513 is disposed on the side bezel of the terminal 500, it can detect the user's grip signal on the terminal 500, and the processor 501 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 513. When the pressure sensor 513 is disposed on the lower layer of the touch display screen 505, the processor 501 can control the operable controls on the UI based on the user's pressure operation on the touch display screen 505. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0160] The fingerprint sensor 514 is used to collect a user's fingerprint. The processor 501 identifies the user based on the fingerprint collected by the fingerprint sensor 514, or vice versa. When the user's identity is verified as trusted, the processor 501 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 514 can be located on the front, back, or side of the electronic device 500. When the electronic device 500 has a physical button or manufacturer logo, the fingerprint sensor 514 can be integrated with the physical button or manufacturer logo.

[0161] An optical sensor 515 is used to collect ambient light intensity. In one embodiment, the processor 501 can control the display brightness of the touch screen 505 based on the ambient light intensity collected by the optical sensor 515. Specifically, when the ambient light intensity is high, the display brightness of the touch screen 505 is increased; when the ambient light intensity is low, the display brightness of the touch screen 505 is decreased. In another embodiment, the processor 501 can also dynamically adjust the shooting parameters of the camera assembly 506 based on the ambient light intensity collected by the optical sensor 515.

[0162] The proximity sensor 516, also known as a distance sensor, is typically located on the front panel of the electronic device 500. The proximity sensor 516 is used to detect the distance between the user and the front of the electronic device 500. In one embodiment, when the proximity sensor 516 detects that the distance between the user and the front of the device 500 is gradually decreasing, the processor 501 controls the touchscreen display 505 to switch from a screen-on state to a screen-off state; when the proximity sensor 516 detects that the distance between the user and the front of the electronic device 500 is gradually increasing, the processor 501 controls the touchscreen display 505 to switch from a screen-off state to a screen-on state.

[0163] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the electronic device 500, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0164] According to embodiments of this disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, they cause at least one processor to perform a method for predicting key point coordinates of a face image according to this disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R, BD-R The computer program can be stored in a computer-readable storage medium such as a BD-RE, Blu-ray or optical disc storage device, hard disk drive (HDD), solid-state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card, or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, or any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0165] According to embodiments of this disclosure, a computer program product may also be provided, wherein instructions in the computer program product are executable by a processor of a computer device to implement the method for predicting key point coordinates of a face image as described above.

[0166] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A method for predicting key point coordinates in a face image, comprising: Multi-level feature maps of face images are obtained through convolutional neural network layers; The last level feature map in the multi-level feature map is fully connected through a fully connected layer to obtain an initial query matrix, wherein the initial query matrix represents the initial features of the key points of the face image, and the number of elements in the initial query matrix is ​​equal to the number of key points of the face image. The memory feature matrix is ​​obtained by flattening and connecting multi-level feature maps; The memory feature matrix and the initial query matrix are input into at least one cascaded decoder layer to determine the key point coordinates of the face image.

2. The method as described in claim 1, in, Each decoder layer consists of a cascaded self-attention module layer, a deformable attention module layer, and a keypoint coordinate prediction layer. The step of determining the coordinates of key points in a face image includes: The initial query matrix with embedded location information, the initial query matrix with embedded location information, and the initial query matrix are used as the query matrix, key matrix, and value matrix of the self-attention module layer of the first decoder layer and input into the self-attention module layer of the first decoder layer; The output matrix, memory feature matrix, and keypoint coordinates predicted by the previous decoder layer of the current decoder layer are input into the deformable attention module layer of the current decoder layer to obtain the output matrix of the deformable attention module layer. The output matrix and memory feature matrix of the self-attention module layer are the query matrix and value matrix of the deformable attention module layer, respectively. The keypoint coordinates predicted by the previous decoder layer of the deformable attention module layer of the first decoder layer are the initial keypoint coordinates obtained based on the initial query matrix. The output matrix of the deformable attention module layer of the current decoder layer, the output matrix of the deformable attention module layer with embedded position information, and the output matrix of the deformable attention module layer with embedded position information are used as the value matrix, query matrix, and key matrix of the self-attention module layer of the cascaded next decoder layer and input to the self-attention module layer of the next decoder layer. The output matrix of the deformable attention module layer of the current decoder layer is input into the key point coordinate prediction layer of the current decoder layer to obtain the key point coordinates of the face image predicted by the current decoder layer, where the key point coordinates of the face image predicted by the last decoder layer are used as the key point coordinates of the final face image.

3. The method as described in claim 2, wherein, The output matrix of the self-attention module layer of each decoder layer is obtained by the following formula: in, This represents the i-th row vector in the output matrix. This represents the attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer. This represents the j-th row vector in the initial query matrix or the output matrix of the deformed attention layer of the previous decoder layer, and N represents the number of key points in the face image.

4. The method of claim 3, wherein, The output matrix of the deformable attention module layer of each decoder layer is obtained by the following formula: in, The feature representing the update of the i-th key point, This represents the attention weights obtained by performing fully connected and softmax operations on the query matrix input to the deformable attention module layer. The feature in the memory feature matrix corresponding to the coordinates of the k-th reference point is represented by the positional offset between the coordinates of the k-th reference point and the coordinates of the i-th keypoint in the keypoint coordinates predicted by the previous decoder layer. This offset is obtained by performing a fully connected operation on the query matrix input to the deformable attention module layer. Here, K is a preset value.

5. The method of claim 2, wherein, The keypoint coordinates predicted by each decoder layer are obtained using the following formula: in, This represents the Sigmoid function, where y represents the keypoint coordinates predicted by the current decoder layer. This represents the keypoint coordinates predicted by the previous decoder layer or the initial keypoint coordinates. This represents the output of the keypoint coordinate prediction layer, indicating y relative to... The offset.

6. The method of claim 2, wherein the convolutional network layers, fully connected layers, and at least one decoder layer are obtained by training on training image samples based on the following regression loss function: in, This represents the keypoint coordinates of the training image samples predicted by each decoder layer. This represents the actual keypoint coordinates of the training image samples. The number of decoder layers, .

7. An apparatus for predicting the coordinates of key points in a face image, comprising: The encoder is configured as follows: Multi-level feature maps of face images are obtained through convolutional neural network layers. An initial query matrix is ​​obtained by fully connecting the last-level feature map in the multi-level feature map through a fully connected layer. This initial query matrix represents the initial features of the key points in the face image, and the number of elements in the initial query matrix is ​​equal to the number of key points in the face image. The memory feature matrix is ​​obtained by flattening and concatenating multi-level feature maps; and The decoder includes at least one cascaded decoder layer, wherein the at least one decoder layer is configured to determine key point coordinates of a face image based on a memory feature matrix and an initial query matrix received from the encoder.

8. The apparatus as claimed in claim 7, in, Each decoder layer consists of a cascaded self-attention module layer, a deformable attention module layer, and a keypoint coordinate prediction layer. The self-attention module layer of the first decoder layer is configured to: obtain the output matrix of the self-attention module layer of the first decoder layer based on the received initial query matrix embedded with position information, the initial query matrix embedded with position information, and the initial query matrix, wherein the received initial query matrix embedded with position information, the initial query matrix embedded with position information, and the initial query matrix are respectively the query matrix, the key matrix, and the value matrix input to the self-attention module layer of the first layer. Each self-attention module layer of the decoder layer other than the first decoder layer is configured to: obtain the output matrix of the deformable attention module layer of the current decoder layer based on the output matrix of the deformable attention module layer of the cascaded previous decoder layer, the output matrix of the deformable attention module layer of the previous decoder layer embedded with position information, and the output matrix of the deformable attention module layer of the previous decoder layer embedded with position information. Here, the output matrix of the deformable attention module layer of the previous decoder layer, the output matrix of the deformable attention module layer of the previous decoder layer embedded with position information, and the output matrix of the deformable attention module layer of the previous decoder layer embedded with position information are the value matrix, query matrix, and key matrix of the self-attention module layer of the current decoder layer. The deformable attention module layer of each decoder layer is configured to: obtain the output matrix of the deformable attention module layer of the current decoder layer based on the output matrix of the self-attention module layer of the current decoder layer, the memory feature matrix, and the key point coordinates predicted by the cascaded previous decoder layer. The output matrix and memory feature matrix of the self-attention module layer of the current decoder layer are the query matrix and value matrix of the deformable attention module layer of the current decoder layer. The key point coordinates predicted by the previous decoder layer of the first decoder layer are the initial key point coordinates obtained based on the initial query matrix. Each keypoint coordinate prediction layer is configured to obtain the keypoint coordinates of the face image predicted by each decoder layer based on the output matrix of the deformable attention module layer of the current decoder layer. In this process, the key point coordinates of the face image predicted by the last decoder layer are used as the key point coordinates of the final face image.

9. The apparatus of claim 8, wherein, Each decoder layer's self-attention module layer obtains its output matrix based on the following formula: in, This represents the i-th row vector in the output matrix. This represents the attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module layer and the j-th row vector of the key matrix input to the self-attention module layer. This represents the j-th row vector in the initial query matrix or the output matrix of the deformed attention layer of the previous decoder layer, and N represents the number of key points in the face image.

10. The apparatus of claim 9, wherein, Each deformable attention module layer of the decoder layer obtains its output matrix based on the following formula: in, The feature representing the update of the i-th key point, This represents the attention weights obtained by performing fully connected and softmax operations on the query matrix input to the deformable attention module layer. The feature in the memory feature matrix corresponding to the coordinates of the k-th reference point is represented by the positional offset between the coordinates of the k-th reference point and the coordinates of the i-th keypoint in the keypoint coordinates predicted by the previous decoder layer. This offset is obtained by performing a fully connected operation on the query matrix input to the deformable attention module layer. Here, K is a preset value.

11. The apparatus of claim 8, wherein, Each decoder layer's keypoint coordinate prediction layer predicts keypoint coordinates based on the following formula: in, This represents the Sigmoid function, where y represents the keypoint coordinates predicted by the current decoder layer. This represents the keypoint coordinates predicted by the previous decoder layer or the initial keypoint coordinates. This represents the output of the keypoint coordinate prediction layer, indicating y relative to... The offset.

12. The apparatus of claim 8, wherein the encoder and decoder are trained using training image samples based on the following regression loss function: in, This represents the keypoint coordinates of the training image samples predicted by each decoder layer. This represents the actual keypoint coordinates of the training image samples. The number of decoder layers, .

13. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the method for predicting key point coordinates of a face image as described in any one of claims 1-6.

Images