Method and device for detecting key points of face image
Through the combination of convolutional neural network and decoder layer, a model detects key points styles of multiple face images is realized, solving the problem of insufficient model diversity and adaptability in the existing technology, and reducing training costs and memory consumption.
Patent Information
- Application Number
- CN202411289274.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-13
- Publication Date
- 2024-12-03
AI Technical Summary
Existing face image keypoint detection methods require training multiple models to detect keypoints of different styles or types, resulting in wasted time and memory, and models of different datasets are difficult to adapt and cannot be applied well to other datasets.
The multi-level feature map is obtained through the convolution neural network layer, the initial query matrix is obtained using the full connection layer, and the memory feature matrix is obtained through flattening and connecting operations. The cascading decoder layer determines the face based on the key point style of the user input. The coordinates of the corresponding key point of the image enable a model to detect key points in multiple styles.
A model detects key points in multiple styles is implemented, reducing the time and memory waste caused by training multiple models, and improving the adaptability of the model, and can be applied between different data sets.
Smart Images

Figure CN119068534A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of facial landmark detection, and more specifically, to a method and device for detecting the coordinates of landmarks in a facial image. Background Art
[0002] In recent years, due to the rapid development of deep neural networks, the technology of landmark detection in face images has made significant progress. Existing methods for landmark detection in face images mainly include methods based on heatmap regression and methods based on coordinate regression. The method based on heatmap regression first generates a heatmap based on the given key point coordinates, where each heatmap represents the probability of a key point position, and then obtains the key point according to the peak probability position on the heatmap. The heatmap-based method can preserve the spatial structure of image features, thereby achieving good performance. The coordinate regression-based method directly maps the input image to the key point coordinates. In the deep learning framework, the input image is input into the CNN model to obtain image features. Then the features are directly mapped to coordinates through a fully connected prediction layer. Recently, graph neural networks and transformers have been used to learn the key point structure in face images to improve detection accuracy.
[0003] However, existing methods can only detect or predict key points of one style or type through one model. In order to obtain key points of different styles or types, different models need to be trained, which leads to a waste of time and memory. In addition, different datasets have different annotation types. A model trained on one dataset cannot be well applied to other datasets. Summary of the invention
[0004] The object of the present invention is to provide a method and device for detecting the coordinates of key points of a face image, so as to at least solve the problems in the above-mentioned related technologies, or not solve any of the above-mentioned problems.
[0005] According to one aspect of an exemplary embodiment of the present disclosure, a method for detecting key points of a face image is provided, comprising: obtaining a multi-level feature map of the face image through a convolutional neural network layer; fully connecting the last level feature map of the multi-level feature map through a fully connected layer to obtain an initial query matrix; obtaining a memory feature matrix by flattening and connecting the multi-level feature map; and determining the coordinates of a first key point of the face image corresponding to the at least one key point style based on a user input for specifying at least one key point style among a plurality of key point styles, the memory feature matrix and the initial query matrix using at least one cascaded decoder layer.
[0006] According to the embodiments of the present disclosure, one model can be used to detect the coordinates of key points of at least one style among multiple styles specified by a user. Compared with the prior art in which a model can only detect one type of key points, the technology of the present application can reduce the waste of time and memory caused by training multiple different models.
[0007] Optionally, each decoder layer includes a cascaded mask module, a self-attention module, a deformable attention module and a key point coordinate prediction block, and wherein the step of determining the coordinates of the first key point includes: based on at least one key point style among the multiple key point styles specified by the user input, obtaining a mask matrix corresponding to the at least one key point; based on the mask matrix, masking the query matrix, key matrix and value matrix input to the self-attention module for predicting the coordinates of the first key point.
[0008] Optionally, the number of elements in the initial query matrix is equal to N, where N indicates the sum of the number of key points corresponding to each key point style in the multiple key point styles; wherein the step of determining the coordinates of the first key point includes: masking the initial query matrix and the position information corresponding to the initial query matrix using a mask module of the first decoder layer based on the mask matrix to set some elements in the initial query matrix and the position information to zero, wherein the some elements correspond to key points other than the first key point in the key points corresponding to the multiple key point styles; and masking the masked initial query matrix embedded with the masked position information, the masked initial query matrix embedded with the masked position information, and the masked initial query The query matrix is input into the self-attention module of the first decoder layer as the query matrix, key matrix and value matrix of the self-attention module of the first decoder layer; the output matrix of the self-attention module of the current decoder layer, the memory feature matrix and the coordinates of the first key point predicted by the previous decoder layer after masking are input into the deformable attention module of the current decoder layer to obtain the output matrix of the deformable attention module of the current decoder layer, wherein the output matrix of the self-attention module of the current decoder layer and the memory feature matrix are the query matrix and value matrix of the deformable attention module of the current decoder layer, wherein the coordinates of the first key point predicted by the previous decoder layer after masking input into the deformable attention module of the first decoder layer are The key point coordinates are obtained by masking the initial key point coordinates obtained based on the initial query matrix based on the mask matrix, wherein the coordinates of the first key point predicted by the previous decoder layer after masking are obtained by setting some elements in the coordinates of the first key point predicted by the previous decoder layer to zero based on the mask matrix by the mask module, wherein the some elements correspond to key points other than the first key point among the key points corresponding to the styles of the multiple key points; based on the mask matrix, the mask module of the next decoder layer of the current decoder layer is used to mask the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer so that the current decoder The output matrix of the deformable attention module of the decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer are set to zero, wherein the partial elements correspond to the key points other than the first key point among the key points corresponding to the styles of the multiple key points; the masked output matrix of the deformable attention module of the current decoder layer, the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information, and the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information are input as the value matrix, query matrix and key matrix of the self-attention module of the next decoder layer to the self-attention module of the next decoder layer;And inputting the output matrix of the deformable attention module of the current decoder layer and the coordinates of the first key point predicted by the previous decoder layer after masking into the key point coordinate prediction module of the current decoder layer to obtain the coordinates of the first key point predicted by the current decoder layer, wherein the coordinates of the first key point predicted by the last decoder layer in the at least one decoder layer are used as the final coordinates of the first key point. ;
[0009] According to an embodiment of the present disclosure, by performing a mask operation, the coordinates of key points of a user-specified style may be acquired.
[0010] Optionally, the output matrix QE of the self-attention module of each decoder layer is obtained by the following formula:
[0011]
[0012] in, represents the i-th row vector in the output matrix QE, α ij represents the attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module and the j-th row vector of the key matrix input to the self-attention module, q j Represents the j-th row vector in the output matrix of the deformable attention module of the previous decoder layer after masking.
[0013] According to an embodiment of the present disclosure, the self-attention module learns the structural dependencies between key points. This information is important for key point positioning, thereby improving the prediction accuracy of key point coordinates
[0014] Optionally, the output matrix QD of the deformable attention module of each decoder layer is obtained by the following formula:
[0015]
[0016] Among them, f i represents the updated features of the i-th key point in QD, β ik represents the attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the deformable attention module, x ik Represents the feature corresponding to the k-th reference point coordinate in the memory feature matrix, wherein the position offset between the k-th reference point coordinate and the i-th key point coordinate in the coordinate of the first key point predicted by the previous decoder layer after masking is obtained by performing a fully connected operation on the query matrix input to the deformable attention module, wherein K is a preset value.
[0017] Instead of calculating the relationship between each element in the output matrix of the self-attention module and the memory feature matrix, the deformable attention module only focuses on a small set of features obtained by sampling the memory features according to the key point coordinates predicted by the previous decoder layer, which greatly reduces the area of feature search and thus speeds up the convergence of the model.
[0018] Optionally, the coordinates of the first keypoint predicted by each decoder layer are obtained by the following formula:
[0019] y=σ(y O +σ -1 (y R ))
[0020] Among them, y represents the coordinates of the first key point predicted by the current decoder layer, y R Represents the coordinates of the first key point predicted by the previous decoder layer after masking, y O Indicates y relative to y R offset.
[0021] According to one aspect of an exemplary embodiment of the present disclosure, there is provided a device for detecting key points of a face image, comprising: an encoder, configured to obtain a multi-level feature map of the face image through a convolutional neural network layer, fully connect the last level feature map of the multi-level feature map through a fully connected layer to obtain an initial query matrix, and obtain a memory feature matrix by flattening and connecting the multi-level feature map; a decoder, comprising at least one cascaded decoder layer, wherein the at least one decoder layer is configured to determine the coordinates of the key points of the face image corresponding to at least one key point style of the multiple key point styles based on a user input for specifying the at least one key point style of the multiple key point styles, the memory feature matrix and the initial query matrix.
[0022] Optionally, each decoder layer includes a cascaded mask module, a self-attention module, a deformable attention module and a key point coordinate prediction block, and wherein the decoder is configured to: obtain a mask matrix corresponding to at least one key point based on at least one key point style among the multiple key point styles specified by the user input; and based on the mask matrix, mask the query matrix, key matrix and value matrix input to the self-attention module for predicting the coordinates of the first key point.
[0023] Optionally, the number of elements in the initial query matrix is equal to N, where N indicates the sum of the number of key points corresponding to each key point style in the multiple key point styles; wherein the decoder is configured to: mask the initial query matrix and the position information corresponding to the initial query matrix using a mask module of the first decoder layer based on the mask matrix, so as to set some elements in the initial query matrix and the position information to zero, wherein the some elements correspond to key points other than the first key point in the key points corresponding to the multiple key point styles; and perform masking on the masked initial query matrix embedded with the masked position information, the masked initial query matrix embedded with the masked position information, and the masked initial query matrix. The query matrix, key matrix and value matrix of the self-attention module of the first decoder layer are input into the self-attention module of the first decoder layer; the output matrix of the self-attention module of the current decoder layer, the memory feature matrix and the coordinates of the first key point predicted by the previous decoder layer after masking are input into the deformable attention module of the current decoder layer to obtain the output matrix of the deformable attention module of the current decoder layer, wherein the output matrix of the self-attention module of the current decoder layer and the memory feature matrix are the query matrix and value matrix of the deformable attention module of the current decoder layer, wherein the coordinates of the first key point predicted by the previous decoder layer after masking input into the deformable attention module of the first decoder layer are based on the The mask matrix masks the initial key point coordinates obtained based on the initial query matrix to obtain the key point coordinates, wherein the masked coordinates of the first key point predicted by the previous decoder layer are obtained by setting some elements in the coordinates of the first key point predicted by the previous decoder layer to zero based on the mask matrix by the mask module, wherein the some elements correspond to the key points other than the first key point among the key points corresponding to the styles of the multiple key points; based on the mask matrix, the mask module of the next decoder layer of the current decoder layer is used to mask the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer so that the current decoder The output matrix of the deformable attention module of the layer and some elements in the position information corresponding to the output matrix of the deformable attention module of the current decoder layer are set to zero, wherein the some elements correspond to the key points other than the first key point among the key points corresponding to the styles of the multiple key points; the masked output matrix of the deformable attention module of the current decoder layer, the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information, and the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information are input as the value matrix, query matrix and key matrix of the self-attention module of the next decoder layer to the self-attention module of the next decoder layer;And inputting the output matrix of the deformable attention module of the current decoder layer and the coordinates of the first key point predicted by the previous decoder layer after masking into the key point coordinate prediction module of the current decoder layer to obtain the coordinates of the first key point predicted by the current decoder layer, wherein the coordinates of the first key point predicted by the last decoder layer in the at least one decoder layer are used as the final coordinates of the first key point. ;
[0024] Optionally, the output matrix QE of the self-attention module of each decoder layer is obtained by the following formula:
[0025]
[0026] in, represents the i-th row vector in the output matrix QE, α ij represents the attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module and the j-th row vector of the key matrix input to the self-attention module, q j Represents the j-th row vector in the output matrix of the deformable attention module of the previous decoder layer after masking.
[0027] Optionally, the output matrix QD of the deformable attention module of each decoder layer is obtained by the following formula:
[0028]
[0029] Among them, f i represents the updated features of the i-th key point in QD, β ik represents the attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the deformable attention module, x ik Represents the feature corresponding to the k-th reference point coordinate in the memory feature matrix, wherein the position offset between the k-th reference point coordinate and the i-th key point coordinate in the coordinate of the first key point predicted by the previous decoder layer after masking is obtained by performing a fully connected operation on the query matrix input to the deformable attention module, wherein K is a preset value.
[0030] Optionally, the coordinates of the first keypoint predicted by each decoder layer are obtained by the following formula:
[0031] y=σ(y O +σ -1 (y R ))
[0032] Among them, y represents the coordinates of the first key point predicted by the current decoder layer, y R Represents the coordinates of the first key point predicted by the previous decoder layer after masking, yO Indicates y relative to y R offset.
[0033] According to an embodiment of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the method for detecting key point coordinates of a face image as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The above and other objects and features of the present invention will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate an example, in which:
[0035] Figure 1 A flowchart of a method for predicting facial key point coordinates according to an embodiment of the present disclosure is shown;
[0036] Figure 2A and Figure 2B is a flowchart showing a method for predicting the coordinates of key points of a human face according to an embodiment of the present disclosure;
[0037] Figure 3 is a diagram showing an example of the structure of the first decoder layer,
[0038] Figure 4 is a block diagram showing the structure of an apparatus for predicting key point coordinates of a face image according to an embodiment of the present disclosure;
[0039] Figure 5 is a block diagram showing the structure of an apparatus for predicting key point coordinates of a face image according to another embodiment of the present disclosure; and
[0040] Figure 6 is a block diagram illustrating a structure of an electronic device 600 for predicting key point coordinates of a face image according to another exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0041] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same or similar elements, features and structures. However, it is not intended that the present disclosure be limited to specific embodiments by the various embodiments described herein, and it is intended that: the present disclosure covers all modifications, equivalents and / or substitutes of the present disclosure, as long as they are within the scope of the attached claims and their equivalents. The terms and words used in the following specification and claims are not limited to their dictionary meanings, but are only used to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only, and not for the purpose of limiting the present disclosure defined by the attached claims and their equivalents.
[0042] It should be understood that the singular includes the plural unless the context clearly indicates otherwise.The terms "include", "comprising" and "having" used herein indicate the presence of disclosed functions, operations or elements, but do not exclude other functions, operations or elements.
[0043] For example, the expression "A or B" or "at least one of A and / or B" may indicate A and B, A or B. For example, the expression "A or B" or "at least one of A and / or B" may indicate (1) A, (2) B, or (3) both A and B.
[0044] In various embodiments of the present disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to another component (e.g., a second component) or being "coupled" or "connected" to another component (e.g., the second component), the component may be directly connected to the other component or may be connected through another component (e.g., a third component). In contrast, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to another component (e.g., the second component) or being directly coupled to or directly connected to another component (e.g., the second component), there is no other component (e.g., the third component) between the component and the other component.
[0045] The expression "configured to" used in describing various embodiments of the present disclosure may be used interchangeably with expressions such as "suitable for", "having the ability to", "designed to", "suitable for", "manufactured to", and "capable of", for example, depending on the circumstances. The term "configured to" may not necessarily indicate that it is "specially designed to" in terms of hardware. On the contrary, the expression "a device configured to..." in some cases may indicate that the device and another device or part "can..." For example, the expression "a processor configured to perform A, B, and C" may indicate a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit CPU or an application processor (AP)) for performing the corresponding operations by executing at least one software program stored in a memory device.
[0046] The terms used herein are to describe certain embodiments of the present disclosure, but are not intended to limit the scope of other embodiments. Unless otherwise noted herein, all terms used herein (including technical or scientific terms) may have the same meaning as those generally understood by those skilled in the art. Typically, the terms defined in the dictionary should be considered to have the same meaning as the contextual meaning in the relevant field, and, unless clearly defined herein, should not be understood differently or understood to have an overly formal meaning. In any case, the terms defined in the present disclosure are not intended to be interpreted as excluding embodiments of the present disclosure.
[0047] As understood by those skilled in the art, key point detection can be formulated as N coordinate detection or prediction problems, where N is the number of facial key points.
[0048] As is understood by those skilled in the art, the key point style or type described herein indicates how many key points are used to annotate a face image, and where the key points are annotated in the face image.
[0049] Figure 1 Examples of 4 key point style annotations are shown. Figure 1 ,Model 1 can detect 19 key points, Model 2 can detect 29 key points, Model 3 can detect 68 key points, and Model 4 can detect 98 key points.
[0050] In the prior art, a key point prediction model can only predict key points of one key point style or type. For example, model 1 can only detect 98 key points and model 2 can only detect 68 key points. Therefore, in order to detect 98 points and 68 points, model 1 and model 2 need to be trained respectively.
[0051] According to the embodiment of the present disclosure, the method and apparatus for detecting key points of a facial image can detect key points of facial images of different types or styles through one model.
[0052] Figure 2A and Figure 2B A schematic diagram showing an example of the overall architecture of a method for detecting a facial image according to an embodiment of the present disclosure is shown.
[0053] Reference Figure 2A and Figure 2B The model of the present invention (for example, it may include a backbone network for extracting pyramid features from a user, a query matrix initialization module, a flatten and connect (Flatten & Concate) module, a decoder, etc.) outputs the coordinates of the key points of the face image corresponding to the specified key point style based on the user input of the specified key point style. The decoder may include multiple cascaded decoder layers, each decoder layer may include a cascaded first mask module, a self-attention module, a deformable attention module and a key point coordinate prediction module and a second mask module (which will be described in detail later below).
[0054] Those skilled in the art will appreciate that although Figure 2BEach decoder layer in shows a first mask module and a second mask module, but this is only an example and does not limit the present disclosure. For example, each decoder layer may include only one mask module for performing masking on the output matrix of the deformable attention module of the previous decoder layer of the current decoder layer (wherein the output matrix of the deformable attention module of the previous decoder layer of the first decoder layer is the initial query matrix) and the output matrix of the previous decoder layer (wherein the output matrix of the previous decoder layer of the first decoder layer is the initial key point coordinates).
[0055] As another example, each decoder layer may include three mask modules, corresponding to the self-attention module, deformable attention module and key point coordinate prediction module of each decoder layer. For example, the first mask module is used to mask the query matrix, key matrix and value matrix input to the self-attention module of the current decoder layer, the second mask module is used to mask the output matrix of the key point coordinate prediction module of the previous decoder layer (the output matrix of the key point coordinate prediction module of the previous decoder layer of the first decoder layer is the initial key point coordinate), and the masked output matrix is input to the deformable attention module of the current decoder layer, and the third mask module is also used to mask the output matrix of the key point coordinate prediction module of the previous decoder layer, and the masked output matrix is input to the key point coordinate prediction module of the current decoder layer.
[0056] Figure 3 is a flow chart showing a method for detecting key points of a human face according to an embodiment of the present disclosure. Figure 3 In step S301, a multi-level feature map of a face image (e.g. Figure 1 Those skilled in the art should understand that the extracted multi-level feature map is a pyramid feature map, in which the low-level feature map represents the local features of the image and the high-level feature map represents the global features of the image.
[0057] As an example, the pyramid features of the face image can be obtained through the backbone network to obtain feature maps of different sizes at each level. ResNet18 is used in this patent, but it is not limited to ResNet18. For example, the backbone network can be ResNet-34, ResNet-50, vgg, mobile-Net, etc.
[0058] As an example, if the given input image size is 256×256×3, the sizes of feature maps at each level can be 64×64×64, 32×32×128, 16×16×256, and 8×8×512 respectively.
[0059] In step S302, the last level feature map in the multi-level feature map (i.e., the last level feature or the highest level feature of the pyramid feature) is fully connected through the fully connected layer to obtain the initial query matrix Q init , where the initial query matrix represents the initial features of the key points of the face image.
[0060] In step S303, a memory feature matrix is obtained by performing a flattening and concatenation operation on the multi-level feature map.
[0061] Specifically, after extracting pyramid features from the input image using the backbone network, 1×1 convolutions can be applied to these feature maps to obtain feature maps with the same number of output channels. These feature maps are then flattened and concatenated (Flatten & Contate) together as the memory feature matrix M.
[0062] In step S304, based on user input for specifying at least one key point style among multiple key point styles, a memory feature matrix and an initial query matrix, the coordinates of a first key point of the face image corresponding to the at least one key point style are determined using at least one cascaded decoder layer.
[0063] As an example, each decoder layer includes a cascaded mask module, a self-attention module, a deformable attention module, and a key point coordinate prediction block, and wherein the step of determining the coordinates of the first key point includes: based on the at least one key point style of the multiple key point styles specified by the user input, obtaining a mask matrix corresponding to the at least one key point; and based on the mask matrix, masking the query matrix, key matrix, and value matrix input to the self-attention module for predicting the coordinates of the first key point. As understood by those skilled in the art, the first key point includes multiple key points.
[0064] As an example, if the user input specifies a 98-point keypoint style, then at least one decoder layer of the cascade may determine the coordinates of the 98 points at specific locations of the face image.
[0065] As another example, if the user input specifies a 98-point keypoint style and a 19-point keypoint style, at least one decoder layer of the cascade may determine the coordinates of 98 points at specific locations of a face image and the coordinates of 19 keypoints at specific locations.
[0066] As an example, the number of elements in the initial query matrix is equal to N, where N indicates the sum of the numbers of key points corresponding to each key point style in the plurality of key point styles.
[0067] For example, if the prediction model is trained to detect keypoints of any of the 19-point, 29-point, and 68-point keypoint types and / or any combination, then the value of N is 116.
[0068] As an example, Q init It can be obtained as follows:
[0069] Q init =FC(F),
[0070] Among them, F marks the feature map from the last layer of the backbone network, and its size can be recorded as (H×W)×C, where H and W are the width and height of the feature map in space, and C is the feature dimension of the feature map. FC represents the fully connected layer, which maps the spatial scale of each feature channel (H×W) into a vector of size N.
[0071] Query Matrix Q init It has a size of N×C, where N is the number of keypoints in the face image and C is the feature dimension. It is essentially a learnable matrix that will be used to extract keypoint related features and transform them into coordinates.
[0072] According to an exemplary embodiment of the present disclosure, each decoder layer may have the same structure except that an input matrix and an output matrix of each decoder layer are different.
[0073] As an example, each decoder layer includes a cascaded mask module, a self-attention module, a deformable attention module, and a key point coordinate prediction module. It should be understood by those skilled in the art that each decoder layer may also include other modules, units, or layers as needed.
[0074] As an example, the step of determining the coordinates of the first key point includes: masking the initial query matrix and the position information corresponding to the initial query matrix using a mask module of the first decoder layer based on the mask matrix to set some elements in the initial query matrix and the position information to zero, wherein the some elements correspond to other key points in the key points of the plurality of key point styles other than the first key point. As understood by those skilled in the art, the position information indicates position information for performing position embedding on the query matrix.
[0075] As an example, a masking operation may be performed using a mask matrix (Q_mask).
[0076] As an example, the mask matrix may be a variable matrix of size N, wherein the value of each element in the initial mask matrix may be 0, and the value of the element corresponding to the key point of the key point style specified by the user is set to 1 based on the user input, and the remaining elements are set to 0.
[0077] As an example, a position embedding of length N may be generated for a mask matrix.
[0078] For example, if N = n 1 +n 2 +n 3 +…n i , then n 1 、n 2 、n 3 ,…n i The number of points annotated for each key point style. If the user inputs a specified detection style2 (for example, 68 points), the mask matrix with the n 2 The elements at the positions corresponding to the key points are set to 1, and the remaining elements are set to 0.
[0079] For example, if N = 214 (i.e., 98+68+29+19), that is, the model can detect key points of 4 key point styles, then the N elements in the initial mask matrix are all 0. If the user specifies output style2, that is, 68 key point detection results, then the elements at positions 99 to 167 (99+68) in Q_mask are set to 1, and the other positions remain 0.
[0080] For example, if Q is an N*C feature, Q_mask is an N-dimensional vector in which some elements are 1 and some are 0, and Q*Qmask is executed, the elements in Q corresponding to the elements in Q_mask that are 0 can be set to 0, thereby obtaining the masked Q.
[0081] As an example, the step of determining the coordinates of the first key point may also include: inputting the masked initial query matrix embedded with the masked position information, the masked initial query matrix embedded with the masked position information, and the masked initial query matrix as the query matrix, key matrix, and value matrix of the self-attention module of the first decoder layer to the self-attention module of the first decoder layer.
[0082] As an example, the step of determining the coordinates of the first key point may also include: inputting the output matrix of the self-attention module of the current decoder layer, the memory feature matrix and the coordinates of the first key point predicted by the previous decoder layer into the deformable attention module of the current decoder layer to obtain the output matrix of the deformable attention module of the current decoder layer, wherein the output matrix and the memory feature matrix of the self-attention module of the current decoder layer are the query matrix and the value matrix of the deformable attention module of the current decoder layer, wherein the coordinates of the first key point predicted by the previous decoder layer after masking input to the deformable attention module of the first decoder layer are the key point coordinates obtained by masking the initial key point coordinates obtained based on the initial query matrix based on the mask matrix, wherein the coordinates of the first key point predicted by the previous decoder layer after masking are obtained by setting some elements in the coordinates of the first key point predicted by the previous decoder layer to zero based on the mask matrix by the mask module, wherein the some elements correspond to key points other than the first key point among the key points corresponding to the styles of the multiple key points.
[0083] As an example, a full connection operation may be performed on the query matrix to obtain the initial keypoint coordinates.
[0084] As an example, the step of determining the coordinates of the first key point may also include: masking the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer using the mask module of the next decoder layer of the current decoder layer based on the mask matrix so that some elements in the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer are set to zero, wherein the some elements correspond to key points other than the first key point among the key points corresponding to the styles of the multiple key points.
[0085] The process of performing the AND mask operation in this step is similar to the process of performing the mask operation described above, so it will not be repeated here.
[0086] As an example, the step of determining the coordinates of the first key point may also include: inputting the masked output matrix of the deformable attention module of the current decoder layer, the masked output matrix of the deformable attention module of the current decoder layer embedded with masked position information, and the masked output matrix of the deformable attention module of the current decoder layer embedded with masked position information as the value matrix, query matrix and key matrix of the self-attention module of the next decoder layer into the self-attention module of the next decoder layer; and inputting the output matrix of the deformable attention module of the current decoder layer and the masked coordinates of the first key point predicted by the previous decoder layer into the key point coordinate prediction module of the current decoder layer to obtain the coordinates of the first key point predicted by the current decoder layer, wherein the coordinates of the first key point predicted by the last decoder layer in the at least one decoder layer are used as the final coordinates of the first key point.
[0087] Those skilled in the art should understand that, although the above describes masking the position information and the query matrix separately, this is only an example and does not limit the present disclosure. For example, the position information may be first embedded in the query matrix, and then the query matrix with the embedded position information may be masked to obtain a masked query matrix with the masked position information embedded.
[0088] Figure 4 is a diagram showing an example of the structure of the first decoder layer, and the other decoder layers have the same structure.
[0089] Although Figure 4 The mask module is not shown, but does not limit the present disclosure. For example, the position information after masking, the Q after masking init The coordinates of the reference points after masking are obtained by masking the position information, Q init And the reference point coordinates (i.e., the coordinates of the first key point predicted by the previous decoder or the output of the previous decoder layer (for the first decoder, the initial key point coordinates)) are obtained by performing masking. Since the mask operation is described in detail above, the following mainly describes the self-attention module and deformable attention module included in the decoder layer.
[0090] Reference Figure 4 , each decoder layer may include a self-attention module, a deformable attention module, and a key point prediction module. The self-attention module of each decoder layer may include a self-attention unit and a residual summation and normalization unit.
[0091] The deformable attention module of each decoder layer may include a deformable-attention unit, a residual summation and normalization (Add&Norm) unit, and a feed-forward network or layer (Feed-forwardlayer, FFN) unit.
[0092] The key point coordinate prediction module of each decoder layer may include a coordinate offset unit (e.g., MLP) and a coordinate determination unit, wherein the coordinate determination unit determines the coordinates of the first key point predicted by the current decoder layer based on the coordinate offset obtained by the coordinate offset unit of the current decoder layer and the coordinates of the first key point obtained by the previous decoder layer.
[0093] The query matrix, key matrix, and value matrix of the self-attention module of the first decoder layer (i.e., the first decoder layer) are the masked Q matrix with the masked position information embedded in it. init , the masked Q with the masked position information embedded init and the masked Q init The query matrix, key matrix, and value matrix of the self-attention module of other decoder layers are respectively the output matrix of the deformable attention module of the masked previous decoder layer embedded with the masked position information, the output matrix of the deformable attention module of the masked previous decoder layer embedded with the masked position information, and the output matrix of the deformable attention layer of the masked previous decoder layer.
[0094] The self-attention module only takes the masked query matrix as input (more specifically, the masked query matrix and the masked position information as input). The query matrix will learn the structural dependencies between keypoints, which can capture poses, expressions, etc. from keypoint locations.
[0095] The self-attention module inputs QP, QP, and Q as query matrix, key matrix, and value matrix, respectively. Where QP = Q + P, P is the learnable position embedding (masked position embedding). For the first decoder layer, QP is the masked initial query matrix Q embedded with the masked position information. init , for other decoder layers QP is the output matrix of the deformable attention module of the previous decoder layer after masking with the masked position information embedded); for the first decoder layer Q is the masked Q init , for other decoder layers, Q is the output matrix of the deformable attention module of the previous decoder layer after masking. After the processing of the self-attention module, the output matrix QE can be obtained, which can be obtained by formula (1):
[0096]
[0097] in, represents the i-th row vector in the output matrix QE, α ij represents the dot product of the i-th row vector of the query matrix input to the self-attention module and the j-th row vector of the key matrix input to the self-attention module and the normalized result, q j represents the j-th row vector in the output matrix of the deformable attention module of the previous decoder layer after masking (i.e., the value matrix input to the self-attention module), and N represents the number of key points (i.e., the sum of the number of key points corresponding to each key point style in multiple key point styles).
[0098] Specifically, α can be obtained by the following formula ij ,
[0099]
[0100] α ij Then it represents the value of the (i,j)th element in the matrix α, d k is the dimension of the row vector of the key matrix, K T is the transposed matrix of K.
[0101] Among them, the sofmax operation is a prior art, and its meaning is as follows:
[0102]
[0103] The softmax operation normalizes all input values to (0,1) and ensures that the sum of all inputs is 1.
[0104] The denominator in the formula represents the sum of all input exponents, and the numerator represents the exponent of a certain value.
[0105] As an example, the coordinate matrix of the first key point predicted by the key point coordinate prediction module of the previous decoder layer (the initial key point coordinate matrix for the first decoder layer), the output matrix of the self-attention module (as a query matrix) and the memory feature matrix (as a value matrix) can be respectively input into the deformable attention module of the current decoder layer to obtain the output matrix of the deformable attention module.
[0106] The deformable attention module of each decoder layer obtains the updated features of the key points based on the output matrix of the self-attention module, the memory feature matrix, and the coordinate matrix of the first key point predicted by the previous decoder layer, that is, the output matrix of the deformable attention module. Among them, the initial key point coordinate matrix is obtained by performing a full connection operation on the initial query matrix. Those skilled in the art should understand that the key point coordinates and key point coordinate matrices described herein have the same or similar meanings.
[0107] As an example, the output matrix QD of the deformable attention module can be obtained as follows:
[0108]
[0109] Among them, f i represents the updated features of the i-th key point in QD, β ik represents the attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the deformable attention module, x ik represents the feature corresponding to the k-th reference point coordinate in the memory feature matrix, wherein the position offset between the k-th reference point coordinate and the i-th key point coordinate in the coordinate of the first key point predicted by the previous decoder layer after masking is obtained by performing a full connection operation on the query matrix input to the deformable attention module, wherein K is a preset value (those skilled in the art should understand that K in Formula 4 (which is a scalar) is independent of K in Formula (2) (which is a vector or matrix)). That is, the k-th reference point coordinate is obtained by adding the position offset to the coordinate of the i-th key point, and the parameter matrix of the full connection operation is related to K. Specifically, β ik The details can be obtained as follows:
[0110] β i =Softmax(W k QE i ) (5)
[0111] β ik β i The kth element in , where QE i represents the row vector of the i-th C dimension of the input query matrix of the deformable attention module, W K is a K×C matrix, representing the QE i The fully connected parameter matrix that performs the fully connected operation is a learnable matrix. As mentioned above, first obtain W K The attention weight β is obtained by normalizing the dot product with QE through the Softmax operation. ik .
[0112] x ik represents the initial keypoint coordinates (for the first decoder layer) or the element p in the keypoint coordinate matrix predicted by the previous decoder layer i (i.e., the coordinates of the i-th key point predicted by the previous decoder) and the position offset Δp ik The added coordinates are used as features indexed by the coordinates in the value matrix M (for example, based on the obtained coordinates, the features corresponding to the coordinates in M are determined). Since k takes the value of 1, ..., K, K features can be indexed in M for each key point.
[0113] Where, the position offset Δp ik It represents the kth reference point among the K reference points obtained by fully connecting the input query matrix (its coordinates are the coordinates of the i-th reference point plus the position offset Δp ik ) is the relative offset between the position of the i-th key point, and the position offset can be obtained by the following formula:
[0114] Δp i =W′ K QE i (6)
[0115] Δp ik Denotes Δp i The kth element in , k = 1, ..., K,
[0116] QE i Represents the row vector of the i-th C dimension of the input query matrix, W′ K Represents a 2K×C matrix, indicating the QE i The fully connected operation parameter matrix that performs the fully connected operation is a learnable matrix. 2 indicates that each position consists of two values, the horizontal coordinate and the vertical coordinate.
[0117] K represents the number of reference points required for each key point, and its size can be preset.
[0118] In order to more clearly understand the present disclosure, the third element in the coordinates predicted by the previous decoder layer is taken as an example for explanation, and K=4 is set.
[0119] The third element is the coordinate p of the third key point predicted by the previous decoder 3 (When the current decoder layer is the first decoder layer, p 3 Indicates the initial coordinates of the third key point, which is obtained by the initial query matrix Q init Based on the preset K, by performing a full connection operation on QE i Perform a full connection operation to obtain Δp 3 , Δp 3 It consists of four elements: Δp 31 , Δp 32 , Δp 33 , Δp 34 . p 3 +Δp 31 Get x 31 The coordinates of the first reference point corresponding to the memory feature matrix are used to determine the element x 31 , that is, the coordinate in the memory feature matrix is p 3 +Δp 31 Similarly, p3 +Δp 32 、p 3 +Δp 33 、p 3 +Δp 3 Get with x 32 、x 33 、x 34 The corresponding coordinates of the second reference point, the third reference point, and the fourth reference point are used to determine the element x in the memory feature matrix. 32 、x 33 and x i34 Finally, we can get f 3 , that is, the updated features of the third key point, the updated features of other key points can be obtained in the same way, that is, the output matrix of the deformable attention module represents the updated features of the key points of the face image.
[0120] As described above, the deformable attention module takes QE as the query matrix and the memory feature matrix M as the value matrix. Instead of calculating the relationship between each element of QE and M, it only focuses on a small set of features obtained by sampling M according to the reference point (i.e., the initial key point coordinate matrix or the key point coordinate matrix predicted by the previous decoder layer) (e.g., when calculating the features of the i-th key point, the features of K points near the i-th key point are used).
[0121] As an example, after obtaining the output matrix of the deformable module, the offset y of the key point coordinates predicted by the current decoder layer relative to the key point coordinates predicted by the previous decoder layer can be obtained by the coordinate offset unit (e.g., multi-layer perceptron (MLP) unit) in the key point coordinate prediction module of each decoder layer. O That is, the output matrix QD of the deformable attention module is used as the input of the coordinate offset unit, and the output is y O .
[0122] As an example, the coordinate offset unit can be implemented by a 3-layer fully connected network with a ReLU activation function, where the first two layers are composed of a linear full connection followed by a ReLU activation function, and the last layer directly outputs the coordinate deviation information (i.e., coordinate offset) through a full connection without a ReLU activation function. For example, Figure 4 The multi-layer perceptron (MLP) in takes QD as input and outputs coordinate offset information. The formula of the ReLU activation function is:
[0123] ReLU(x)=max(0,x) (7)
[0124] In obtaining y OAfter that, the coordinate determination unit of the key point coordinate prediction module can be used to obtain the key point coordinates predicted by the current decoder layer as follows:
[0125] y=σ(y O +σ -1 (y R )) (8)
[0126] Among them, y represents the coordinates of the first key point predicted by the current decoder layer, y R represents the initial keypoint coordinate matrix (for the first decoder layer) or the coordinates of the first keypoint predicted by the previous decoder layer, y O Represents the output of the coordinate offset unit of the key point coordinate prediction module, indicating that y is relative to y R The input of the left offset unit is the output matrix of the deformable attention module.
[0127] Among them, the σ function is the existing technology, and the specific σ function formula is:
[0128]
[0129] Finally, the coordinates of the first keypoint predicted by the last decoder layer are determined as the coordinates of the final predicted first keypoint.
[0130] In order to more clearly understand the present disclosure, a model with three decoder layers is now used for illustration.
[0131] For the first decoder layer, the masked Q embedded with the masked position information init As query matrix, key matrix, masked Q init (without embedding position information) is input to the first decoder layer as a matrix of values.
[0132] For the second decoder layer, the masked QD of the first decoder layer with embedded masked position information is used as the query matrix and key matrix, and the masked QD of the first decoder layer (without embedded position information) is input to the second decoder layer as the value matrix.
[0133] For the third decoder layer, the masked QD of the second decoder layer with embedded masked position information is used as the query matrix and key matrix, and the masked QD of the second decoder layer (without embedded position information) is input to the third decoder layer as the value matrix.
[0134] The first decoder layer predicts the key point coordinates using the initial key point coordinate matrix; the second decoder layer predicts the key point coordinates using the key point coordinates predicted by the first decoder layer, and the third decoder layer predicts the key point coordinates using the key point coordinates predicted by the second decoder layer. It should be understood by those skilled in the art that the model with three decoder layers is only an example, and the model may have only one decoder layer or more decoder layers, and the decoder layers other than the first decoder layer have similar input and output operations.
[0135] As an example, the model can be trained using the L1 norm loss (the absolute value of the difference between the predicted value and the true value) function between the key point coordinates of the predicted training image samples and the true key point coordinates of the training image samples. The regression loss function L used in the training reg as follows:
[0136]
[0137] Among them, y l represents the keypoint coordinates of the training image samples predicted by each decoder layer, represents the actual coordinates of the key points of the training image samples, L d is the number of decoder layers, l is the index of the decoder layer, The length of y is N, only some positions are filled with the actual key point coordinates, and the rest are 0, 0 Indicates the initial key point location.
[0138] According to the method for predicting facial key point coordinates according to the embodiment of the present disclosure, the entire prediction model can be trained end-to-end.
[0139] Reference above Figure 1-Figure 4 The method of detecting key points of face images is explained below. Figure 5-Figure 6 A device for detecting key points of a face image according to an implementation of the present disclosure is described.
[0140] Reference Figure 5 According to an embodiment of the present disclosure, the apparatus 500 for detecting facial key points may include an encoder 501 and a decoder 502. Those skilled in the art should understand that the apparatus 500 may additionally include other components, and the components included in the apparatus 500 may be split or combined.
[0141] As an example, the encoder 501 can be configured to obtain a multi-level feature map of the face image through a convolutional neural network layer, fully connect the last level feature map in the multi-level feature map through a fully connected layer to obtain an initial query matrix, and obtain a memory feature matrix by flattening and connecting the multi-level feature map.
[0142] As an example, the decoder may include at least one decoder layer in cascade, wherein the at least one decoder layer may be configured to determine the coordinates of key points of the face image corresponding to at least one key point style based on user input for specifying at least one key point style among a plurality of key point styles, a memory feature matrix, and an initial query matrix.
[0143] As an example, each decoder layer includes a cascaded mask module, a self-attention module, a deformable attention module and a key point coordinate prediction block, and wherein the step of determining the coordinates of the first key point includes: based on at least one key point style among the multiple key point styles specified by the user input, obtaining a mask matrix corresponding to the at least one key point; and based on the mask matrix, masking the query matrix, key matrix and value matrix input to the self-attention module for predicting the coordinates of the first key point.
[0144] As an example, the number of elements in the initial query matrix is equal to N, where N indicates the sum of the number of key points corresponding to each key point style in the multiple key point styles; wherein the decoder 501 can be configured to: mask the initial query matrix and the position information corresponding to the initial query matrix using the mask module of the first decoder layer based on the mask matrix, so as to set some elements in the initial query matrix and the position information to zero, wherein the some elements correspond to the key points other than the first key point in the key points corresponding to the multiple key point styles; the masked initial query matrix embedded with the masked position information, the masked initial query matrix embedded with the masked position information, and the masked initial query matrix The matrix is input into the self-attention module of the first decoder layer as the query matrix, key matrix and value matrix of the self-attention module of the first decoder layer; the output matrix of the self-attention module of the current decoder layer, the memory feature matrix and the coordinates of the first key point predicted by the previous decoder layer after masking are input into the deformable attention module of the current decoder layer to obtain the output matrix of the deformable attention module of the current decoder layer, wherein the output matrix of the self-attention module of the current decoder layer and the memory feature matrix are the query matrix and value matrix of the deformable attention module of the current decoder layer, wherein the coordinates of the first key point predicted by the previous decoder layer after masking input into the deformable attention module of the first decoder layer are the basis The key point coordinates are obtained by masking the initial key point coordinates obtained based on the initial query matrix using the mask matrix, wherein the coordinates of the first key point predicted by the previous decoder layer after masking are obtained by setting some elements in the coordinates of the first key point predicted by the previous decoder layer to zero based on the mask matrix by the mask module, wherein the some elements correspond to the key points other than the first key point among the key points corresponding to the styles of the multiple key points; based on the mask matrix, the mask module of the next decoder layer of the current decoder layer is used to mask the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer so that the current decoder The output matrix of the deformable attention module of the decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer are set to zero, wherein the partial elements correspond to the key points other than the first key point among the key points corresponding to the styles of the multiple key points; the masked output matrix of the deformable attention module of the current decoder layer, the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information, and the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information are input as the value matrix, query matrix and key matrix of the self-attention module of the next decoder layer to the self-attention module of the next decoder layer;And inputting the output matrix of the deformable attention module of the current decoder layer and the coordinates of the first key point predicted by the previous decoder layer after masking into the key point coordinate prediction module of the current decoder layer to obtain the coordinates of the first key point predicted by the current decoder layer, wherein the coordinates of the first key point predicted by the last decoder layer in the at least one decoder layer are used as the final coordinates of the first key point. ;
[0145] As an example, the output matrix QE of the self-attention module of each decoder layer is obtained by the following formula:
[0146]
[0147] in, represents the i-th row vector in the output matrix QE, α ij represents the attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module and the j-th row vector of the key matrix input to the self-attention module, q j Represents the j-th row vector in the output matrix of the deformable attention module of the previous decoder layer after masking.
[0148] As an example, the output matrix QD of the deformable attention module of each decoder layer is obtained by the following formula:
[0149]
[0150] Among them, f i represents the updated features of the i-th keypoint in QD, β ik represents the attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the deformable attention module, x ik Represents the feature corresponding to the k-th reference point coordinate in the memory feature matrix, where the position offset between the k-th reference point coordinate and the i-th key point coordinate in the coordinate of the first key point predicted by the previous decoder layer after masking is obtained by performing a fully connected operation on the query matrix input to the deformable attention module, where K is a preset value.
[0151] As an example, the coordinates of the first keypoint predicted by each decoder layer are obtained by the following formula:
[0152] y=σ(y O +σ -1 (y R ))
[0153] Among them, y represents the coordinates of the first key point predicted by the current decoder layer, y RRepresents the coordinates of the first key point predicted by the previous decoder layer after masking, y O Indicates y relative to y R offset.
[0154] As an example, the convolutional network layer, the fully connected layer, and the at least one decoder layer are obtained by training with training image samples based on the following regression loss function:
[0155]
[0156] Among them, L reg represents the regression loss, y l represents the keypoint coordinates of the training image samples predicted by each decoder layer, represents the actual key point coordinates of the training image sample, L d is the number of decoder layers and l is the index of the decoder layer.
[0157] As an example, the apparatus 500 may additionally include a fully connected layer to perform a fully connected operation on the initial query matrix to obtain initial coordinates of key points of the face image.
[0158] Figure 6 is a block diagram illustrating a structure of an electronic device 600 for predicting key point coordinates of a face image according to another exemplary embodiment of the present disclosure.
[0159] Typically, the electronic device 600 includes a processor 601 and a memory 602 .
[0160] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0161] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the method for detecting key points of a face image disclosed in the present invention.
[0162] In some embodiments, the electronic device 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: a radio frequency circuit 604, a touch display screen 605, a camera 606, an audio circuit 607, a positioning component 608 and a power supply 609.
[0163] In some embodiments, the electronic device 600 further includes one or more sensors 610 , including but not limited to: an acceleration sensor 611 , a gyroscope sensor 612 , a pressure sensor 613 , a fingerprint sensor 614 , an optical sensor 615 , and a proximity sensor 616 .
[0164] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device 600, and the electronic device 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.
[0165] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to execute the method for detecting key points of a face image according to the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.
[0166] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to implement the method for predicting key point coordinates of a face image as described above.
[0167] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.
Claims
1. A method for detecting key points of a face image, include: Obtaining a multi-level feature map of the face image through a convolutional neural network layer; Fully connecting the last level feature map in the multi-level feature map through a fully connected layer to obtain an initial query matrix; Obtain the memory feature matrix by flattening and connecting the multi-level feature maps; as well as Based on user input for specifying at least one key point style among a plurality of key point styles, the memory feature matrix and the initial query matrix, coordinates of a first key point of the face image corresponding to the at least one key point style are determined using at least one decoder layer of the cascade.
2. The method according to claim 1, in, Each decoder layer consists of a cascade of mask modules, self-attention modules, deformable attention modules, and keypoint coordinate prediction blocks, and The step of determining the coordinates of the first key point includes: Based on the at least one key point style among the multiple key point styles specified by the user input, obtaining a mask matrix corresponding to the at least one key point; and Based on the mask matrix, the query matrix, the key matrix and the value matrix input to the self-attention module are masked for predicting the coordinates of the first key point.
3. The method according to claim 2, in, The number of elements in the initial query matrix is equal to N, where N indicates the sum of the numbers of key points corresponding to each key point style in the plurality of key point styles; The step of determining the coordinates of the first key point includes: masking an initial query matrix and position information corresponding to the initial query matrix using a mask module of a first decoder layer based on the mask matrix to set some elements in the initial query matrix and the position information to zero, wherein the some elements correspond to key points other than the first key point among the key points corresponding to the multiple key point styles; Inputting the masked initial query matrix embedded with the masked position information, the masked initial query matrix embedded with the masked position information, and the masked initial query matrix as the query matrix, key matrix, and value matrix of the self-attention module of the first decoder layer to the self-attention module of the first decoder layer; Inputting the output matrix of the self-attention module of the current decoder layer, the memory feature matrix and the masked coordinates of the first key point predicted by the previous decoder layer into the deformable attention module of the current decoder layer to obtain the output matrix of the deformable attention module of the current decoder layer, wherein the output matrix of the self-attention module of the current decoder layer and the memory feature matrix are the query matrix and the value matrix of the deformable attention module of the current decoder layer, wherein the coordinates of the first key point predicted by the previous decoder layer after masking input into the deformable attention module of the first decoder layer are the key point coordinates obtained by masking the initial key point coordinates obtained based on the initial query matrix based on the mask matrix, wherein the coordinates of the first key point predicted by the previous decoder layer after masking are obtained by setting some elements in the coordinates of the first key point predicted by the previous decoder layer to zero based on the mask matrix by the mask module, wherein the some elements correspond to key points other than the first key point among the key points corresponding to the styles of the multiple key points; Based on the mask matrix, mask the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer using the mask module of the next decoder layer of the current decoder layer so that some elements in the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer are set to zero, wherein the some elements correspond to the key points other than the first key point among the key points corresponding to the multiple key point styles; Inputting the masked output matrix of the deformable attention module of the current decoder layer, the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information, and the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information into the self-attention module of the next decoder layer as the value matrix, query matrix, and key matrix of the self-attention module of the next decoder layer; and The output matrix of the deformable attention module of the current decoder layer and the coordinates of the first key point predicted by the previous decoder layer after masking are input into the key point coordinate prediction module of the current decoder layer to obtain the coordinates of the first key point predicted by the current decoder layer, wherein the coordinates of the first key point predicted by the last decoder layer in the at least one decoder layer are used as the final coordinates of the first key point.
4. The method according to claim 3, in, The output matrix QE of the self-attention module of each decoder layer is obtained by the following formula: in, represents the i-th row vector in the output matrix QE, α ij represents the attention weight obtained by normalizing the dot product of the i-th row vector of the query matrix input to the self-attention module and the j-th row vector of the key matrix input to the self-attention module, q j Represents the j-th row vector in the output matrix of the deformable attention module of the previous decoder layer after masking.
5. The method according to claim 4, in, The output matrix QD of the deformable attention module of each decoder layer is obtained by the following formula: Among them, f i represents the updated features of the i-th key point in QD, β ik represents the attention weight obtained by performing a fully connected operation and a softmax operation on the query matrix input to the deformable attention module, x ik Represents the feature corresponding to the k-th reference point coordinate in the memory feature matrix, wherein the position offset between the k-th reference point coordinate and the i-th key point coordinate in the coordinate of the first key point predicted by the previous decoder layer after masking is obtained by performing a fully connected operation on the query matrix input to the deformable attention module, wherein K is a preset value.
6. The method according to claim 3, in, The coordinates of the first keypoint predicted by each decoder layer are obtained by the following formula: and=σ(y O +σ -1 (and R )) Among them, y represents the coordinates of the first key point predicted by the current decoder layer, y R Represents the coordinates of the first key point predicted by the previous decoder layer after masking, y O Indicates y relative to y R offset.
7. A device for detecting key points of a face image, include: An encoder is configured to obtain a multi-level feature map of the face image through a convolutional neural network layer, fully connect the last level feature map in the multi-level feature map through a fully connected layer to obtain an initial query matrix, and obtain a memory feature matrix by flattening and connecting the multi-level feature map; A decoder comprising at least one cascaded decoder layer, wherein the at least one decoder layer is configured to determine the coordinates of key points of the face image corresponding to at least one key point style based on user input for specifying at least one key point style among a plurality of key point styles, a memory feature matrix, and an initial query matrix.
8. The device according to claim 7, in, Each decoder layer consists of a cascade of mask modules, self-attention modules, deformable attention modules, and keypoint coordinate prediction blocks, and Wherein, the decoder is configured as: Based on the at least one key point style among the multiple key point styles specified by the user input, obtaining a mask matrix corresponding to the at least one key point; and Based on the mask matrix, the query matrix, the key matrix and the value matrix input to the self-attention module are masked for predicting the coordinates of the first key point.
9. The device as claimed in claim 8, in, The number of elements in the initial query matrix is equal to N, where N indicates the sum of the numbers of key points corresponding to each key point style in the plurality of key point styles; Wherein, the decoder is configured as: masking an initial query matrix and position information corresponding to the initial query matrix using a mask module of a first decoder layer based on the mask matrix to set some elements in the initial query matrix and the position information to zero, wherein the some elements correspond to key points other than the first key point among the key points corresponding to the multiple key point styles; Inputting the masked initial query matrix embedded with the masked position information, the masked initial query matrix embedded with the masked position information, and the masked initial query matrix as the query matrix, key matrix, and value matrix of the self-attention module of the first decoder layer to the self-attention module of the first decoder layer; Inputting the output matrix of the self-attention module of the current decoder layer, the memory feature matrix and the masked coordinates of the first key point predicted by the previous decoder layer into the deformable attention module of the current decoder layer to obtain the output matrix of the deformable attention module of the current decoder layer, wherein the output matrix of the self-attention module of the current decoder layer and the memory feature matrix are the query matrix and value matrix of the deformable attention module of the current decoder layer, wherein the coordinates of the first key point predicted by the previous decoder layer after masking input into the deformable attention module of the first decoder layer are the key point coordinates obtained by masking the initial key point coordinates obtained based on the initial query matrix based on the mask matrix, wherein the coordinates of the first key point predicted by the previous decoder layer after masking are obtained by setting some elements in the key point coordinates predicted by the previous decoder layer to zero based on the mask matrix by the mask module, wherein the some elements correspond to key points other than the first key point among the key points corresponding to the multiple key point styles; Based on the mask matrix, mask the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer using the mask module of the next decoder layer so that some elements in the output matrix of the deformable attention module of the current decoder layer and the position information corresponding to the output matrix of the deformable attention module of the current decoder layer are set to zero, wherein the some elements correspond to key points other than the first key point among the key points corresponding to the multiple key point styles; Inputting the masked output matrix of the deformable attention module of the current decoder layer, the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information, and the masked output matrix of the deformable attention module of the current decoder layer embedded with the masked position information into the self-attention module of the next decoder layer as the value matrix, query matrix, and key matrix of the self-attention module of the next decoder layer; and The output matrix of the deformable attention module of the current decoder layer and the coordinates of the first key point predicted by the previous decoder layer after masking are input into the key point coordinate prediction module of the current decoder layer to obtain the coordinates of the first key point predicted by the current decoder layer, wherein the coordinates of the first key point predicted by the last decoder layer in the at least one decoder layer are used as the final coordinates of the first key point.
10. A computer-readable storage medium storing a computer program, in, When the computer program is executed by a processor, the method for detecting key point coordinates of a face image as described above is implemented.
