A method, device, electronic device and storage medium for detecting key points of human face

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for detecting facial key points. The method includes: performing dimensionality upscaling on an image to be detected to generate a first-scale feature map; converting the first-scale feature map into multiple second-scale feature maps in a preset multi-scale search space; fusing the second-scale feature maps to generate a facial key point heatmap, wherein the facial key point heatmap includes facial key point probabilities; and determining the coordinates of the facial key points based on the facial key point probabilities. This invention, based on a multi-scale neural search space, converts the first-scale feature map into a second-scale feature map that meets the required parameter quantity and accuracy requirements, and fuses the second-scale feature maps to determine the coordinates of the facial key points, thereby achieving precise localization of facial key points in an image.

Description

Technical Field

[0001] The present invention relates to the field of biometric identification technology, and in particular to a human face key point detection method, a human face key point detection device, an electronic device and a storage medium. Background Art

[0002] With the development of science and technology, the application of face recognition technology in the field of image processing is becoming more and more popular. Facial key point detection is a key step in the field of face recognition and analysis. It is the premise and breakthrough for other face-related problems such as automatic face recognition, expression analysis, 3D face reconstruction and 3D animation. At present, the existing face key point detection methods all use manually designed convolutional neural network measurement algorithms, that is, face key point detection requires the design and training of multiple convolutional neural networks, resulting in a huge number of parameters and numerous steps, and the accuracy of the detected face key points cannot be guaranteed. Summary of the invention

[0003] In view of the above problems, embodiments of the present invention are proposed to provide a method for detecting key points in a face, a device for detecting key points in a face, an electronic device and a storage medium that overcome the above problems or at least partially solve the above problems.

[0004] In order to solve the above problems, an embodiment of the present invention discloses a method for detecting key points of a face, comprising:

[0005] Performing dimension-up processing on the image to be detected to generate a first-scale feature map;

[0006] In a preset multi-scale search space, converting the first-scale feature map into a plurality of second-scale feature maps;

[0007] fusing the second scale feature map to generate a facial key point heat map, wherein the facial key point heat map includes facial key point probabilities;

[0008] The facial key point coordinates are determined according to the facial key point probabilities.

[0009] Optionally, the method further comprises:

[0010] Get the original image;

[0011] The original image is scaled to form the image to be detected.

[0012] Optionally, the step of converting the first scale feature map into a plurality of second scale feature maps in a preset multi-scale search space includes:

[0013] In the preset multi-scale search space, determining a target path, wherein the target path includes multiple stages;

[0014] In the stage, the first scale feature map is subjected to feature fusion and feature encoding to generate the second scale feature map.

[0015] Optionally, the preset multi-scale search space includes a plurality of path selection modules, and the step of determining the target path in the preset multi-scale search space includes:

[0016] Performing forward propagation on the path selection module to obtain an evaluation index;

[0017] determining a target evaluation indicator from the evaluation indicators;

[0018] The path selection modules corresponding to the target evaluation indicators are combined to generate the target path.

[0019] Optionally, the stage corresponds to a resolution and a number of channels; in the stage, the step of performing feature fusion and feature encoding on the first scale feature map to generate the second scale feature map includes:

[0020] Adjusting the first scale feature map according to the number of channels;

[0021] Sampling the adjusted first scale feature map according to the resolution;

[0022] Performing weighted calculation on the sampled first scale feature map according to preset path parameters to generate a fused feature map;

[0023] The fused feature map is encoded according to a preset candidate operation to generate the second scale feature map.

[0024] Optionally, the preset candidate operation includes a jump connection and an input residual network; the preset candidate operation corresponds to an operation parameter; and the step of encoding the fused feature map according to the preset operation parameter to generate the second scale feature map includes:

[0025] Performing skip connection and inputting residual network operation on the fused feature map;

[0026] The weighted sum of the operated fusion feature map is performed according to the operation parameters to generate the second scale feature map.

[0027] Optionally, the step of fusing the second scale feature map to generate a facial key point heat map includes:

[0028] Sampling the second-scale feature map to generate a first-resolution image;

[0029] fusing the first resolution image with the channel of the preset multi-scale search space as the dimensional direction;

[0030] The fused first-resolution image is channel compressed to generate the facial key point heat map.

[0031] The embodiment of the present invention also discloses a facial key point detection device, comprising:

[0032] A dimension-increasing module is used to perform dimension-increasing processing on the image to be detected to generate a first scale feature map;

[0033] A conversion module, used to convert the first scale feature map into a plurality of second scale feature maps in a preset multi-scale search space;

[0034] A fusion module, used for fusing the second scale feature map to generate a face key point heat map, wherein the face key point heat map includes face key point probabilities;

[0035] A determination module is used to determine the coordinates of the facial key points according to the facial key point probabilities.

[0036] An embodiment of the present invention further discloses an electronic device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the steps of the above-mentioned facial key point detection method when executed by the processor.

[0037] The embodiment of the present invention further discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the facial key point detection method described above are implemented.

[0038] The embodiments of the present invention include the following advantages:

[0039] The embodiment of the present invention generates a first-scale feature map by performing dimensionality upscaling processing on an image to be detected; converts the first-scale feature map into multiple second-scale feature maps in a preset multi-scale search space; automatically searches the multi-scale search space and converts the first-scale feature map according to a trade-off between parameter quantity and accuracy, thereby avoiding multiple neural networks that are manually designed and trained; fuses the second-scale feature maps to generate a face key point heat map, wherein the face key point heat map includes face key point probabilities; determines the face key point coordinates based on the face key point probabilities, thereby achieving accurate positioning of the face key points in the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flowchart of a method for detecting key points of a face according to an embodiment of the present invention;

[0041] Figure 2 is a flowchart of another method for detecting key points of a face according to an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of a multi-scale search space according to an embodiment of the present invention;

[0043] Figure 4 is an example diagram of a target path according to an embodiment of the present invention;

[0044] Figure 5 is a flow chart of a method for detecting key points of a human face according to an example of the present invention;

[0045] Figure 6 It is a structural block diagram of a facial key point detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0047] Facial key point detection (positioning), also known as face alignment, is a common visual task based on face images. The goal is to find the coordinate position of each semantic key point in the face image, such as the tip of the nose, the corner of the mouth, etc. At present, in the method of facial key point detection, convolutional neural network models and other neural networks used for classification are often used for recognition. One or more convolutional neural network models are manually designed and trained, and put into use after the training is completed. However, the number of parameters designed by the manually designed network is large, and the manual design steps are numerous. The obtained neural network has certain limitations, which leads to errors in facial key point detection. Furthermore, after the design of the manually designed network is completed, the network structure remains unchanged and the parameters are fixed, resulting in low adaptability to different tasks, which also leads to errors in facial key point detection.

[0048] Reference Figure 1 , shows a flowchart of a method for detecting key points of a face according to an embodiment of the present invention, which may specifically include the following steps:

[0049] Step 101, performing dimensionality upscaling on the image to be detected to generate a first scale feature map;

[0050] In practical applications, the image to be detected can be processed by upscaling, the number of channels of the image to be detected can be increased, and an image with a preset resolution and a preset number of channels can be generated as the first scale feature map. For example, the preset resolution can be 256×256 pixels, and the preset number of channels can be 3 channels, that is, the first scale feature map is a 3-channel image with a size of 256×256 pixels. It should be noted that multi-scale refers to multi-resolution and multi-channel.

[0051] Step 102: converting the first scale feature map into a plurality of second scale feature maps in a preset multi-scale search space;

[0052] In an embodiment of the invention, the multi-scale network search space is a multi-resolution and multi-channel network search space. The multi-scale search space can be obtained by training through automatic search technology. In the preset multi-scale network search space, the first scale feature map can be classified and fused multiple times and converted into multiple second scale feature maps. Among them, the second scale feature map represents a feature map with a specific resolution and number of channels obtained after the first scale feature map passes through the multi-scale search space. The second scale can specifically represent multiple different scales. For example, the first scale feature map is a 3×256×256 channel image. After the multi-scale search space, it is converted to obtain a total of 4 feature maps of different scales of 36×128×128, 72×64×64, 144×32×32, and 288×16×16. 36×128×128, 72×64×64, 144×32×32, and 288×16×16 can all be used as the second scale.

[0053] Step 103, fusing the second scale feature map to generate a facial key point heat map, wherein the facial key point heat map includes facial key point probabilities;

[0054] The obtained multiple second-scale feature maps are fused to generate a face key point heat map, wherein the face key point heat map has N channels, N is the number of face key points, and the heat map of each channel represents a predicted probability map of the current key point, and the predicted probability map includes the probability of the face key points.

[0055] Step 104, determining the coordinates of the facial key points according to the facial key point probabilities.

[0056] According to the probability of facial key points, the coordinates corresponding to the facial key point with the maximum probability value are selected and exported to generate the coordinates of N facial key points.

[0057] The embodiment of the present invention generates a first-scale feature map by performing dimensionality upscaling processing on an image to be detected; converts the first-scale feature map into multiple second-scale feature maps in a preset multi-scale search space; automatically searches the multi-scale search space and converts the first-scale feature map according to a trade-off between parameter quantity and accuracy, thereby avoiding multiple neural networks that are manually designed and trained; fuses the second-scale feature maps to generate a face key point heat map, wherein the face key point heat map includes face key point probabilities; determines the face key point coordinates based on the face key point probabilities, thereby achieving accurate positioning of the face key points in the image.

[0058] Reference Figure 2 , shows a flowchart of another embodiment of a method for detecting key points of a face according to the present invention, which may specifically include the following steps:

[0059] Step 201, obtaining an original image;

[0060] In practical applications, the original image for which facial key point detection is required can be obtained first, and the original image can be obtained in real time or read from a specified storage address. The execution storage address can be a local address or an external address, which is not specifically limited in the embodiment of the present invention.

[0061] Step 202, scaling the original image to form the image to be detected;

[0062] After obtaining the original image, since different original images may have multiple resolutions, they need to be unified. Therefore, the original image can be scaled to an image to be detected with a specified resolution. In one example of the present invention, the specified resolution can be 256×256. That is, the original images are scaled to 256×256 images to be detected. Specifically, when the resolution of the original image is greater than 256×256, the original image can be downsampled. When the resolution of the original image is less than 256×256, the original image can be upsampled. When the resolution of the original image is equal to 256×256, the original image can be directly used as the image to be detected, and the scaling coefficient is 1 at this time.

[0063] Step 203, performing dimensionality upscaling on the image to be detected to generate a first scale feature map;

[0064] In practical applications, the image to be detected can be processed by dimension upscaling to increase the number of channels of the image and generate a first-scale feature map.

[0065] In addition, dimensionality increase and scaling can be performed through the input module in the convolutional neural network, which includes a 3×3 convolution module (convolution layer + batch normalization layer + Relu activation layer) with a stride of 2. The image to be detected is input into the output module, and the convolution operation is performed through the 3×3 convolution module to obtain the first scale feature map.

[0066] Step 204: converting the first scale feature map into a plurality of second scale feature maps in a preset multi-scale search space;

[0067] In the embodiment of the present invention, the multi-scale network search space is a multi-resolution and multi-channel network search space. The specific structure of the multi-scale search space can be referred to Figure 3 As shown in FIG, the search space is defined as a supernet, which mainly includes an input module, a path selection block and an output module.

[0068] The input module is the input module of the processing step 203, which is used to perform dimension upscaling and scaling (downsampling) on ​​the input image.

[0069] Path selection module: It includes feature fusion layers and basic layers. The function of the feature fusion layers is to unify the scales of different input feature maps and merge them into a new feature map, while the function of the basic layers is to control the depth of the path selection module.

[0070] Output module: mainly responsible for feature fusion and face key point prediction

[0071] Therefore, in the multi-scale search space, the first-scale feature map can be input into the path selection module, and the path selection module performs feature fusion and feature encoding on the first-scale feature map to convert it into multiple second-scale feature maps.

[0072] Specifically, it may include:

[0073] Sub-step S2041, determining a target path in the preset multi-scale search space, wherein the target path includes multiple stages;

[0074] Thanks to the design of the multi-scale search space, there is a multi-path architecture from input to output in the multi-scale search space. Therefore, the target path can be determined based on the trade-off between the number of parameters and the accuracy. The target path includes at least one path selection module in each stage of the multi-scale search space. Figure 4 As shown, it can be obtained from Figure 3 The multi-scale search space determines a target path, which goes through five stages of the multi-scale search space; the first stage selects a 256×128×128 path selection module, the second stage selects a 72×64×64 path selection module, the third stage selects a 144×32×32 path selection module, the fourth stage selects a 288×16×16 path selection module, and the fifth stage selects 36×128×128, 144×32×32, and 288×16×16 path selection modules. It can be seen that each stage corresponds to a resolution and number of channels.

[0075] In a preferred embodiment of the present invention, the step of determining the target path in the preset multi-scale search space includes:

[0076] Sub-step S20411, performing forward propagation on the path selection module to obtain an evaluation index;

[0077] Sub-step S20412, determining a target evaluation index from the evaluation indexes;

[0078] Sub-step S20413, combining the path selection modules corresponding to the target evaluation indicators to generate the target path.

[0079] In order to select a multi-path architecture in the supernet of the multi-scale search space. The importance of each path can be directly evaluated, and the target path can be determined by the importance. Specifically, the evaluation index (NME, Normalize Mean Error) can be used as the evaluation standard for the importance. In practical applications, a path selection module can be randomly selected, and one input path of the module is removed each time. The supernet is forward propagated on the verification set to obtain the corresponding NME, and this process is repeated until all the input paths on this module are evaluated, and the path with the largest NME is retained. The remaining connections are deleted from the supernet, and then after 5 rounds of fine-tuning, the architecture parameters and network parameters are updated again. The above process is repeated until the input paths of all modules are selected. At this point, a multi-path network structure is obtained, and each path corresponds to an NME, which represents the contribution of the current path to the overall network. The Dijkstra algorithm is used to select the first K corresponding path selection modules for combination to generate the target path. K can be selected as needed. In an example of the present invention, the K can be 3.

[0080] Sub-step S2042: In the stage, feature fusion and feature encoding are performed on the first scale feature map to generate the second scale feature map.

[0081] In each stage, feature fusion and feature encoding are performed on the first-scale feature map according to the scale requirement corresponding to the stage, until multiple second-scale feature maps are generated by feature fusion and feature encoding of all stages.

[0082] In a preferred embodiment of the present invention, in the stage, the step of performing feature fusion and feature encoding on the first scale feature map to generate the second scale feature map includes:

[0083] Sub-step S20421, adjusting the first scale feature map according to the number of channels;

[0084] First, according to the setting of the number of channels in the path selection module selected at this stage, the number of channels of the first scale feature map can be adjusted, such as Figure 3 Taking the second path selection module in the fourth stage as an example, the number of channels of this path selection module is 72. The number of channels corresponding to the feature map output in the third stage can be unified to 72 through a 3×3 convolution module (kernel).

[0085] Sub-step S20422, sampling the adjusted first scale feature map according to the resolution;

[0086] After adjusting the number of channels, the first scale feature map can further adjust its resolution by upsampling or downsampling to make it consistent with the resolution of the path selection module. When the resolution of the first scale feature map is greater than the resolution of the path selection module, a downsampling operation is used; when the resolution of the first scale feature map is less than the resolution of the path selection module, an upsampling operation is used. For example, Figure 3 Taking the second path selection module in the fourth stage as an example, after adjusting the number of channels, the feature map with a resolution of 32×32 needs to be upsampled and interpolated, while the feature map with a resolution of 128×128 needs to be downsampled.

[0087] Sub-step S20423, performing weighted calculation on the sampled first scale feature map according to preset path parameters to generate a fused feature map;

[0088] After the number of channels and resolution are unified, the first scale feature map can be weighted and summed according to the preset path parameters to generate a fused feature map. Among them, the path parameter β is the structural parameter in the multi-scale search space, and the softmax operation is used to determine the path weight p ij , and then perform weighted summation according to the path weight to generate a fusion feature map. Path weight p ij The formula is:

[0089]

[0090] The path weight represents the probability of the i-th path selection module to the j-th path selection module, where m represents the number of outputs that the i-th path selection module can output.

[0091] Sub-step S20424: encoding the fused feature map according to a preset candidate operation to generate the second scale feature map.

[0092] After obtaining the fused feature map, the fused feature map is encoded according to the preset candidate operation to generate the second-scale feature map.

[0093] In a preferred embodiment of the present invention, the preset candidate operation includes a jump connection and an input residual network; the preset candidate operation corresponds to an operation parameter; and the step of encoding the fused feature map according to the preset operation parameter to generate the second scale feature map includes:

[0094] Sub-step S204241, performing skip connection on the fused feature map and inputting a residual network operation;

[0095] In practical applications, the fused feature map is pre-connected in the base layer of the path selection module and input into the residual network. The residual network consists of two cascaded 3×3 convolution modules and an identity mapping operation. The feature map is processed through the skip connection and input into the residual network to control the depth of the neural network.

[0096] Sub-step 204242: performing weighted summation on the operated fusion feature map according to the operation parameters to generate the second scale feature map.

[0097] After performing the jump connection and inputting it into the fused feature map of the residual network, the operation weight w can be determined according to the operation parameters corresponding to the jump connection and the residual network. o , and then according to the operation weight w o Perform weighted summation to generate the second scale feature map. Among them, the operation weight w o The formula is:

[0098]

[0099] Among them, o is the current candidate operation, o ′ are the two candidate operations obtained after sampling, is the set of all candidate operations.

[0100] Step 205, fusing the second scale feature map to generate a facial key point heat map, wherein the facial key point heat map includes facial key point probabilities;

[0101] After obtaining the second-scale feature map, the second-scale feature map can be input into the output module in the multi-scale search space, and all the second-scale feature maps can be fused to generate the facial key point probability of each facial key point and the corresponding facial key point probability heat map.

[0102] Specifically, the second-scale feature map can be sampled to generate a first-resolution image; the first-resolution image can be fused with the channel of the preset multi-scale search space as the dimensional direction; the fused first-resolution image can be channel compressed to generate the facial key point heat map.

[0103] Since the second scale feature map contains multiple scales, the specific position descriptions of the corresponding position key points between different scales are different. To this end, the second scale feature map is sampled, and the sampling includes upsampling and downsampling, and the second scale feature map is converted into a first resolution image with a first resolution. The first resolution can be set according to actual needs. In an example of the present invention, the first resolution can be 64×64.

[0104] Then, all the first-resolution images are compressed to N channels with a 1×1 convolution kernel, which represents the N facial key points that need to be predicted. A facial key point heat map with a scale of N×64×64 is generated, where N is the number of facial key points.

[0105] Step 206, determining the coordinates of the facial key points according to the facial key point probabilities.

[0106] In the face key point heat maps of all channels, select the face key point heat map with the highest face key point probability corresponding to each face key point, use the coordinates of the face key point in the face key point heat map as the face key point coordinates corresponding to the face key point, repeatedly select N face key points, and generate N face key point coordinates.

[0107] The embodiment of the present invention obtains an original image, scales it to generate an image to be detected; performs dimensionality upscaling on the image to be detected to generate a first scale feature map; converts the first scale feature map into multiple second scale feature maps in a preset multi-scale search space; fuses the second scale feature maps to generate a face key point heat map, the face key point heat map includes face key point probabilities; determines the face key point coordinates based on the face key point probabilities, and realizes accurate positioning of the face key points in the image. And because the operation parameters and path parameters are variable, the network architecture has stronger applicability.

[0108] In order to enable those skilled in the art to better understand the embodiments of the present invention, the embodiments of the present invention are described below by using an example:

[0109] You can refer to Figure 5 , shows a flowchart of a method for detecting key points of a human face according to an example of the present invention; Figure 3 The multi-scale search space shown is used as an example for explanation.

[0110] First, we obtain the face data used for training and verification. The face data used for model training is the training set, and the face data used to verify the accuracy of the model is the verification set.

[0111] The multi-scale search space is defined as a supernet, including an input module, a path selection module, and an output module. Input module: Contains a 3×3 convolution module (convolution layer + normalization layer (BatchNormalization) + Relu activation layer) with a stride of 2, which is responsible for upscaling (increasing the number of channels) and downsampling (reducing the resolution) the input image. Path selection module: Contains feature fusion layers and basic layers. The function of the feature fusion layer is to unify the scales of different input feature maps and merge them into a new feature map. Among them, the number of stacked layers for the basic layer is 7 in the first four stages and 0 in the last stage.

[0112] Then, in order to relax the discrete search space in the hypernet into a continuous learnable search space, the architecture parameters (i.e., operation parameter α and path parameter β) are introduced, and the softmax operation is used to obtain the operation weights w o and path weight p ij Among them, the architecture parameters can directly participate in the training of the supernet using the gradient descent algorithm, and are updated alternately with the network parameters. Among them, the architecture parameters can be used to describe the importance of candidate operations and candidate paths in the search space, while the network parameters are used to describe the learnable parameters of the network structure, where the former is used for network structure search and the latter is used for network structure learning.

[0113] In order to reduce memory overhead in supernet training, the parameters are updated based on sampling. Specifically, in the architecture parameter update phase, two candidate operations (skip connection and residual network) are sampled and the Adam optimizer (lr = 3 × 10 -4 , weight decay = 10 -3 , beta1=0.5,beta2=0.999) are iteratively trained on the validation set with a batch size of 8; in the network parameter update phase, only one candidate operation is sampled and the SGD (Stochastic Gradient Descent) optimizer (lr decay=[0.1,1×10 -4 ], momentum=0.9, weight decay=4×10 -5 ) It is iteratively trained on a training set with a batch size of 16. In each round of iteration, the architecture parameters α and β are updated first, and then the network parameters are optimized, for a total of 40 rounds.

[0114] After the search is completed, a converged supernet is obtained. A random path selection module in the supernet is forward propagated on the validation set to obtain the corresponding NME. This process is repeated until all input paths on this module are evaluated, and the path with the largest NME is retained. The remaining connections are deleted from the supernet. After 5 rounds of fine-tuning, the architecture parameters and network parameters are updated again. The above process is repeated until the input paths of all modules are selected. The Dijkstra algorithm is then used to select the three optimal paths to obtain a multi-path network architecture.

[0115] After obtaining the multipath network architecture, the multipath network architecture is retrained. The architecture parameters and network parameters in the supernet will not be involved in the retraining phase. Therefore, the multipath network is initialized and then the Adam optimizer (lr = 10 -4 ) The search architecture is iteratively trained for 60 rounds on a training set with a batch size of 16 and tested for inference on a test set with a batch size of 8, resulting in a converged multi-path network architecture.

[0116] After the test image is up-scaled and sampled to generate an image of size 3×256×256, it first passes through the input module to generate a feature map of scale 256×128×128, and then passes through the 13 path selection modules contained in stages 1-5 to perform feature fusion and feature encoding on the feature maps of different scales in the previous stage. Stage 5 generates a total of four feature maps of different scales, namely 36×128×128, 72×64×64, 144×32×32, and 288×16×16. Finally, the output module generates a heat map of facial key points of scale N×64×64. The heat map of each channel represents the predicted probability map of the current key point. The coordinates of N facial key points can be directly derived by selecting the maximum probability value.

[0117] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0118] Reference Figure 6 , shows a structural block diagram of an embodiment of a facial key point detection device of the present invention, which may specifically include the following modules:

[0119] A dimension-increasing module 601 is used to perform dimension-increasing processing on the image to be detected to generate a first scale feature map;

[0120] A conversion module 602, configured to convert the first scale feature map into a plurality of second scale feature maps in a preset multi-scale search space;

[0121] A fusion module 603 is used to fuse the second scale feature map to generate a face key point heat map, wherein the face key point heat map includes face key point probabilities;

[0122] The determination module 604 is used to determine the coordinates of the facial key points according to the facial key point probabilities.

[0123] In an optional embodiment of the present invention, the device further comprises:

[0124] An acquisition module, used for acquiring the original image;

[0125] The scaling module is used to scale the original image to form the image to be detected.

[0126] In an optional embodiment of the present invention, in the preset multi-scale search space, the conversion module 602 includes:

[0127] A path search submodule, used to determine a target path in the preset multi-scale search space, wherein the target path includes multiple stages;

[0128] The first conversion submodule is used to perform feature fusion and feature encoding on the first scale feature map in the stage to generate the second scale feature map.

[0129] In an optional embodiment of the present invention, the preset multi-scale search space includes a plurality of path selection modules, and the path search submodule includes:

[0130] A forward propagation unit, used to perform forward propagation on the path selection module to obtain an evaluation index;

[0131] A determination unit, used to determine a target evaluation indicator from the evaluation indicators;

[0132] The first combining unit is used to combine the path selection modules corresponding to the target evaluation indicators to generate the target path.

[0133] In an optional embodiment of the present invention, the stage corresponds to a resolution and a number of channels; the first conversion submodule includes:

[0134] an adjusting unit, configured to adjust the first scale feature map according to the number of channels;

[0135] A sampling unit, configured to sample the adjusted first scale feature map according to the resolution;

[0136] A calculation unit, used for performing weighted calculation on the sampled first scale feature map according to preset path parameters to generate a fused feature map;

[0137] A fusion unit is used to encode the fused feature map according to a preset candidate operation to generate the second scale feature map.

[0138] In an optional embodiment of the present invention, the preset candidate operation includes a jump connection and an input residual network; the preset candidate operation corresponds to an operation parameter; and the fusion unit includes:

[0139] An operation subunit, used for performing skip connection and inputting residual network operation on the fused feature map;

[0140] The calculation subunit is used to perform weighted summation on the operated fusion feature map according to the operation parameters to generate the second scale feature map.

[0141] In an optional embodiment of the present invention, the fusion module 603 includes:

[0142] A second conversion submodule, used for sampling the second scale feature map to generate a first resolution image;

[0143] A fusion submodule, used for fusing the first resolution image with the channel of the preset multi-scale search space as the dimensional direction;

[0144] The compression submodule is used to perform channel compression on the fused first-resolution image to generate the facial key point heat map.

[0145] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0146] An embodiment of the present invention further provides an electronic device, including:

[0147] A processor and a storage medium, wherein the storage medium stores a computer program executable by the processor, and when the electronic device is running, the processor executes the computer program to perform the method as described in any one of the embodiments of the present invention. The specific implementation method and technical effect are similar to those of the method embodiment, and will not be repeated here.

[0148] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the method as described in any one of the embodiments of the present invention. The specific implementation method and technical effect are similar to those of the method embodiment, and will not be repeated here.

[0149] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0150] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0151] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0154] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0155] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0156] The above is a detailed introduction to a facial key point detection method, device, electronic device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for a general technician in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for detecting key points of a face, characterized in that: include: Performing dimension-up processing on the image to be detected to generate a first-scale feature map; In a preset multi-scale search space, converting the first-scale feature map into a plurality of second-scale feature maps; fusing the second scale feature map to generate a facial key point heat map, wherein the facial key point heat map includes facial key point probabilities; Determining the coordinates of facial key points according to the facial key point probabilities; Among them, the step of converting the first scale feature map into multiple second scale feature maps in the preset multi-scale search space includes: determining a target path in the preset multi-scale search space, the target path including multiple stages; the stages correspond to resolution and number of channels; adjusting the first scale feature map according to the number of channels; sampling the adjusted first scale feature map according to the resolution; performing weighted calculation on the sampled first scale feature map according to preset path parameters to generate a fused feature map; encoding the fused feature map according to a preset candidate operation to generate the second scale feature map.

2. The method according to claim 1, characterized in that The method further comprises: Get the original image; The original image is scaled to form the image to be detected.

3. The method according to claim 1 or 2, characterized in that: The preset multi-scale search space includes a plurality of path selection modules, and the step of determining a target path in the preset multi-scale search space includes: Performing forward propagation on the path selection module to obtain an evaluation index; determining a target evaluation indicator from the evaluation indicators; The path selection modules corresponding to the target evaluation indicators are combined to generate the target path.

4. The method according to claim 1 or 2, characterized in that: The preset candidate operations include a jump connection and an input residual network; the preset candidate operations correspond to operation parameters; The step of encoding the fused feature map according to the preset candidate operation to generate the second scale feature map comprises: Performing skip connection and inputting residual network operation on the fused feature map; The weighted sum of the operated fusion feature map is performed according to the operation parameters to generate the second scale feature map.

5. The method according to claim 1 or 2, characterized in that: The step of fusing the second scale feature map to generate a facial key point heat map comprises: Sampling the second-scale feature map to generate a first-resolution image; fusing the first resolution image with the channel of the preset multi-scale search space as the dimensional direction; The fused first-resolution image is channel compressed to generate the facial key point heat map.

6. A facial key point detection device, characterized in that: include: A dimension-increasing module is used to perform dimension-increasing processing on the image to be detected to generate a first scale feature map; A conversion module, used to convert the first scale feature map into a plurality of second scale feature maps in a preset multi-scale search space; A fusion module, used for fusing the second scale feature map to generate a face key point heat map, wherein the face key point heat map includes face key point probabilities; A determination module, used to determine the coordinates of facial key points according to the facial key point probabilities; Among them, in the preset multi-scale search space, the conversion module includes: a path search submodule, which is used to determine the target path in the preset multi-scale search space, and the target path includes multiple stages; a first conversion submodule, which is used to perform feature fusion and feature encoding on the first scale feature map in the stage to generate the second scale feature map; the stage corresponds to a resolution and a number of channels; the first conversion submodule includes: an adjusting unit, configured to adjust the first scale feature map according to the number of channels; A sampling unit, configured to sample the adjusted first scale feature map according to the resolution; A calculation unit, used for performing weighted calculation on the sampled first scale feature map according to preset path parameters to generate a fused feature map; A fusion unit is used to encode the fused feature map according to a preset candidate operation to generate the second scale feature map.

7. An electronic device, characterized in that: The electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the facial key point detection method as described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the facial key point detection method according to any one of claims 1 to 5 are implemented.

Images