Facial nose shape recognition method, device, electronic device and medium

Abstract

The present application discloses a method, device, electronic device and medium for facial nose shape recognition. The method includes: obtaining a nose shape image to be processed; extracting texture features of the nose shape image to be processed to obtain a texture image of the nose shape image to be processed; fusing the texture image with the nose shape image to be processed to obtain a nose shape enhanced image of the nose shape image to be processed, wherein the nose shape enhanced image is a multi-channel image; inputting the nose shape enhanced image into a facial nose shape recognition model for processing to obtain a nose shape label vector of the nose shape image to be processed, wherein the nose shape label vector includes the predicted probability that the nose in the nose shape image to be processed belongs to different nose shapes; determining the nose shape corresponding to the nose shape image to be processed according to the nose shape label vector, and outputting the nose shape corresponding to the nose in the nose shape image to be processed.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a facial nose shape recognition method, device, electronic equipment and medium. Background Art

[0002] With the rapid development of mobile communication technology and the improvement of people's living standards, various smart terminals have been widely used in people's daily work and life. Among them, application software (Application, App) with functions such as beauty selfie and photo skin testing is also increasing. Many users hope to use some automatic detection apps to understand their facial conditions, such as nose shape recognition.

[0003] Since there are many types of nose shapes and many details in the division, it is more difficult to identify the nose shape. At present, most people use traditional image processing methods to identify the nose shape based on the texture of the nose shape, but the accuracy of this detection algorithm is not high enough. Summary of the invention

[0004] The present application provides a facial nose shape recognition method, device, electronic device and medium.

[0005] In a first aspect, a method for recognizing a facial nose shape is provided, comprising:

[0006] Obtain a nose shape image to be processed;

[0007] Extracting texture features of the nose shape image to be processed to obtain a texture image of the nose shape image to be processed;

[0008] Fusion of the texture image with the nose shape image to be processed to obtain a nose shape enhanced image of the nose shape image to be processed, wherein the nose shape enhanced image is a multi-channel image;

[0009] Inputting the nose shape enhanced image into a facial nose shape recognition model for processing, obtaining a nose shape label vector of the nose shape image to be processed, wherein the nose shape label vector includes a predicted probability that the nose in the nose shape image to be processed belongs to different nose shapes;

[0010] The nose shape corresponding to the nose shape image to be processed is determined according to the nose shape label vector, and the nose shape corresponding to the nose in the nose shape image to be processed is output.

[0011] In a second aspect, a facial nose shape recognition device is provided, comprising:

[0012] An acquisition module, used for acquiring a nose shape image to be processed;

[0013] A feature extraction module, used for extracting texture features of the nose shape image to be processed to obtain a texture image of the nose shape image to be processed;

[0014] A fusion module, used for fusing the texture image with the nose shape image to be processed to obtain a nose shape enhanced image of the nose shape image to be processed, wherein the nose shape enhanced image is a multi-channel image;

[0015] A processing module, used for inputting the nose shape enhanced image into a facial nose shape recognition model for processing, and obtaining a nose shape label vector of the nose shape image to be processed, wherein the nose shape label vector includes a predicted probability that the nose in the nose shape image to be processed belongs to different nose shapes;

[0016] The processing module is further used to determine the nose shape corresponding to the nose shape image to be processed according to the nose shape label vector, and output the nose shape corresponding to the nose in the nose shape image to be processed.

[0017] According to a third aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the first aspect and any possible implementation thereof.

[0018] In a fourth aspect, a computer storage medium is provided, wherein the computer storage medium stores one or more instructions, wherein the one or more instructions are suitable for being loaded by a processor and executing the steps of the above-mentioned first aspect and any possible implementation thereof.

[0019] The embodiment of the present application obtains a nose shape image to be processed; extracts texture features of the nose shape image to be processed to obtain a texture image of the nose shape image to be processed; fuses the texture image with the nose shape image to be processed to obtain a nose shape enhanced image of the nose shape image to be processed, wherein the nose shape enhanced image is a multi-channel image; inputs the nose shape enhanced image into a facial nose shape recognition model for processing to obtain a nose shape label vector of the nose shape image to be processed, wherein the nose shape label vector includes a predicted probability that the nose in the nose shape image to be processed belongs to different nose shapes; determines the nose shape corresponding to the nose shape image to be processed according to the nose shape label vector, and outputs the nose shape corresponding to the nose in the nose shape image to be processed. By adopting the texture image of the nose shape image to be processed and fusing it with the original nose shape image into a multi-channel nose shape enhanced image, and then inputting the multi-channel nose shape enhanced image into the facial nose shape recognition model for processing, the nose shape texture features of the nose shape image to be processed are enhanced; the facial nose shape recognition model can process the nose shape enhanced image to obtain the nose shape corresponding to the nose in the nose shape image to be processed, so as to make the nose shape recognition more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0021] Figure 1 A schematic diagram of a flow chart of a facial nose shape recognition method provided in an embodiment of the present application;

[0022] Figure 2 A schematic diagram of a nose area provided in an embodiment of the present application;

[0023] Figure 3A A flowchart of a method for training a facial nose shape recognition model provided in an embodiment of the present application;

[0024] Figure 3B A schematic diagram of the structure of a facial nose shape recognition model provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of the structure of a facial nose shape recognition device provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0028] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] The neural network (NN) involved in the embodiments of the present application is a complex network system formed by a large number of simple processing units (called neurons) widely connected to each other. It reflects many basic characteristics of human brain functions and is a highly complex nonlinear dynamic learning system. Neural networks have large-scale parallel, distributed storage and processing, self-organization, self-adaptation and self-learning capabilities, and are particularly suitable for processing imprecise and fuzzy information processing problems that require simultaneous consideration of many factors and conditions.

[0031] Convolutional Neural Networks (CNN) are a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0032] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0033] See also Figure 1 , Figure 1 : is a flow chart of a facial nose shape recognition method provided in an embodiment of the present application. The method may include:

[0034] 101. Obtain a nose shape image to be processed.

[0035] The execution subject of the embodiment of the present application may be a facial nose recognition device, which may be an electronic device. In a specific implementation, the electronic device may be a terminal, which may also be referred to as a terminal device, including but not limited to other portable devices such as mobile phones, laptop computers or tablet computers with touch-sensitive surfaces (e.g., touch screen displays and / or touch pads). It should also be understood that in some embodiments, the above-mentioned device is not a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., touch screen displays and / or touch pads).

[0036] The nose shape image to be processed may be an image of the nose area of ​​the face.

[0037] In one implementation, the above step 101 may specifically include:

[0038] 11. Get the face image to be processed.

[0039] 12. Perform facial key point detection on the above-mentioned face image to be processed to determine the coordinates of the facial key points in the above-mentioned face image to be processed.

[0040] 13. Based on the coordinates of the facial key points in the facial image to be processed, the nose shape area of ​​the facial image to be processed is intercepted as the nose shape image to be processed.

[0041] The above-mentioned face image to be processed is an image containing a face. Specifically, the acquired face image to be processed can be pre-processed such as cropping to obtain a standardized face image to be processed, and then the above-mentioned face key point detection is performed to determine the coordinates of the face key points. In the embodiment of the present application, any face key point detection model or algorithm can be used to perform key point detection, for example, 68 key points of the face contour can be located. Optionally, each key point detected has a corresponding key point identifier (number).

[0042] After determining the above-mentioned facial key point coordinates, the key point coordinates of various parts of the face can be referred to, such as the key point coordinates representing the center of the nose, to divide the area where the nose is located, that is, to capture the nose shape area of ​​the face image to be processed.

[0043] Further optionally, the above step 13 specifically includes:

[0044] According to the preset key point identifier, the coordinates of the nose center key point corresponding to the preset key point identifier are obtained from the coordinates of the face key points;

[0045] According to the preset width threshold, the preset height threshold and the coordinates of the nose center key point, the nose shape area of ​​the face image to be processed is determined and intercepted as the nose shape image to be processed.

[0046] Specifically, the above-mentioned preset key point identifier can be a preset identifier of the center point of the nose in the face. First, the key point position where the preset key point identifier is located is determined, and then the nose shape area is divided according to the preset width threshold and the preset height threshold: the preset width threshold is used as the width, the preset height threshold is used as the length, and the rectangular nose shape area is intercepted with the center key point of the nose as the center, so as to obtain the nose shape image to be processed.

[0047] See also Figure 2 A schematic diagram of a nose-shaped area is shown in FIG. Figure 2 As shown, multiple facial key points detected in the face image to be processed are displayed, and each key point corresponds to a digital number as a key point identifier. Figure 2 The rectangular area in is the nose shape area, which is obtained by intercepting a rectangular area with a preset length of 2r and a width of 2r with the nose center key point (the 31st face key point) as the center in the above method.

[0048] The above method can accurately obtain the nose shape image from the image to be processed, and obtain the nose shape image to be processed of standard size, which is suitable for the facial nose shape recognition model, so that the facial nose shape recognition model can accurately perform nose shape recognition processing on the nose shape image to be processed, so as to obtain a nose shape recognition result with higher accuracy.

[0049] 102. Extract texture features of the nose shape image to be processed to obtain a texture image of the nose shape image to be processed.

[0050] The texture features of the nose image to be processed can be understood as the edge contour information of the nose, that is, the edge information of the nose edge, nostril, etc. in the nose area. Specifically, a network model can be used to extract features of the nose image to be processed to obtain a texture image of the nose image to be processed, which can be a binary nose edge contour map of the same size as the original nose image to be processed.

[0051] In an optional implementation, the Sobel operator may be used to extract the contour information of the nose to obtain the above texture image.

[0052] The Sobel operator is a discrete differentiation operator, which is used to calculate the approximate gradient of the grayscale of an image. The larger the gradient, the more likely it is an edge. The Sobel operator combines Gaussian smoothing and differential derivation, and is also called a first-order differential operator. The derivative operator is derived in both the horizontal and vertical directions to obtain the gradient image of the image in the X and Y directions. The embodiment of the present application does not limit the algorithm for extracting texture images.

[0053] 103. Fusing the texture image with the nose shape image to be processed to obtain a nose shape enhanced image of the nose shape image to be processed, wherein the nose shape enhanced image is a multi-channel image.

[0054] When the texture image of the nose shape image to be processed is obtained, the texture image can be fused with the nose shape image to be processed so that the texture feature data represented by the texture image can be used as the data of a channel of the nose shape image to be processed, and the channel can be called a nose shape texture channel.

[0055] The general RGB format image has data of three channels: R, G, and B. The RGB color mode is a color standard in the industry. It obtains various colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other.

[0056] In the embodiment of the present application, the nose shape image to be processed is fused with its texture image, that is, the extracted texture feature data is used as the fourth channel data, and the original image information is added to obtain the above-mentioned nose shape enhanced image. It can be understood that the fourth channel is a binary nose edge contour map of the same size as the nose shape image to be processed. At this time, the nose shape enhanced image is a multi-channel image, including at least three channels of R, G, and B and a channel of texture feature data. The nose shape texture features of the nose shape image to be processed are enhanced through the above steps, and then step 104 is executed.

[0057] 104. Input the nose shape enhanced image into a facial nose shape recognition model for processing to obtain a nose shape label vector of the nose shape image to be processed, wherein the nose shape label vector includes a predicted probability that the nose in the nose shape image to be processed belongs to different nose shapes.

[0058] The above-mentioned facial nose shape recognition model is a pre-trained network model, which can process the above-mentioned multi-channel nose shape enhanced image. The nose shape label vector of the nose shape image to be processed can be obtained through the facial nose shape recognition model. The nose shape label vector is a probability distribution, including the predicted probability that the nose in the above-mentioned nose shape image to be processed belongs to different nose shapes, for example, expressed as P = [P 1 ,P 2 ,P 3 ,P 4 ,P 5 ,P 6 ], where element P 1 -P 6 They respectively represent the predicted probabilities that the nose in the nose shape image to be processed belongs to six different nose shapes.

[0059] In an optional implementation, the above step 104 includes:

[0060] 41. Perform feature extraction on the nose shape enhanced image based on the facial nose shape recognition model to obtain a feature map set of the nose shape enhanced image, wherein the feature map set includes multiple category feature maps corresponding to different nose shapes and a texture feature map, wherein the texture feature map is generated based on the texture image;

[0061] 42. Fusing the above-mentioned multiple category feature maps with the above-mentioned texture feature map respectively to obtain multiple target feature maps of the above-mentioned nose shape enhanced image;

[0062] 43. Generate the nose shape label vector of the nose shape image to be processed according to the multiple target feature maps.

[0063] Specifically, the trained facial nose shape recognition model can perform image feature extraction to obtain a feature map set of the nose shape enhanced image, including multiple category feature maps corresponding to different nose shapes and a texture feature map generated based on the above texture image.

[0064] For example, for the nose shape enhanced image I∈R 4×40×40 As the input image of the model, extract the deep feature F∈R corresponding to the input image T*W*H , where T represents the number of channels of the above-mentioned deep features, T is equal to the number of nose types n plus 1, and W and H represent the width and height of each feature map. Among the extracted n+1 feature maps, a texture image F′ is included, which is the feature map corresponding to the nose texture channel of the original input image. The category feature map for each nose type can be expressed as F i ∈R W×H , i=(1,2,...,n), then the deep feature is F={F 1 ,F 2 ,...,F n ,F′}.

[0065] In the facial nose shape recognition model, the acquisition of feature map F is first a process of convolution kernel feature extraction, that is, convolution operation. Each convolution kernel convolution operation will obtain a feature map. The number of feature maps will be obtained as many as the number of convolution kernels. In the feature extraction structure of the facial nose shape recognition model, the number of convolution kernels in the last convolution operation is set to n+1, where n represents the number of nose shapes. That is, the trained facial nose shape recognition model can obtain n category feature maps (F 1 ,F 2 ,...,F n ), corresponding to different nose shape features, the remaining convolution kernel is used to obtain a feature map representing texture features (texture image F′), so that the entire depth adjustment map F = {F 1 ,F 2 ,...,F n ,F′}.

[0066] After obtaining the feature map, the above multiple category feature maps F iBy fusing them with the texture image F′ respectively, multiple target feature maps of the nose shape enhanced image can be obtained. The multiple target feature maps obtained at this time still correspond to the nose shape corresponding to the original image, and the predicted probability of the nose shape can be obtained through the above multiple target feature maps, that is, the nose shape label vector of the nose shape image to be processed is generated. Specifically, in the facial nose shape recognition model, the feature map can be converted into the result of type probability distribution through the fully connected layer and the Softmax layer. The input of the fully connected layer is regarded as the feature map extracted by the network from the input nose shape enhanced image, and one feature map corresponds to a point in the multidimensional space. In the fully connected layer, the preset weight matrix is ​​multiplied by the vector representation of the input feature map, and the preset bias parameter is added to map n (-∞, +∞) real numbers to K (-∞, +∞) real numbers (the result can be understood as a score). Softmax then maps the K (-∞, +∞) real numbers to K (0, 1) real numbers (the result can be understood as a probability), while ensuring that their sum is 1, where K = the number of nose type categories, thereby obtaining the predicted probability of each nose type.

[0067] Specifically, in order to further enhance the nose texture feature, the texture image F′ needs to be compared with the feature map F i Perform fusion operation.

[0068] Specifically, the above step 42 may include:

[0069] The above-mentioned multiple category feature maps are respectively matrix-multiplied and weighted with the above-mentioned texture feature map to obtain multiple target feature maps of the above-mentioned nose shape enhanced image.

[0070] The above feature map data can be expressed in matrix form. Each category feature map can be multiplied by the above texture feature map to be expressed as F i *F′, and use the preset weight parameters for weighted processing to obtain the corresponding target feature map.

[0071] For example, the final target feature map can be expressed as:

[0072] F={F 1 *α*F′,F 2 *β*F′,...,F n *θ*F′};

[0073] The parameters such as α, β, θ are the preset weight parameters mentioned above, which can be the self-learning parameters in the model training.

[0074] 105. Determine the nose shape corresponding to the nose shape image to be processed according to the nose shape label vector, and output the nose shape corresponding to the nose in the nose shape image to be processed.

[0075] According to the predicted probability in the nose shape label vector, the nose shape corresponding to the nose shape image to be processed can be determined. Specifically, it can be determined that the nose shape corresponding to the largest predicted probability is the nose shape corresponding to the nose shape image to be processed, and the nose shape corresponding to the nose in the nose shape image to be processed can be output. Optionally, when there are multiple maximum predicted probabilities (the values ​​of the predicted probabilities are the same), the nose shape corresponding to each maximum predicted probability can be output. The embodiment of the present application does not limit other contents and output forms of the output results.

[0076] See also Figure 3A , Figure 3A : is a flowchart of a method for training a facial nose shape recognition model provided by an embodiment of the present application. The method may include:

[0077] 301. Acquire multiple sample nose shape images, wherein each sample nose shape image is annotated with a sample nose shape label vector, and the sample nose shape label vector is used to indicate the probability that the nose in the sample nose shape image belongs to a different nose shape;

[0078] 302. Extracting a texture image of each sample nose shape image; fusing each sample nose shape image with a texture image corresponding to each sample nose shape image to generate a sample nose shape enhanced image corresponding to each sample nose shape image, so as to obtain a plurality of sample nose shape enhanced images, wherein the sample nose shape enhanced images are multi-channel images;

[0079] 303. Based on the multiple sample nose shape enhanced images, train the network model to obtain a facial nose shape recognition model.

[0080] The executing entity of the embodiment of the present application can be a facial nose shape recognition model training device, or the above-mentioned facial nose shape recognition device, or an electronic device. In a specific implementation, the above-mentioned electronic device can be a terminal, which can also be called a terminal device, including but not limited to a desktop computer with a touch-sensitive surface (for example, a touch screen display and / or a touchpad).

[0081] The above-mentioned multiple sample nose shape images can be images of different nose regions of different faces, or can be obtained by intercepting from face images. Key point detection can be performed on the sample face image first, and then the sample nose shape image can be intercepted. Since the model input needs to be normalized, in order not to destroy the spatial structure of the nose shape, so that the model can better learn the spatial feature information of the nose shape contour features, when intercepting the nose shape image, the nose center key point can be used as the center point H(x, y), and the width and height can be preset values ​​such as 2r to intercept the nose area as the sample nose shape image. This step can be referred to Figure 1 The specific description of step 101 in the illustrated embodiment will not be repeated here.

[0082] In order to better learn the characteristic information between different nose shapes, when annotating the nose shape of the sample nose shape image, the one-hot method can be used for annotation, that is, the above sample nose shape label vector is used, where each value represents the probability of belonging to a certain nose shape. For example, when there are 6 types of nose shapes, when annotating the sample, the sample nose shape label vector P = [0,0,0.8,0.2,0,0] means that the sample nose shape image has a probability of 0.8 belonging to the third type of nose shape, and a probability of 0.2 belonging to the fourth type of nose shape...

[0083] Ne-hot encoding is the process of converting categorical variables into a form that is easy to use for machine learning algorithms. General annotation methods such as binary annotation, in multi-classification problems, each sample corresponds to only one category (i.e., only the value of the corresponding feature is 1, and the value of the rest is 0). The method provided in the embodiment of the present application annotates the probability of belonging to a certain category, so that the classification prediction result obtained by the model is also expressed as the probability of belonging to a certain category, so that when performing loss functions (such as cross entropy loss) or accuracy calculations, the calculation complexity is simplified; and the annotated values ​​can also be understood as the similarity with a certain category, so that the distance calculation between features is more reasonable, and the accuracy of the model is improved.

[0084] Among them, the above step 52 and the above step 53 Figure 1 The specific methods in step 102-step 103 in the illustrated embodiment are similar, and reference may be made to the aforementioned specific description, which will not be repeated here.

[0085] Optionally, before training the network model based on the multiple sample nose shape enhanced images, the method further includes:

[0086] Subtract the mean value of the data of the plurality of sample nose-enhanced images on the target channel from the data of the target sample nose-enhanced image on the target channel to obtain the de-averaged data of the target sample nose-enhanced image on the target channel, wherein the target sample nose-enhanced image is any one of the plurality of sample nose-enhanced images, and the target channel is any one of the plurality of channels;

[0087] Calculating the variance of the de-averaged data of the plurality of sample nose-enhanced images on the target channel;

[0088] The data of the target sample nose shape enhanced image on the target channel is updated to the ratio of the de-averaged data of the target sample nose shape enhanced image on the target channel to the variance, so as to update the data of the multiple sample nose shape enhanced images on the multiple channels.

[0089] Before training, the sample nose-enhanced images can be normalized, which can effectively accelerate the convergence of weight parameters and better find the optimal solution.

[0090] For the convenience of description, the target sample nose shape enhanced image I represents any one of the above sample nose shape enhanced images, and the target channel represents any channel in the above multi-channels. The target sample nose shape enhanced image is a four-channel input image, which can be specifically represented as I W*H*4 The target sample nose shape enhanced image can be firstly de-meaned, and the data of the target sample nose shape enhanced image on the target channel is subtracted from the mean value of the data of multiple sample nose shape enhanced images on the target channel, that is, the following formula can be used specifically:

[0091]

[0092] Where i = 1, 2, 3, ..., N, N is the total number of sample nose-enhanced images, I R ,I G ,I B ,I S Respectively represent the data of the four channels (R, G, B, S) of the target sample nose shape enhanced image I. i,R ,I i,G ,I i,B ,I i,S Respectively represent the values ​​of the four channels of the i-th sample nose-enhanced image.

[0093] Through the above de-averaging process, the de-averaged data I(I′) of the target sample nose-shaped enhanced image can be obtained. R ,I′ G ,I′ B ,I′ S ).

[0094] Furthermore, the variance of the above averaged data is calculated, and the following formula can be used:

[0095]

[0096] Furthermore, the ratio of the above averaged data to the above variance on each channel can be obtained:

[0097]

[0098] The data of the target sample nose-enhanced image on the target channel can be updated to the ratio of the averaged data to the variance on the target channel, and the final data I(I′) of each sample nose-enhanced image on four channels can be obtained. R ,I′ G ,I′ B ,I′ S), that is, performing the above steps on each channel of each sample nose shape enhanced image, so that the data of multiple sample nose shape enhanced images on the above multiple channels can be updated.

[0099] By removing the mean and adjusting the variance, the sample data, i.e., the sample nose-enhanced image, conforms to the normal distribution, which can effectively accelerate the convergence of the weight parameters during model training and better find the optimal solution.

[0100] In the facial nose shape recognition model in the embodiment of the present application, the convolutional neural network model of the feature extraction model may include: VGG, ResNet, Incention and other network structures. Taking into account the characteristics of the nose shape image and the applicability of the recognition system, a small-model convolutional neural network can be selected as the basic network architecture, and convolution kernels of different sizes can be used to obtain different receptive fields and obtain multi-scale feature fusion. The average pooling layer is used to replace the fully connected layer in the convolutional neural network to effectively reduce the parameters of the model. Any network architecture can be used for replacement, and there is no restriction on the feature extraction structure.

[0101] Figure 3B A schematic diagram of the structure of a facial nose shape recognition model provided in an embodiment of the present application is shown in FIG. Figure 3B As shown, the facial nose shape recognition model may include a CNN network (feature extraction layer) for feature extraction, a feature fusion layer and an average pooling layer, wherein:

[0102] The sample nose shape enhancement image is extracted through the feature extraction layer with feature map F = {F 1 ,F 2 ,...,F n ,F′}, the number of convolution kernels in the feature extraction layer is n+1, n is the number of category feature maps, and the output feature map includes n of the above category feature maps and a texture image F′ representing the texture features of the nose shape.

[0103] In the feature extraction structure of the facial nose shape recognition model, the number of convolution kernels in the final convolution operation is set to n+1, where n represents the number of nose shapes, that is, n convolution kernels are used to obtain n sample category feature maps (F 1 ,F 2 ,...,F n), corresponding to n nose shape features respectively, and the remaining convolution kernel is used to obtain a feature map (texture image F′) representing the sample texture feature. Among them, the four-channel data [R, G, B, V] (here is an example of sorting) of the sample nose shape enhanced image includes three-channel data of R, G, and B and the fourth channel data of the texture feature data V. During the convolution operation, because the sorting of these four channels is unchanged, when visualizing the feature map in practice, it can be found that the last feature map can best feedback the texture feature data V. Therefore, when setting the network, the last feature map is the feature map of the sample texture feature, that is, it is found by the channel visualization feature map during the feature extraction process, so that the model can be set up, including the setting of the above-mentioned convolution kernel, and the required feature map can be learned through the convolution operation, that is, the code. Further, the extracted feature maps are feature fused in the feature fusion layer, and each of the sample nose shape images is fused with the texture image F′ corresponding to each of the sample nose shape images, to generate a sample nose shape enhancement image corresponding to each of the sample nose shape images, so as to obtain a plurality of sample nose shape enhancement images. For example, assuming that the previous convolution layer includes 32 convolution kernels, 32 feature maps are extracted, and one of the texture images is fused with the remaining 31 feature maps respectively, to obtain 31 sample nose shape enhancement images;

[0104] The obtained sample nose shape enhancement image is then downsampled through the average pooling layer to restore it to the size of the original sample nose shape enhancement image. In the subsequent classification process, the softmax layer can be used to map the outputs of multiple neurons to the (0,1) interval, which can be understood as probability to perform nose shape classification. The facial nose shape recognition model can use a preset loss function for model training. Figure 3B The KL-loss shown in is the KL divergence loss function. According to the probability distribution of the labels, the KL divergence loss function can be used to optimize the model in the direction of minimizing the KL loss value during training, so as to better learn the similar features between nose shape categories.

[0105] Specifically, assume that the nose shape label vector output by the facial nose shape recognition model, i.e., the predicted probability distribution, is P = [P 1 ,P 2 ,P 3 ,P 4 ,P 5 ,P 6 ], and the probability distribution of the sample nose label vector, i.e. the true label, is T = [T 1 ,T 2 ,T 3 ,T 4 ,T 5 ,T 6 ], then the loss function of the facial nose recognition model can be:

[0106]

[0107] Where Z is the total number of sample nose shape enhanced images, and n represents the number of nose shape categories.

[0108] In the embodiment of the present application, the Adam algorithm can be used to optimize the model parameters, and training is performed according to the preset number of iterations, initial learning rate, learning rate decay parameter and weight decay parameter until the network model converges and the training ends. For example, the number of iterations is set to 500 times, the initialization learning rate is set to 0.001, the weight decay is set to 0.0005, and the learning rate decays to 1 / 10 of the original after every 50 iterations. Different algorithms can be selected as needed to optimize the model parameters. The embodiment of the present application does not limit this.

[0109] After the training is completed, the trained face nose recognition model can be obtained and applied to Figure 1 Facial nose shape recognition in the illustrated embodiment.

[0110] This application aims to enhance the texture features of nose shape by combining the advantages of image processing technology and convolutional neural network, with the characteristics of small inter-class differences and large intra-class differences between nose shape features. A four-channel nose shape image containing texture image data is constructed to enhance the model's learning of nose shape features; a nose shape label vector is constructed so that the model can better learn similar features between nose shapes, thereby reducing the impact of small inter-class differences and improving the accuracy of model recognition.

[0111] Based on the description of the above-mentioned facial nose shape recognition method embodiment, the present application embodiment also discloses a facial nose shape recognition device. Figure 4 The facial nose shape recognition device 400 comprises:

[0112] An acquisition module 410 is used to acquire a nose shape image to be processed;

[0113] A feature extraction module 420 is used to extract texture features of the nose shape image to be processed to obtain a texture image of the nose shape image to be processed;

[0114] A fusion module 430 is used to fuse the texture image with the nose shape image to be processed to obtain a nose shape enhanced image of the nose shape image to be processed, wherein the nose shape enhanced image is a multi-channel image;

[0115] A processing module 440 is used to input the nose shape enhanced image into a facial nose shape recognition model for processing, and obtain a nose shape label vector of the nose shape image to be processed, wherein the nose shape label vector includes a predicted probability that the nose in the nose shape image to be processed belongs to different nose shapes;

[0116] The processing module 440 is further configured to determine the nose shape corresponding to the nose shape image to be processed according to the nose shape label vector, and output the nose shape corresponding to the nose in the nose shape image to be processed.

[0117] According to one embodiment of the present application, Figure 1 and Figure 3A Each step involved in the method shown can be performed by Figure 4 The operations performed by various modules in the facial nose shape recognition device 400 are not described in detail here.

[0118] The facial nose shape recognition device 400 in the embodiment of the present application can obtain a nose shape image to be processed; perform texture feature extraction on the above-mentioned nose shape image to be processed to obtain a texture image of the above-mentioned nose shape image to be processed; fuse the above-mentioned texture image with the above-mentioned nose shape image to be processed to obtain a nose shape enhanced image of the above-mentioned nose shape image to be processed, and the above-mentioned nose shape enhanced image is a multi-channel image; input the above-mentioned nose shape enhanced image into the facial nose shape recognition model for processing, and obtain a nose shape label vector of the above-mentioned nose shape image to be processed, and the above-mentioned nose shape label vector includes the above-mentioned nose shape image to be processed. The predicted probability that the nose in the image belongs to different nose shapes; the nose shape corresponding to the nose shape image to be processed is determined according to the nose shape label vector, and the nose shape corresponding to the nose in the nose shape image to be processed is output; the texture image of the nose shape image to be processed is fused with the original nose shape image to form a multi-channel nose shape enhanced image, and then the multi-channel nose shape enhanced image is input into the facial nose shape recognition model for processing, thereby enhancing the nose shape texture features of the nose shape image to be processed; the facial nose shape recognition model can process the nose shape enhanced image to obtain the nose shape corresponding to the nose in the nose shape image to be processed, which can make the nose shape recognition more accurate.

[0119] Based on the description of the above method embodiment and device embodiment, the present application embodiment also provides an electronic device. Figure 5 The electronic device 500 at least includes a processor 501, an input device 502, an output device 503, and a computer storage medium 504. The processor 501, the input device 502, the output device 503, and the computer storage medium 504 in the electronic device may be connected via a bus or other means.

[0120] The computer storage medium 504 can be stored in the memory of the electronic device. The computer storage medium 504 is used to store a computer program. The computer program includes program instructions. The processor 501 is used to execute the program instructions stored in the computer storage medium 504. The processor 501 (or CPU (Central Processing Unit)) is the computing core and control core of the electronic device, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding function; in one embodiment, the processor 501 of the embodiment of the present application can be used to perform a series of processing, including such as Figure 1 and Figure 3A Methods in the embodiments shown, etc.

[0121] The embodiment of the present application also provides a computer storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It is understandable that the computer storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The computer storage medium provides a storage space that stores the operating system of the electronic device. In addition, one or more instructions suitable for being loaded and executed by the processor 501 are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor.

[0122] In one embodiment, the processor 501 may load and execute one or more instructions stored in a computer storage medium to implement the corresponding steps in the above embodiment; in a specific implementation, the processor 501 may load and execute one or more instructions stored in a computer storage medium to implement the corresponding steps in the above embodiment; Figure 1 and / or Figure 3A Any steps in the method will not be repeated here.

[0123] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0124] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the module is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0125] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0126] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a magnetic disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

Claims

1. A facial nose shape recognition method, characterized in that: include: Obtain a nose shape image to be processed; Extracting texture features of the nose shape image to be processed to obtain a texture image of the nose shape image to be processed; Fusion the texture image with the nose shape image to be processed to obtain a nose shape enhanced image of the nose shape image to be processed, wherein the nose shape enhanced image is a multi-channel image; Inputting the nose shape enhanced image into a facial nose shape recognition model for processing, obtaining a nose shape label vector of the nose shape image to be processed, wherein the nose shape label vector includes a predicted probability that the nose in the nose shape image to be processed belongs to different nose shapes; The step of inputting the nose shape enhanced image into a facial nose shape recognition model for processing to obtain a nose shape label vector of the nose shape image to be processed comprises: extracting features of the nose shape enhanced image based on the facial nose shape recognition model to obtain a feature map set of the nose shape enhanced image, wherein the feature map set comprises a plurality of category feature maps corresponding to different nose shapes, and a texture feature map, wherein the texture feature map is generated based on the texture image; The multiple category feature maps are respectively fused with the texture feature map to obtain multiple target feature maps of the nose shape enhanced image; and a nose shape label vector of the nose shape image to be processed is generated according to the multiple target feature maps; The nose shape corresponding to the nose shape image to be processed is determined according to the nose shape label vector, and the nose shape corresponding to the nose in the nose shape image to be processed is output.

2. The facial nose shape recognition method according to claim 1, characterized in that: The step of fusing the multiple category feature maps with the texture feature map to obtain multiple target feature maps of the nose shape enhanced image includes: The multiple category feature maps are matrix-multiplied and weighted with the texture feature map respectively to obtain multiple target feature maps of the nose shape enhanced image.

3. The facial nose shape recognition method according to claim 1 or 2, characterized in that: The step of obtaining the nose shape image to be processed comprises: Obtain the face image to be processed; Performing facial key point detection on the face image to be processed to determine the coordinates of the facial key points in the face image to be processed; Based on the coordinates of the facial key points in the face image to be processed, a nose shape area of ​​the face image to be processed is intercepted as the nose shape image to be processed.

4. The facial nose shape recognition method according to claim 3, characterized in that: The method of intercepting the nose shape region of the face image to be processed as the nose shape image to be processed based on the coordinates of the face key points in the face image to be processed includes: According to the preset key point identifier, obtaining the nose center key point coordinates corresponding to the preset key point identifier from the face key point coordinates; According to the preset width threshold, the preset height threshold and the coordinates of the nose center key point, the nose shape area of ​​the face image to be processed is determined and intercepted as the nose shape image to be processed.

5. The facial nose shape recognition method according to claim 1, characterized in that: The facial nose shape recognition model is obtained through training, and the training method of the facial nose shape recognition model includes: Acquire multiple sample nose shape images, wherein each sample nose shape image is annotated with a sample nose shape label vector, and the sample nose shape label vector is used to indicate the probability that the nose in the sample nose shape image belongs to a different nose shape; Extracting a texture image of each sample nose shape image; fusing each sample nose shape image with a texture image corresponding to each sample nose shape image to generate a sample nose shape enhanced image corresponding to each sample nose shape image, so as to obtain a plurality of sample nose shape enhanced images, wherein the sample nose shape enhanced images are multi-channel images; Based on the multiple sample nose shape enhanced images, the network model is trained to obtain the facial nose shape recognition model.

6. The facial nose shape recognition method according to claim 5, characterized in that: Before training the network model based on the multiple sample nose shape enhanced images, the method further includes: Subtract the mean of the data of the multiple sample nose-enhanced images on the target channel from the data of the target sample nose-enhanced image on the target channel to obtain the de-averaged data of the target sample nose-enhanced image on the target channel, wherein the target sample nose-enhanced image is any sample nose-enhanced image among the multiple sample nose-enhanced images, and the target channel is any channel among the multiple channels; Calculate the variance of the de-averaged data of the plurality of sample nose-enhanced images on the target channel; The data of the target sample nose shape enhanced image on the target channel is updated as the ratio of the de-averaged data of the target sample nose shape enhanced image on the target channel to the variance, so as to update the data of the multiple sample nose shape enhanced images on the multiple channels.

7. A facial nose shape recognition device, characterized in that: include: An acquisition module, used for acquiring a nose shape image to be processed; A feature extraction module, used for extracting texture features of the nose shape image to be processed to obtain a texture image of the nose shape image to be processed; A fusion module, used for fusing the texture image with the nose shape image to be processed to obtain a nose shape enhanced image of the nose shape image to be processed, wherein the nose shape enhanced image is a multi-channel image; A processing module, used for inputting the nose shape enhanced image into a facial nose shape recognition model for processing, and obtaining a nose shape label vector of the nose shape image to be processed, wherein the nose shape label vector includes a predicted probability that the nose in the nose shape image to be processed belongs to different nose shapes; The processing module is specifically used to extract features from the nose shape enhanced image based on the facial nose shape recognition model to obtain a feature map set of the nose shape enhanced image, wherein the feature map set includes a plurality of category feature maps corresponding to different nose shapes and a texture feature map, wherein the texture feature map is generated based on the texture image; The multiple category feature maps are respectively fused with the texture feature map to obtain multiple target feature maps of the nose shape enhanced image; and a nose shape label vector of the nose shape image to be processed is generated according to the multiple target feature maps; The processing module is further used to determine the nose shape corresponding to the nose shape image to be processed according to the nose shape label vector, and output the nose shape corresponding to the nose in the nose shape image to be processed.

8. An electronic device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the facial nose shape recognition method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the processor executes the steps of the facial nose shape recognition method according to any one of claims 1 to 6.

Images