Wrinkle segmentation method, device and readable medium based on gradient saliency perception

Abstract

The present invention discloses a wrinkle segmentation method, device and readable medium based on gradient saliency perception, which relates to the field of image processing, including: constructing a facial wrinkle data set and a facial wrinkle segmentation network; inputting the original facial image in the facial wrinkle data set into the facial wrinkle segmentation network to obtain a corresponding wrinkle segmentation prediction map, calculating the saliency of each pixel in the original facial image, further calculating the weight of each pixel in the original facial image, constructing a gradient saliency weighted loss function based on the pixel value of each pixel in the original facial image and its weight and the pixel value of the corresponding pixel in the wrinkle annotation mask map corresponding to the original facial image, training the facial wrinkle segmentation network based on the gradient saliency weighted loss function, and obtaining a trained facial wrinkle segmentation network. The present invention solves the problem of excessive or incorrect segmentation of wrinkles in facial images in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a wrinkle segmentation method, device and readable medium based on gradient saliency perception. Background Art

[0002] With the rapid development of science and technology and people's pursuit of beauty, facial wrinkle segmentation technology has been widely used in many new fields. For example, in the field of medical diagnosis, segmenting facial wrinkles can help doctors assess the patient's skin health and aging degree, so as to develop personalized treatment plans. For another example, in the field of photo beautification, according to the different types and degrees of facial wrinkles, the corresponding beautification effects are applied to improve the quality of photos and user satisfaction. In addition, in the field of face recognition, facial wrinkles, as a unique biometric feature, are widely used in user identity authentication to improve the accuracy and security of recognition. In the field of health monitoring, wrinkle segmentation can be used to assess the physiological and mental health of individuals.

[0003] There are general deficiencies in the wrinkle segmentation technologies currently available on the market. Existing technologies usually do not set reasonable weights for wrinkle pixels on the face. Wrinkle pixel weights that are too high cause the model to focus too much on wrinkle details during training, and may mistake very small or insignificant features for wrinkles, resulting in over-segmentation. If wrinkle pixel weights are too low, the model may be more inclined to incorrectly segment wrinkles as background during training, resulting in a lower recall rate, which means that the model may miss some real wrinkles. Existing technologies also usually do not perform weighted perception of complex texture areas in facial images, resulting in incomplete segmentation of complex texture areas. Summary of the invention

[0004] The purpose of this application is to propose a wrinkle segmentation method, device and readable medium based on gradient saliency perception in response to the above-mentioned technical problems.

[0005] In a first aspect, the present invention provides a wrinkle segmentation method based on gradient saliency perception, comprising the following steps:

[0006] Obtain the original face image and annotate the wrinkle pixels to generate the corresponding wrinkle annotation mask map, and construct a face wrinkle dataset based on the original face image and its corresponding wrinkle annotation mask map;

[0007] Construct a facial wrinkle segmentation network based on an improved UNet deep backbone network. The facial wrinkle segmentation network adds a multi-scale attention gate module on the basis of the traditional UNet deep backbone network.

[0008] The original face image in the face wrinkle data set is input into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map, the significance of each pixel in the original face image in the face wrinkle data set is calculated, and the weight of each pixel in the original face image is obtained according to the significance calculation of each pixel in the original face image, and a gradient significance weighted loss function is constructed based on the pixel value of each pixel in the original face image and its weight and the pixel value of the corresponding pixel in the wrinkle annotation mask map corresponding to the original face image, and the face wrinkle segmentation network is trained based on the gradient significance weighted loss function to obtain a trained face wrinkle segmentation network;

[0009] A face image to be processed is obtained and input into a trained face wrinkle segmentation network to obtain a wrinkle segmentation prediction map corresponding to the face image to be processed.

[0010] Preferably, the facial wrinkle segmentation network includes a contraction path, an expansion path and a multi-scale attention gate module, the contraction path includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module, the expansion path includes a fifth convolution module, a sixth convolution module and a seventh convolution module, the first convolution module, the second convolution module, the third convolution module, the fourth convolution module, the fifth convolution module, the sixth convolution module and the seventh convolution module each include two first convolution layers with a convolution kernel size of 3×3, each first convolution layer with a convolution kernel size of 3×3 is connected to a first ReLU activation function layer, and the first convolution module and the second convolution module, the second convolution module and the third convolution module, and the third convolution module and the fourth convolution module all include A pooling kernel of 2×2 is the maximum pooling layer; the output features of the fourth convolution module are upsampled and input into the multi-scale attention gate module together with the output features of the third convolution module; the output features of the fourth convolution module are spliced ​​with the output features of the multi-scale attention gate module after upsampling and input into the fifth convolution module; the output features of the fifth convolution module are upsampled and spliced ​​with the output features of the second convolution module and input into the sixth convolution module; the output features of the sixth convolution module are upsampled and spliced ​​with the output features of the first convolution module and input into the seventh convolution module; the upsampling uses a deconvolution layer with a convolution kernel size of 2×2; the output features of the seventh convolution module pass through a second convolution layer with a convolution kernel size of 1×1, and output a wrinkle segmentation prediction map.

[0011] Preferably, the multi-scale attention gate module includes a third convolution layer, a fourth convolution layer, a second ReLU activation function layer, a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer and a Sigmoid activation function layer, the convolution kernel sizes of the third convolution layer, the fourth convolution layer and the eighth convolution layer are all 1×1, the convolution kernel sizes of the fifth convolution layer, the sixth convolution layer and the seventh convolution layer are 3×3, 5×5 and 7×7 respectively, the output features of the third convolution module and the output features of the fourth convolution module are upsampled and input into the third convolution layer and the fourth convolution layer respectively, and the third convolution layer is used as the input layer. The output features of the convolution layer and the output features of the fourth convolution layer are added and passed through the second ReLU activation function layer to obtain the intermediate features. The intermediate features are respectively subjected to multi-scale convolution through the fifth, sixth and seventh convolution layers. The output features of the fifth, sixth and seventh convolution layers are summed and averaged to obtain the fused features. The fused features are sequentially passed through the eighth convolution layer and the Sigmoid activation function layer to obtain the attention features. The attention features are element-wise multiplied with the output features of the third convolution module to obtain the output features of the multi-scale attention gate module.

[0012] Preferably, the original face image in the face wrinkle dataset is input into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map, which specifically includes:

[0013] During the training process, the original face image is passed through the face wrinkle segmentation network to generate the corresponding wrinkle segmentation prediction map , as shown below:

[0014] ;

[0015] in, Represents the function corresponding to the facial wrinkle segmentation network, represents the network parameters in the facial wrinkle segmentation network, represents the original face image, Represents the original face image Corresponding wrinkle segmentation prediction map, the width and height of the original face image and its corresponding wrinkle segmentation prediction map are W and H respectively.

[0016] Preferably, calculating the significance of each pixel in the original face image in the face wrinkle data set, and obtaining the weight of each pixel in the original face image according to the significance of each pixel in the original face image, specifically includes:

[0017] Grayscale the original face image and perform gradient detection to obtain the face gradient amplitude map, as shown in the following formula:

[0018] ;

[0019] ;

[0020] Among them, I g Represents the grayscale image of the original face image. The width and height of the grayscale image of the original face image are W and H respectively. x and y represent the pixel index in the horizontal and vertical directions respectively. ;I g (x,y),I g (x+1,y) and I g (x, y+1) represent the grayscale image I of the original face image g The pixel values ​​of the pixels at the positions (x, y), (x+1, y) and (x, y+1) are in the range of [0, 255] for each pixel in the grayscale image of the original face image, and I g (W+1,H) and I g The pixel value of (W,H+1) is set to 255; w Represents the horizontal gradient map, G w (x,y) represents the horizontal gradient map G w The pixel value of the pixel at the (x, y) position; G h Represents the vertical gradient map, G h (x,y) represents the horizontal gradient map G h The pixel value of the pixel at the (x,y) position; Indicates absolute value operation;

[0021] Calculate the gradient amplitude of each pixel in the grayscale image of the original face image to obtain the gradient amplitude map G, as shown in the following formula:

[0022] ;

[0023] Where G(x,y) is the gradient amplitude of the pixel point at the position (x,y) of the gradient amplitude map G;

[0024] Calculate the size centered at each pixel as The sum of the gradient amplitudes in the local window is used as the significance corresponding to each pixel, as shown in the following formula:

[0025] ;

[0026] Where u and v are the pixel offsets in the local window, k is the height and width of the local window, and S(x,y) is the saliency of the pixel at the position (x,y) of the original face image.

[0027] The saliency of each pixel in the original face image is normalized to obtain the weight of each pixel in the original face image, as shown in the following formula:

[0028] ;

[0029] in, Represents the weight of the pixel at the (x, y) position of the original face image, is the temperature coefficient, , exp represents the exponential function with the natural number e as the base.

[0030] Preferably, the expression of the gradient saliency weighted loss function is as follows:

[0031] ;

[0032] in, represents the gradient saliency weighted loss function. The height and width of the original face image are W and H respectively. x and y represent the pixel index in the horizontal and vertical directions respectively. ; represents the weight of the pixel at the position (x, y) of the original face image, M represents the wrinkle annotation mask corresponding to the original face image, Represents the pixel value of the pixel at the position (x,y) of the wrinkle annotation mask image, Represents the original face image The corresponding wrinkle segmentation prediction map, Represents the wrinkle segmentation prediction map The pixel value of the pixel at the (x,y) position, represents the logarithmic function with base e as the natural number, Represents the Sigmoid activation function.

[0033] In a second aspect, the present invention provides a wrinkle segmentation device based on gradient saliency perception, comprising:

[0034] A data annotation module is configured to obtain an original face image and annotate wrinkle pixels, generate a corresponding wrinkle annotation mask map, and construct a face wrinkle dataset based on the original face image and its corresponding wrinkle annotation mask map;

[0035] A model building module is configured to build a facial wrinkle segmentation network based on an improved UNet deep backbone network, where a multi-scale attention gate module is added to the traditional UNet deep backbone network;

[0036] A model training module is configured to input the original face image in the face wrinkle data set into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map, calculate the significance of each pixel in the original face image in the face wrinkle data set, obtain the weight of each pixel in the original face image according to the significance calculation of each pixel in the original face image, construct a gradient significance weighted loss function based on the pixel value of each pixel in the original face image and its weight and the pixel value of the corresponding pixel in the wrinkle annotation mask map corresponding to the original face image, train the face wrinkle segmentation network based on the gradient significance weighted loss function, and obtain a trained face wrinkle segmentation network;

[0037] The segmentation module is configured to obtain the face image to be processed and input it into a trained face wrinkle segmentation network to obtain a wrinkle segmentation prediction map corresponding to the face image to be processed.

[0038] In a third aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) The wrinkle segmentation method based on gradient saliency perception proposed in the present invention builds a facial wrinkle segmentation network based on the UNet deep backbone network for facial wrinkle feature learning to achieve complete and accurate segmentation of facial wrinkles.

[0043] (2) The wrinkle segmentation method based on gradient saliency perception proposed in this paper introduces a multi-scale attention gate module into the bottleneck layer of the UNet deep backbone network, which can enhance the model's sensitivity to wrinkle features of different scales while maintaining low computational overhead, thereby improving the expression of global semantic information and achieving semantic compression. By adjusting the feature weights of different regions, the network can pay more attention to key wrinkle areas, especially in complex skin textures and wrinkle structures, effectively improving the accuracy and robustness of wrinkle segmentation.

[0044] (3) The wrinkle segmentation method based on gradient saliency perception proposed in the present invention constructs a gradient saliency weighted loss function, calculates the saliency of each pixel in the original face image, calculates the weight of each pixel based on the saliency, and uses it to weight the wrinkle-background binary classification loss of the corresponding pixel, so that the face wrinkle segmentation network pays more attention to the pixels with high gradient saliency, and guides the face wrinkle segmentation network to pay more attention to the wrinkle pixel segmentation rather than the background pixel segmentation which is dominant in number, thereby improving the integrity of wrinkle segmentation and the effect of face wrinkle segmentation, meeting the needs of different application fields, and improving user experience and application effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of the process of a wrinkle segmentation method based on gradient saliency perception according to an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a facial wrinkle segmentation network of a wrinkle segmentation method based on gradient saliency perception according to an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a multi-scale attention gate module of a wrinkle segmentation method based on gradient saliency perception according to an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a wrinkle segmentation device based on gradient saliency perception according to an embodiment of the present application;

[0050] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Figure 1 A wrinkle segmentation method based on gradient saliency perception provided by an embodiment of the present application is shown, comprising the following steps:

[0053] S1, obtain the original face image and annotate the wrinkle pixels, generate the corresponding wrinkle annotation mask map, and construct a face wrinkle dataset based on the original face image and its corresponding wrinkle annotation mask map.

[0054] Specifically, the original face image is processed by Frangi filtering to generate an initial wrinkle annotation mask map, and the initial wrinkle annotation mask map is manually enhanced and annotated to generate a wrinkle annotation mask map M. The size of the original face image and the wrinkle annotation mask map is set to [W, H]. The original face image and its wrinkle annotation mask map are randomly horizontally flipped, rotated (±15°) and brightness adjusted (±10%) synchronously for data enhancement, so that a facial wrinkle dataset is constructed.

[0055] S2, builds a facial wrinkle segmentation network based on the improved UNet deep backbone network. The facial wrinkle segmentation network adds a multi-scale attention gate module on the basis of the traditional UNet deep backbone network.

[0056] In a specific embodiment, the facial wrinkle segmentation network includes a contraction path, an expansion path and a multi-scale attention gate module, the contraction path includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module, the expansion path includes a fifth convolution module, a sixth convolution module and a seventh convolution module, the first convolution module, the second convolution module, the third convolution module, the fourth convolution module, the fifth convolution module, the sixth convolution module and the seventh convolution module each include two first convolution layers with a convolution kernel size of 3×3, each first convolution layer with a convolution kernel size of 3×3 is connected to a first ReLU activation function layer, and between the first convolution module and the second convolution module, between the second convolution module and the third convolution module, and between the third convolution module and the fourth convolution module. It includes a pooling kernel 2×2 as the maximum pooling layer; the output features of the fourth convolution module are upsampled and input into the multi-scale attention gate module together with the output features of the third convolution module; the output features of the fourth convolution module are spliced ​​with the output features of the multi-scale attention gate module after upsampling and input into the fifth convolution module; the output features of the fifth convolution module are upsampled and spliced ​​with the output features of the second convolution module and input into the sixth convolution module; the output features of the sixth convolution module are upsampled and spliced ​​with the output features of the first convolution module and input into the seventh convolution module; the upsampling uses a deconvolution layer with a convolution kernel size of 2×2; the output features of the seventh convolution module pass through a second convolution layer with a convolution kernel size of 1×1, and output a wrinkle segmentation prediction map.

[0057] In a specific embodiment, the multi-scale attention gate module includes a third convolution layer, a fourth convolution layer, a second ReLU activation function layer, a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer and a Sigmoid activation function layer. The convolution kernel sizes of the third convolution layer, the fourth convolution layer and the eighth convolution layer are all 1×1, and the convolution kernel sizes of the fifth convolution layer, the sixth convolution layer and the seventh convolution layer are 3×3, 5×5 and 7×7 respectively. The output features of the third convolution module and the output features of the fourth convolution module are input into the third convolution layer and the fourth convolution layer respectively after upsampling. The output features of the three convolutional layers and the output features of the fourth convolutional layer are added and passed through the second ReLU activation function layer to obtain the intermediate features. The intermediate features are respectively subjected to multi-scale convolution through the fifth, sixth and seventh convolutional layers. The output features of the fifth, sixth and seventh convolutional layers are summed and averaged to obtain the fused features. The fused features are sequentially passed through the eighth convolutional layer and the Sigmoid activation function layer to obtain the attention features. The attention features are element-wise multiplied with the output features of the third convolutional module to obtain the output features of the multi-scale attention gate module.

[0058] Specifically, Figure 2As shown in the figure, the facial wrinkle segmentation network includes a contraction path and an expansion path, and is embedded with a multi-scale attention gate module. The contraction path and the expansion path have a total of 7 layers, each of which contains a convolution module, each of which contains two first convolution layers, and each first convolution layer is connected to a first ReLU activation function layer. A maximum pooling layer is set between two adjacent convolution modules in the contraction path. The contraction path is specifically as follows: the original face image is used as the input image, and after two convolution operations in the first layer, it passes through a maximum pooling layer to enter the second layer; the second layer also undergoes two convolution operations, and then passes through a maximum pooling layer to enter the third layer; the third layer also undergoes two convolution operations, and then passes through a maximum pooling layer to enter the fourth layer. The fourth layer undergoes two convolution operations, and the output features of the fourth layer enter the expansion path. The specific expansion path is as follows: the output features of the 4th layer are upsampled and enter the 5th layer, and the upsampled features of the output features of the 4th layer and the output features of the 3rd layer are input into the multi-scale attention gate module, the output features of the multi-scale attention gate module are concatenated with the upsampled features of the output features of the 4th layer, and then two convolution operations are performed in the 5th layer; the output features of the 5th layer are upsampled and enter the 6th layer, the upsampled features of the output features of the 5th layer are concatenated with the output features of the 2nd layer, and two convolution operations are performed in the 6th layer; the output features of the 6th layer are upsampled and enter the 7th layer, the upsampled features of the output features of the 6th layer are concatenated with the output features of the 1st layer, and three convolution operations are performed in the 7th layer to obtain the output feature map, which is the wrinkle segmentation prediction map. The convolution kernel size of the convolution operation of the 1st to 6th layers is 3×3, with a step size of 1; the convolution kernel size of the first two convolution operations of the 7th layer is 3×3, with a step size of 1, and the convolution kernel size of the last convolution operation is 1×1. The pooling kernel of the maximum pooling layer is 2×2, and the upsampling uses a deconvolution layer with a convolution kernel size of 2×2. The number of convolution kernels in the 1st to 6th layers is 64, 128, 256, 512, 256, and 128, respectively. The number of convolution kernels in the first two convolution operations of the 7th layer is 64, and the number of convolution kernels in the third convolution operation is 1. Among them, H i , W i , C i They represent the height, width and number of channels of the output features of the i-th layer respectively.

[0059] Further, such as Figure 3As shown in the figure, the output feature of the third convolution module is used as the input feature z of the multi-scale attention gate module, and the output feature of the fourth convolution module is used as the input feature g of the multi-scale attention gate module after upsampling. Given the dimensions of the input features g and z are (256, 64, 64), indicating that the number of channels is 256, the width is 64, and the height is 64, firstly, the convolution operation with a convolution kernel size of 1×1 is applied to the input features g and z respectively, and the output features g1 and z1 are obtained, and the dimensions are (128, 64, 64). Then, the output features g1 and z1 are added element by element and passed through the second ReLU activation function layer to obtain the intermediate feature ψ, whose dimension remains (128, 64, 64).

[0060] Subsequently, a multi-scale convolution operation with three different sizes of convolution kernels (3×3, 5×5 and 7×7) is applied to the intermediate feature ψ to obtain three output features ψ1, ψ2 and ψ3 with dimensions of (128, 64, 64). The output features ψ1, ψ2 and ψ3 of these three multi-scale convolutions are summed and averaged in the channel dimension to obtain the fusion feature ψ f , with dimensions of (128,64,64).

[0061] Next, a convolution operation with a kernel size of 1×1 is used to compress the number of channels to 1, and a Sigmoid activation function is applied to obtain an attention feature ψ of size (1, 64, 64) t Finally, the input feature z is combined with the attention feature ψ t Element-by-element multiplication yields the output feature z' of the multi-scale attention gate module, whose dimension is (256, 64, 64), which is consistent with the dimension of the input feature z.

[0062] S3, input the original face image in the face wrinkle dataset into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map, calculate the significance of each pixel in the original face image in the face wrinkle dataset, obtain the weight of each pixel in the original face image according to the significance calculation of each pixel in the original face image, construct a gradient significance weighted loss function based on the pixel value of each pixel in the original face image and its weight and the pixel value of the corresponding pixel in the wrinkle annotation mask map corresponding to the original face image, train the face wrinkle segmentation network based on the gradient significance weighted loss function, and obtain a trained face wrinkle segmentation network.

[0063] In a specific embodiment, the original face image in the face wrinkle dataset is input into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map, which specifically includes:

[0064] During the training process, the original face image is passed through the face wrinkle segmentation network to generate the corresponding wrinkle segmentation prediction map , as shown below:

[0065] ;

[0066] in, Represents the function corresponding to the facial wrinkle segmentation network, represents the network parameters in the facial wrinkle segmentation network, represents the original face image, Represents the original face image Corresponding wrinkle segmentation prediction map, the width and height of the original face image and its corresponding wrinkle segmentation prediction map are W and H respectively.

[0067] In a specific embodiment, calculating the significance of each pixel in the original face image in the face wrinkle data set, and obtaining the weight of each pixel in the original face image according to the significance of each pixel in the original face image, specifically includes:

[0068] Grayscale the original face image and perform gradient detection to obtain the face gradient amplitude map, as shown in the following formula:

[0069] ;

[0070] ;

[0071] Among them, I g Represents the grayscale image of the original face image. The width and height of the grayscale image of the original face image are W and H respectively. x and y represent the pixel index in the horizontal and vertical directions respectively. ;I g (x,y),I g (x+1,y) and I g (x, y+1) represent the grayscale image I of the original face image g The pixel values ​​of the pixels at the positions (x, y), (x+1, y) and (x, y+1) are in the range of [0, 255] for each pixel in the grayscale image of the original face image, and I g (W+1,H) and I g The pixel value of (W,H+1) is set to 255; w Represents the horizontal gradient map, G w (x,y) represents the horizontal gradient map G w The pixel value of the pixel at the (x, y) position; G h Represents the vertical gradient map, G h (x,y) represents the horizontal gradient map G h The pixel value of the pixel at the (x,y) position; Indicates absolute value operation;

[0072] Calculate the gradient amplitude of each pixel in the grayscale image of the original face image to obtain the gradient amplitude map G, as shown in the following formula:

[0073] ;

[0074] Where G(x,y) is the gradient amplitude of the pixel point at the position (x,y) of the gradient amplitude map G;

[0075] Calculate the size centered at each pixel as The sum of the gradient amplitudes in the local window is used as the significance corresponding to each pixel, as shown in the following formula:

[0076] ;

[0077] Where u and v are the pixel offsets in the local window, k is the height and width of the local window, and S(x,y) is the saliency of the pixel at the position (x,y) of the original face image.

[0078] The saliency of each pixel in the original face image is normalized to obtain the weight of each pixel in the original face image, as shown in the following formula:

[0079] ;

[0080] in, Represents the weight of the pixel at the (x, y) position of the original face image, is the temperature coefficient, , exp represents the exponential function with the natural number e as the base.

[0081] In a specific embodiment, the expression of the gradient saliency weighted loss function is as follows:

[0082] ;

[0083] in, represents the gradient saliency weighted loss function. The height and width of the original face image are W and H respectively. x and y represent the pixel index in the horizontal and vertical directions respectively. ; represents the weight of the pixel at the position (x, y) of the original face image, M represents the wrinkle annotation mask corresponding to the original face image, Represents the pixel value of the pixel at the position (x,y) of the wrinkle annotation mask image, Represents the original face image The corresponding wrinkle segmentation prediction map, Represents the wrinkle segmentation prediction map The pixel value of the pixel at the (x,y) position, represents the logarithmic function with base e as the natural number, Represents the Sigmoid activation function.

[0084] Specifically, the weight of each pixel is obtained by calculating the saliency of each pixel in the original face image, and the weight is used to weight the wrinkle-background binary classification loss of the corresponding pixel, so the gradient saliency weighted loss function is constructed. The gradient saliency weighted loss function is used to supervise the training of the face wrinkle segmentation network, and the trained face wrinkle segmentation network is obtained. The training process is as follows: the face wrinkle dataset is randomly divided into a training set (80%) and a validation set (20%); model initialization: the face wrinkle segmentation network is randomly initialized; the Adam optimizer is used, and the initial learning rate is set to 0.001; the cosine annealing learning rate scheduling strategy is used, and the minimum learning rate is 1e-5; the batch size is set to 1; a total of 100 rounds of training are performed. The entire training set is traversed in each round, and the gradient significance weighted loss function of the validation set is calculated once every 10 batches. If the gradient significance weighted loss function of the validation set does not decrease for 5 consecutive rounds, the training is terminated early. The evaluation indicators use F1 score and mIoU (mean intersection over union) to evaluate the performance of the model. After every 10 rounds, the model checkpoint is saved, and the parameters of the model with the best performance in the validation set are saved to obtain the trained facial wrinkle segmentation network.

[0085] S4, obtaining a face image to be processed and inputting it into a trained face wrinkle segmentation network to obtain a wrinkle segmentation prediction map corresponding to the face image to be processed.

[0086] Specifically, the trained facial wrinkle segmentation network is deployed and used for inference, and the facial image to be processed is Input the trained facial wrinkle segmentation network to generate the corresponding wrinkle segmentation prediction map . Later, the wrinkle segmentation prediction map can be Perform binarization to obtain the wrinkle segmentation mask map , as shown below:

[0087] ;

[0088] in, The pixel value of 1 represents the wrinkle pixel. The pixel value of 0 represents the background pixel. Represents the Sigmoid activation function.

[0089] Reuse the wrinkle segmentation mask Face images to be processed Perform coloring, that is, segment the mask map according to the wrinkles The spatial position of the wrinkle pixel with a pixel value of 1 is the face image to be processed The pixel points at the corresponding positions are set to purple pixel values ​​[128, 0, 128]. In other embodiments, they can also be set to other color pixel values ​​to further highlight the wrinkles in the face image to be processed. The position in.

[0090] Further references Figure 4 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a wrinkle segmentation device based on gradient saliency perception. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0091] The embodiment of the present application provides a wrinkle segmentation device based on gradient saliency perception, comprising:

[0092] The data annotation module 1 is configured to obtain an original face image and annotate wrinkle pixels, generate a corresponding wrinkle annotation mask map, and construct a face wrinkle dataset based on the original face image and its corresponding wrinkle annotation mask map;

[0093] Model building module 2 is configured to build a facial wrinkle segmentation network based on an improved UNet deep backbone network, where a multi-scale attention gate module is added to the traditional UNet deep backbone network;

[0094] The model training module 3 is configured to input the original face image in the face wrinkle data set into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map, calculate the significance of each pixel in the original face image in the face wrinkle data set, obtain the weight of each pixel in the original face image according to the significance calculation of each pixel in the original face image, construct a gradient significance weighted loss function based on the pixel value of each pixel in the original face image and its weight and the pixel value of the corresponding pixel in the wrinkle annotation mask map corresponding to the original face image, train the face wrinkle segmentation network based on the gradient significance weighted loss function, and obtain a trained face wrinkle segmentation network;

[0095] The segmentation module 4 is configured to obtain the face image to be processed and input it into a trained face wrinkle segmentation network to obtain a wrinkle segmentation prediction map corresponding to the face image to be processed.

[0096] Figure 5 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Figure 5As shown, the electronic device of this embodiment includes: a processor 501 and a memory 502; wherein the memory 502 is used to store computer-executable instructions; the processor 501 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0097] Optionally, the memory 502 may be independent or integrated with the processor 501 .

[0098] When the memory 502 is independently provided, the electronic device further includes a bus 503 for connecting the memory 502 and the processor 501 .

[0099] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 501 executes the computer execution instructions, the above method is implemented.

[0100] The embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 501, the above method is implemented.

[0101] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0102] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0103] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each module may exist physically separately, or two or more modules may be integrated into one unit. The unit formed by the above modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0104] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 501 to perform some steps of the methods of various embodiments of the present application.

[0105] It should be understood that the processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 501 may be any conventional processor 501, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 501, or may be executed by a combination of hardware and software modules in the processor 501.

[0106] The memory 502 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0107] The bus 503 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 503 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 503 in the drawings of the present application is not limited to only one bus 503 or one type of bus 503.

[0108] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0109] An exemplary storage medium is coupled to the processor 501, so that the processor 501 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 501. The processor 501 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 501 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0110] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A wrinkle segmentation method based on gradient saliency perception, characterized in that: The following steps are involved: Acquire an original face image and annotate wrinkle pixels to generate a corresponding wrinkle annotation mask image, and construct a face wrinkle dataset based on the original face image and its corresponding wrinkle annotation mask image; Constructing a facial wrinkle segmentation network based on an improved UNet deep backbone network, wherein the facial wrinkle segmentation network adds a multi-scale attention gate module on the basis of the traditional UNet deep backbone network; Inputting the original face image in the face wrinkle dataset into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map, calculating the significance of each pixel in the original face image in the face wrinkle dataset, and obtaining the weight of each pixel in the original face image according to the significance of each pixel in the original face image, specifically including: Grayscale the original face image and perform gradient detection to obtain a face gradient amplitude map, as shown in the following formula: G w (x,y)=|I g (x+1,y)-I g (x,y)|; G h (x,y)=|I g (x,y+1)-I g (x,y)|; Among them, I g Represents the grayscale image of the original face image. The width and height of the grayscale image of the original face image are W and H respectively. x and y represent the pixel index in the horizontal and vertical directions respectively, x∈[1,W], y∈[1,H]; I g (x,y),I g (x+1,y) and I g (x, y+1) represent the grayscale image I of the original face image g The pixel values ​​of the pixels at the positions (x, y), (x+1, y) and (x, y+1) are in the range of [0, 255] for each pixel in the grayscale image of the original face image, and I g (W+1,H) and I g The pixel value of (W,H+1) is set to 255; w Represents the horizontal gradient map, G w (x,y) represents the horizontal gradient map G w The pixel value of the pixel at the (x, y) position; G h Represents the vertical gradient map, G h (x,y) represents the horizontal gradient map G h The pixel value of the pixel at the (x, y) position; | | represents the absolute value operation; Calculate the gradient amplitude of each pixel in the grayscale image of the original face image to obtain the gradient amplitude map G, as shown in the following formula: Where G(x,y) is the gradient amplitude of the pixel point at the position (x,y) of the gradient amplitude map G; The sum of the gradient amplitudes in a local window of size (2k+1)×(2k+1) centered on each pixel is calculated and used as the saliency corresponding to each pixel, as shown in the following formula: Wherein, u and v are pixel offsets within the local window, k is the height and width of the local window, and S(x,y) is the saliency of the pixel point at the position (x,y) of the original face image; The saliency of each pixel in the original face image is normalized to obtain the weight of each pixel in the original face image, as shown in the following formula: Wherein, K(x,y) represents the weight of the pixel point at the position (x,y) of the original face image, τ is the temperature coefficient, 0<τ<1, and exp represents an exponential function with the natural number e as the base; a gradient significance weighted loss function is constructed based on the pixel value and weight of each pixel point in the original face image and the pixel value of the corresponding pixel point in the wrinkle annotation mask image corresponding to the original face image, and the face wrinkle segmentation network is trained based on the gradient significance weighted loss function to obtain a trained face wrinkle segmentation network; A facial image to be processed is obtained and input into the trained facial wrinkle segmentation network to obtain a wrinkle segmentation prediction map corresponding to the facial image to be processed.

2. The wrinkle segmentation method based on gradient saliency perception according to claim 1, characterized in that: The facial wrinkle segmentation network includes a contraction path, an expansion path and a multi-scale attention gate module, the contraction path includes a first convolution module, a second convolution module, a third convolution module and a fourth convolution module, the expansion path includes a fifth convolution module, a sixth convolution module and a seventh convolution module, the first convolution module, the second convolution module, the third convolution module, the fourth convolution module, the fifth convolution module, the sixth convolution module and the seventh convolution module all include two first convolution layers with a convolution kernel size of 3×3, each first convolution layer with a convolution kernel size of 3×3 is connected to a first ReLU activation function layer, and a pooling kernel of 2×2 is included between the first convolution module and the second convolution module, between the second convolution module and the third convolution module, and between the third convolution module and the fourth convolution module. Pooling layer; up-sample the output features of the fourth convolution module and input them together with the output features of the third convolution module into the multi-scale attention gate module; the output features of the fourth convolution module are spliced ​​with the output features of the multi-scale attention gate module after upsampling and input into the fifth convolution module; the output features of the fifth convolution module are up-sampled and spliced ​​with the output features of the second convolution module and input into the sixth convolution module; the output features of the sixth convolution module are up-sampled and spliced ​​with the output features of the first convolution module and input into the seventh convolution module; the upsampling uses a deconvolution layer with a convolution kernel size of 2×2; the output features of the seventh convolution module pass through a second convolution layer with a convolution kernel size of 1×1, and output a wrinkle segmentation prediction map.

3. The wrinkle segmentation method based on gradient saliency perception according to claim 2, characterized in that: The multi-scale attention gate module includes a third convolution layer, a fourth convolution layer, a second ReLU activation function layer, a fifth convolution layer, a sixth convolution layer, a seventh convolution layer, an eighth convolution layer and a Sigmoid activation function layer. The convolution kernel sizes of the third convolution layer, the fourth convolution layer and the eighth convolution layer are all 1×1, and the convolution kernel sizes of the fifth convolution layer, the sixth convolution layer and the seventh convolution layer are 3×3, 5×5 and 7×7 respectively. The output features of the third convolution module and the output features of the fourth convolution module are input into the third convolution layer and the fourth convolution layer respectively after upsampling. The output features of the third convolution layer are The output features of the fifth convolutional layer and the fourth convolutional layer are added and passed through the second ReLU activation function layer to obtain intermediate features, the intermediate features are respectively subjected to multi-scale convolution by the fifth convolutional layer, the sixth convolutional layer and the seventh convolutional layer, the output features of the fifth convolutional layer, the output features of the sixth convolutional layer and the output features of the seventh convolutional layer are summed and averaged to obtain fused features, the fused features are sequentially passed through the eighth convolutional layer and the Sigmoid activation function layer to obtain attention features, the attention features are element-wise multiplied with the output features of the third convolutional module to obtain the output features of the multi-scale attention gate module.

4. The wrinkle segmentation method based on gradient saliency perception according to claim 1, characterized in that: Inputting the original face image in the face wrinkle dataset into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map specifically includes: During the training process, the original face image is passed through the face wrinkle segmentation network to generate the corresponding wrinkle segmentation prediction map As shown below: Wherein, UNet(·) represents the function corresponding to the facial wrinkle segmentation network, θ represents the network parameters in the facial wrinkle segmentation network, I represents the original face image, It represents the wrinkle segmentation prediction map corresponding to the original face image I, and the width and height of the original face image and its corresponding wrinkle segmentation prediction map are W and H respectively.

5. The wrinkle segmentation method based on gradient saliency perception according to claim 1, characterized in that: The expression of the gradient saliency weighted loss function is as follows: in, represents the gradient significance weighted loss function, the height and width of the original face image are W and H respectively, x and y represent the pixel index in the horizontal direction and the vertical direction respectively, x∈[1,W], y∈[1,H]; K(x,y) represents the weight of the pixel at the position (x,y) of the original face image, M represents the wrinkle annotation mask corresponding to the original face image, and M(x,y) represents the pixel value of the pixel at the position (x,y) of the wrinkle annotation mask. represents the wrinkle segmentation prediction map corresponding to the original face image I, Represents the wrinkle segmentation prediction map The pixel value of the pixel at the (x, y) position, ln(·) represents the logarithmic function with the base being the natural number e, and σ represents the Sigmoid activation function.

6. A wrinkle segmentation device based on gradient saliency perception, characterized in that: include: A data annotation module is configured to obtain an original face image and annotate wrinkle pixels, generate a corresponding wrinkle annotation mask image, and construct a face wrinkle dataset based on the original face image and its corresponding wrinkle annotation mask image; A model building module is configured to build a facial wrinkle segmentation network based on an improved UNet deep backbone network, wherein the facial wrinkle segmentation network adds a multi-scale attention gate module on the basis of a traditional UNet deep backbone network; The model training module is configured to input the original face image in the face wrinkle dataset into the face wrinkle segmentation network to obtain the corresponding wrinkle segmentation prediction map, calculate the significance of each pixel in the original face image in the face wrinkle dataset, and calculate the weight of each pixel in the original face image according to the significance of each pixel in the original face image, specifically including: Grayscale the original face image and perform gradient detection to obtain a face gradient amplitude map, as shown in the following formula: G w (x,y)=|I g (x+1,y)-I g (x,y)|; G h (x,y)=|I g (x,y+1)-I g (x,y)|; Among them, I g Represents the grayscale image of the original face image. The width and height of the grayscale image of the original face image are W and H respectively. x and y represent the pixel index in the horizontal and vertical directions respectively, x∈[1,W], y∈[1,H]; I g (x,y),I g (x+1,y) and I g (x, y+1) represent the grayscale image I of the original face image g The pixel values ​​of the pixels at the positions (x, y), (x+1, y) and (x, y+1) are in the range of [0, 255] for each pixel in the grayscale image of the original face image, and I g (W+1,H) and I g The pixel value of (W,H+1) is set to 255; w Represents the horizontal gradient map, G w (x,y) represents the horizontal gradient map G w The pixel value of the pixel at the (x, y) position; G h Represents the vertical gradient map, G h (x,y) represents the horizontal gradient map G h The pixel value of the pixel at the (x, y) position; | | represents the absolute value operation; Calculate the gradient amplitude of each pixel in the grayscale image of the original face image to obtain the gradient amplitude map G, as shown in the following formula: Where G(x,y) is the gradient amplitude of the pixel point at the position (x,y) of the gradient amplitude map G; The sum of the gradient amplitudes in a local window of size (2k+1)×(2k+1) centered on each pixel is calculated and used as the saliency corresponding to each pixel, as shown in the following formula: Wherein, u and v are pixel offsets within the local window, k is the height and width of the local window, and S(x,y) is the saliency of the pixel point at the position (x,y) of the original face image; The saliency of each pixel in the original face image is normalized to obtain the weight of each pixel in the original face image, as shown in the following formula: Wherein, K(x,y) represents the weight of the pixel point at the position (x,y) of the original face image, τ is the temperature coefficient, 0<τ<1, and exp represents an exponential function with the natural number e as the base; a gradient significance weighted loss function is constructed based on the pixel value and weight of each pixel point in the original face image and the pixel value of the corresponding pixel point in the wrinkle annotation mask image corresponding to the original face image, and the face wrinkle segmentation network is trained based on the gradient significance weighted loss function to obtain a trained face wrinkle segmentation network; The segmentation module is configured to obtain the face image to be processed and input it into the trained face wrinkle segmentation network to obtain a wrinkle segmentation prediction map corresponding to the face image to be processed.

7. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images