An Automatic Segmentation and Restoration Management Platform and Method for Facial Mask Images

Abstract

The present invention discloses a face mask mask image automatic segmentation and restoration management platform and method. An improved DeepLabv3+ network model is used to automatically segment the face mask mask image. After automatic segmentation, a face image restoration network guided by face structure information is used to restore the image, obtaining a real face image not blocked by the mask. The entire process can be selected and controlled through the face mask mask image automatic segmentation and restoration management platform. The face mask mask image automatic segmentation and restoration management platform and method proposed by the present invention can process a large number of face mask mask images under the epidemic with high accuracy, reconstruct the normal face image under the mask, and have a great improvement and enhancement effect on the image processing and recognition of places where face recognition needs to be performed while wearing a mask.

Description

Technical Field

[0001] The present invention relates to the technical field of image restoration, and particularly to an automatic segmentation and restoration management platform and method for face mask images. Background Art

[0002] In recent years, with the continuous development of the field of computer vision, image restoration technology has become a research hotspot in the field of information technology and is widely used. For example, in the field of public security, images obtained from network devices and surveillance cameras are often blurred or contain a lot of noise due to environmental factors such as weather and light. Image restoration technology is used to improve the quality of photos; the wearing of masks causes the loss of more than half of the information of the human face, which has a great impact on face recognition. Improving the accuracy of face recognition under occlusion through image restoration technology is one of the ways; although image restoration technology has achieved good results at present, when removing large-area occlusions, the effect is not ideal because it covers a relatively large facial area and even extends beyond the actual face boundary below the chin. Secondly, there is currently no pair of face images for training objects with and without masks, and there are deficiencies in the processing of face mask images. Summary of the Invention

[0003] To solve the problem of insufficient processing of face mask images, the present invention proposes an automatic segmentation and restoration management platform and method for face mask images.

[0004] In a first aspect, an automatic segmentation and restoration management method for face mask images includes:

[0005] Automatic segmentation of face mask images: training is performed using an improved DeepLabv3+ network model to automatically segment face mask images; the backbone network of the improved DeepLabv3+ network model is a lightweight neural network MobileNetV2 network, and the lightweight neural network MobileNetV2 network removes the classification layer in the original MobileNetV2 and retains feature extraction until the last Bottleneck; the atrous spatial pyramid pooling module adopted by the improved DeepLabv3+ network model is a DCB-ASPP module, and the DCB-ASPP module changes the feature fusion method to a dense connection form without changing the structure of the ASPP module in the original Deeplabv3+ model. At the same time, a CBAM attention mechanism is introduced after the last layer of the ASPP module; the focal loss function is introduced into the loss function of the improved DeepLabv3+ network model for training;

[0006] Face Mask Image Inpainting: Establish a face image inpainting network guided by face structure information, including a face structure sketch generation network and a face inpainting network; the face structure sketch generation network generates a face structure sketch to guide the face inpainting network to repair the image, and a feature matching loss is introduced into the loss function of the face structure sketch generation network; the face structure sketch generation network generates a face structure sketch of the area to be repaired by learning the prior knowledge of the face structure, including a discriminator and a generator based on the encoder-decoder structure; the generator of the face structure sketch generation network further includes a downsampling convolutional layer, a residual block with dilated convolution, an upsampling convolutional layer, and a feature fusion layer that jumps from the downsampling layer to the upsampling layer; wherein, a residual block with dilated convolution is introduced after the downsampling convolutional layer of the face structure sketch generation network, and an instance normalization layer is added between each convolutional layer and activation layer during the downsampling and upsampling processes; the generator of the face inpainting network is based on the encoder-decoder structure, and the generator of the face inpainting network further includes a downsampling layer, a residual block with dilated convolution, a long short-term attention layer, an upsampling layer, and a feature fusion layer that jumps from the downsampling layer to the upsampling layer; the discriminators of both the face structure sketch generation network and the face inpainting network are Markov discriminators.

[0007] Furthermore, the output of any dilated convolutional layer in the DCB-ASPP module in the automatic segmentation of the face mask image is:

[0008] ;

[0009] In the formula, represents the dilation rate of the nth layer, k represents the size of the convolutional kernel, represents the output of all feature layers before cascading the nth layer;

[0010] Calculate the value of the receptive field improved by the DCB-ASPP module based on the receptive field formula, and the receptive field formula is defined as:

[0011] ;

[0012] Wherein, represents the receptive field of the nth layer, represents the receptive field of the (n-1)th layer, represents the size of the convolutional kernel of the nth layer.

[0013] Furthermore, the focal loss function introduced into the loss function of the improved DeepLabv3+ network model is defined as:

[0014] ;

[0015] Where the weight parameter and weight factors All are adjustable parameters. , is the true label of the sample, is the predicted probability that the sample belongs to the corresponding sample category.

[0016] Furthermore, the improved DeepLabv3+ network model training includes the following steps:

[0017] Step A1: Input the self-built face mask image dataset training set, set the number of loop training times, and initialize the network parameter weights;

[0018] Step A2: Start the training loop, load the labeled dataset, determine the current batch size, and adaptively adjust the learning rate;

[0019] Step A3: Freeze the lightweight neural network MobileNetV2 network and train the DCB-ASPP module;

[0020] Step A4: Calculate the focal loss and F1 score based on the predicted value and loss value;

[0021] Step A5: Back propagation, gradient update based on the focal loss value;

[0022] Step A6: Unfreeze the lightweight neural network MobileNetV2 network training and repeat steps A4 and A5 until the entire network converges;

[0023] Step A7: Save the model parameters and output the trained network parameter weights.

[0024] Furthermore, the training objective of the entire face structure sketch generation network is defined as:

[0025] ;

[0026] in, and is the regularization parameter;

[0027] Fighting Losses Defined as:

[0028] ;

[0029] Feature matching loss Defined as:

[0030] ;

[0031] in, Represents a real face image The corresponding real - structure sketch represents the real face image The corresponding grayscale image is the generator is at The predicted face - structure sketch, L represents the discriminator The total number of convolutional layers of is the number of elements in each layer represents the discriminator The activation function output of the i - th layer of

[0032] Furthermore, the face - image restoration network is guided by the face - structure sketch generated by the face - structure sketch generation network to restore the facial semantic structure and texture information of the face image.

[0033] Furthermore, in the face - restoration network, the face - restoration network The loss is defined as:

[0034] ;

[0035] In the formula, represents the real face image represents the output of the generator of the face - restoration network, and N is the number of masked pixel points after normalization;

[0036] The generative adversarial loss is defined as:

[0037] ;

[0038] Among them, represents the structure sketch input to the generator of the face - restoration network;

[0039] The perceptual loss is defined as:

[0040] ;

[0041] Among them, is the activation output mapping of the th layer of the pre - trained VGG - 19 model;

[0042] The style loss and the perceptual loss both need to use the pre - trained VGG - 19 model to map the input image to a high - dimensional feature space to extract high - level semantic structure feature information of the image;

[0043] The total loss of the face - restoration network is:

[0044] ;

[0045] Among them, , , and are respectively the loss, the generative adversarial loss , the perceptual loss and the style loss corresponding regularization parameters.

[0046] On the other hand, a face mask image automatic segmentation and restoration management platform is used to implement the face mask image automatic segmentation and restoration method described above. It is implemented based on the PyQt library in PyCharm and includes a panel control module, a display module, a processing module, and an information printing module;

[0047] The panel control module further includes:

[0048] Function selection sub-module: Perform function selection, including image processing function selection and video processing function selection;

[0049] Video control sub-module: Adjust the video to be processed, including playing, pausing, recognition, previous frame, next frame, and display of capture, frame length, number of frames, and frame rate;

[0050] Image processing sub-module: Process the image, including extraction, correction, restoration, evaluation mask, quality evaluation, and result saving;

[0051] Restoration quality evaluation sub-module: Evaluate the restoration quality of the face mask image, including SSIM (structural similarity), PSNR (peak signal-to-noise ratio), and FID (Fréchet Inception Distance).

[0052] Advantages of the present invention: The present invention proposes a face mask image automatic segmentation and restoration management platform and method. The improved DeepLabv3+ network model is used to automatically segment the face mask image. After automatic segmentation, the face image restoration network guided by the face structure information is used to repair the image to obtain a real face image not blocked by the mask. The entire process can be selected and controlled through the face mask image automatic segmentation and restoration management platform. By using the face mask image automatic segmentation and restoration management platform and method proposed by the present invention to process a large number of face mask images, the normal face images under the masks are reconstructed, the accuracy of mask image segmentation and restoration is improved, and it has a great improvement effect on image processing and recognition in public places such as subways where face recognition by swiping the face is required while wearing masks. Description of the Drawings

[0053] Figure 1It is a schematic diagram of the main interface of an automatic segmentation and restoration management platform for face mask images in the present invention;

[0054] Figure 2 It is a diagram of the CBAM attention mechanism introduced in the automatic segmentation model of face mask images in the present invention;

[0055] Figure 3 It is a diagram of the channel attention module of the automatic segmentation model of face mask images in the present invention;

[0056] Figure 4 It is a diagram of the spatial attention module of the automatic segmentation model of face mask images in the present invention;

[0057] Figure 5 It is a structural diagram of the improved atrous spatial pyramid pooling module of the automatic segmentation model of face mask images in the present invention;

[0058] Figure 6 It is an overall framework diagram of the face structure sketch generation network in the present invention;

[0059] Figure 7 It is an overall framework diagram of the face restoration network in the present invention;

[0060] Figure 8 It is a test result diagram in the embodiment of the present invention;

[0061] Figure 9 It is a schematic diagram of the software login interface of an automatic segmentation and restoration management platform for face mask images in the present invention;

[0062] Figure 10 It is a diagram showing the evaluation of the embodiment of an automatic segmentation and restoration management platform for face mask images in the present invention. Detailed implementation manners

[0063] For a clearer understanding of the technical features, purposes, and effects of the present invention, the specific implementation manners of the present invention will now be described with reference to the accompanying drawings.

[0064] The present invention proposes an automatic segmentation and repair management platform and method for face mask images. In the first aspect, the automatic segmentation of face mask images utilizes an improved DeepLabv3+ network model. The backbone network of the improved DeepLabv3+ network model is the lightweight neural network MobileNetV2 network, and the atrous spatial pyramid pooling module is the improved feature enhancement network DCB-ASPP module. The dense connection method is used to fuse the features of different feature layers, and the CBAM attention mechanism is introduced. The loss function of the improved DeepLabv3+ network model introduces the focal loss function for training. The face mask image repair establishes a face image repair network guided by face structure information, including a face structure sketch generation network and a face repair network. The generated face structure sketch is used to guide the face repair network, and the feature matching loss is introduced into the loss function. The face image repair network combines the perceptual loss and model training, and also includes a repair network attention mechanism.

[0065] In this embodiment, an improved DeepLabv3+ face mask segmentation method is proposed based on the DeepLabv3+ model. The main work is as follows:

[0066] (1) To reduce the number of model parameters, the present invention replaces the backbone network Xception network in DeepLabv3+ with the MobileNetV2 network, which greatly reduces the number of model parameters while maintaining the segmentation accuracy, enabling the model to be better deployed on embedded devices.

[0067] (2) To better utilize the feature information extracted by the backbone network, the dense connection method is used to fuse the features of different feature layers in the original atrous spatial pyramid pooling module (ASPP), and the CBAM attention mechanism is introduced into the improved ASPP module to enhance the feature expression ability of the model and improve the model's ability to obtain context information.

[0068] (3) The focal loss is introduced into the loss function of model training to alleviate the imbalance problem of different types of masks in the training samples and improve the segmentation accuracy of the model.

[0069] In this embodiment, the adopted MobileNetV2 network structure is mainly divided into three main parts. The first part is a common 3×3 convolutional block. The middle part is composed of multiple inverted residual blocks stacked together. The last part is composed of a 1×1 convolution and a 7×7 average pooling. The main network structure of MobileNetV2 is shown in Table 1, where t represents the expansion coefficient of the convolutional channels, c represents the number of output channels, n represents the number of times this layer needs to be repeated, s represents the convolutional stride, c_out represents the number of output channels for model classification, Con2d represents the convolutional layer, Avgpool represents the average pooling layer, and Bottleneck represents the inverted residual stacking layer.

[0070] Table 1 Main Network Structure of MobileNetV2

[0071] <{

[0072] In this embodiment, the CBAM attention mechanism is introduced. The CBAM attention mechanism is a module that can perform Attention in both the channel dimension and the spatial dimension, and has very few parameters. It combines the advantages of channel attention and spatial attention to obtain more reliable attention information, thereby guiding the model to better allocate weights. The framework diagram of the CBAM model is as Figure 2 shown. The input features first learn important channel feature information through the channel attention module, and then use the spatial attention module to learn important key feature points, and finally extract important output features.

[0073] Assume that F represents the input feature map, and its calculation process through CBAM is as follows:

[0074] ;

[0075] In the above formula, represents element-wise multiplication, represents channel attention, represents spatial attention. In the first stage, after the input features pass through the channel attention module, they are multiplied element-wise with the original feature map to obtain ; In the second stage, after performing feature extraction on on the spatial attention module and multiplying it element-wise with the output of the first stage, the final output result is obtained. <{

[0076] Among them, the channel attention module compresses each layer of the input feature map by using max pooling and average pooling, then inputs the compressed features into a shared fully connected layer for connection, then adds and fuses the output features, and activates them through the Sigmoid function. Finally, the weights of each channel of the input feature layer are calculated. The structural diagram of its channel attention module is as Figure 3 shown.

[0077] Suppose F represents the input feature map, and its calculation process through the channel attention module is as follows:

[0078]

[0079] In the above formula, represents the max pooling layer, represents the average pooling layer, represents the fully connected layer, and σ represents the Sigmoid activation function.

[0080] The spatial attention module uses max pooling and average pooling to take the maximum value and average value on the channels of each feature point, then stacks the obtained results, and uses a convolutional kernel with 1 channel number for dimensionality reduction. Finally, it is activated through the Sigmoid function, and finally the weights of each feature point of the input feature layer are calculated. The structural diagram of its spatial attention module is as Figure 4 shown.

[0081] In order to enhance the expression ability of features and improve the ability of the model to obtain a larger receptive field, in this embodiment, without changing the structure of the ASPP module in the original Deeplabv3+ model, the feature fusion method is changed to a dense connection form. At the same time, in order to make the model pay more attention to the segmentation object after feature extraction, the CBAM attention mechanism is introduced after the last layer of the ASPP module to improve the segmentation performance of the model. Finally, the improved ASPP feature enhancement network is named DCB-ASPP, and the DCB-ASPP structure is as Figure 5 shown.

[0082] For any layer in the DCB-ASPP module, its input is the fusion of all previous feature layers, and the output of any layer of dilated convolutional layer can be expressed as:

[0083]

[0084] In the formula, represents the dilation rate of the nth layer, k represents the size of the convolutional kernel, represents the output of all previous feature layers cascaded before the nth layer.

[0085] DCB-ASPP can significantly improve the receptive field of the model by stacking and densely connecting dilated convolutions. Among them, the receptive field refers to the size of the region in the previous layer corresponding to an element in the output result of a certain layer during the convolution process. Its calculation method starts from the first layer after the input layer and calculates sequentially backward, and satisfies the following formula:

[0086] When n = 1:

[0087] When n ≥ 2:

[0088]

[0089] In the above formula, represents the stride of the current layer, represents the total stride of the first n layers of the network, represents the receptive field of the nth layer, represents the receptive field of the (n - 1)th layer, represents the size of the convolution kernel of the nth layer.

[0090] Because the stride of the dilated convolution adopted in the ASPP module is 1, so in the formula the value of can be considered to be always equal to 1, and the formula can be simplified to:

[0091]

[0092] In the original ASPP, since the connection methods of dilated convolutions with different dilation rates are in parallel, so n = 1, that is, the maximum receptive field obtained in ASPP is the receptive field of the dilated convolution with the largest size among the parallel dilated convolutions. For example, when the dilation rate of ASPP in the DeepLabv3+ network is (6, 12, 18), its maximum receptive field is:

[0093] [[ID=�0]]

[0094] After stacking multiple dilated convolution layers through the dense connection method, from the previous derivation formula, it can be obtained that for the N-layer dense connection ASPP module, the maximum receptive field it can obtain is:

[0095]

[0096] Therefore, the corresponding<� value of DCB-ASPP with a dilation rate of (6, 12, 18) can be calculated as:

[0097]

[0098] It can be seen that using DCB-ASPP is greater than that of ASPP , so the receptive field can be significantly increased by using the strategy of dense connection.

[0099] In traditional image segmentation tasks, the cross-entropy loss function is used to measure the similarity between the true label and the predicted value. When there are only two classes in the segmentation task, its definition is as follows:

[0100]

[0101] In the above formula, y is the true label of the sample, and p is the predicted probability that the sample belongs to a certain class.

[0102] For the convenience of formula representation, define as:

[0103]

[0104] Therefore, from the above two formulas, we can get:

[0105]

[0106] To address the imbalance problem of masks with different classes, sizes, and colors in the training samples, the focal loss function is introduced as the loss function for training.

[0107] To balance the weights of positive and negative samples in the total loss, a weight parameter is added. When there are more negative samples, takes a smaller value to reduce the weight of negative samples, and is denoted as the following formula:

[0108]

[0109] Secondly, for hard-to-classify and easy-to-classify samples, a weight factor γ is added. When γ is greater than 0, it will reduce the weight of easy-to-classify samples, making the final loss of the model focus on hard-to-classify samples. Therefore, the focal loss function is defined as:

[0110]

[0111] In the improved DeepLabv3+ network model structure, in the encoder part, first, the feature extraction network in the original network is replaced with the lightweight network MobileNetV2 for feature extraction, reducing the number of model parameters and improving the model calculation speed. Among them, the improved model removes the classification layer in the original MobileNetV2 and retains it until the last Bottleneck for feature extraction. Secondly, the extracted high-level semantic feature information is fed into the DCB-ASPP feature enhancement module. DCB-ASPP is an improved module of the ASPP in the original DeepLabv3+ network model. Each feature layer in the original ASPP module is connected in a dense connection manner to obtain a sufficiently large receptive field and a denser feature pyramid. Then, the CBAM attention mechanism is added after the last layer of ASPP, enabling the model to adaptively weight more important channel features and spatial features during the learning process, improving the model segmentation accuracy. Finally, the enhanced features are compressed in channels through 1x1 convolution to obtain the high-level semantic features finally output by the encoder.

[0112] In the decoder part, first, the output feature of the second Bottleneck of the backbone feature extraction network MobileNetV2 is used as the low-level feature information and is processed by 1x1 convolution for dimensionality reduction. Then, the output feature of the DCB-ASPP feature enhancement module is upsampled by four times and then concatenated with the dimension-reduced low-level feature in channels, and the concatenated features are convolved twice with 3x3. Finally, after one upsampling by four times, the image is restored to the same size as the original Figure 1 image size, and finally the image segmentation prediction output is obtained.

[0113] In this embodiment, the augmented dataset of the publicly evaluated dataset PASCAL VOC2012 in image semantic segmentation is used for experiments. The PASCAL-VOC 2012 dataset has 20 specific categories such as people, vehicles, animals, indoor item scenes, etc., plus the background, a total of 21 categories. The training set in its augmented dataset has a total of 10,582 images, and the validation set has 1,449 images. Since the original PASCAL VOC2012 dataset does not have a test set. Therefore, this article re-divides the provided validation set into a validation set and a test set, where the validation set has 1,000 images and the test set has 449 images, and the input image size is set to 512x512.

[0114] Secondly, to meet the needs, in this embodiment, a self-built face mask dataset is constructed to further verify the performance of the model. Part of the self-built face mask dataset is synthesized by the face dataset Celeba and collected mask templates, and the other part is real face masks collected through camera capture and the network. To improve the generalization performance of the model, this paper uses a total of 40 mask templates to create a mask dataset, of which a total of 2000 images are manually annotated, 1500 for training, 300 for validation, and 200 for testing, to synthesize mask face images.

[0115] In this embodiment, the face image restoration network guided by face structure information mainly performs the following tasks:

[0116] (1) To address the problems of unreasonable facial semantic information and inconsistent facial contours in restored face images, a face image restoration network guided by facial structural information is proposed.

[0117] (2) For the facial structure sketch generation network, in order to better generate the facial structure sketch of the area to be repaired, jump connections and residual blocks with dilated convolutions are introduced into the network generator; in order to make the generator generate results that are more similar to the real structure sketch, feature matching loss is introduced into the loss function, so that the structure sketch generated by the model is richer and more reasonable.

[0118] (3) For the face restoration network, in order to make the restoration network pay more attention to the missing area during restoration, an attention mechanism is introduced into the face restoration network, and the perceptual loss and style loss are combined in the loss function to better reconstruct the facial contour structure and color texture of the face image in the area to be restored.

[0119] In this embodiment, a face image restoration network guided by facial structure information is proposed. The network consists of two parts: a face structure sketch generation network and a face restoration network. The face structure sketch generation network generates a face structure sketch of the area to be restored by learning prior knowledge of facial structure; the face restoration network uses the structure sketch generated by the face structure sketch generation network as conditional guidance information to perform face image restoration.

[0120] The face structure sketch generation network is mainly used to generate the structure sketch of the area to be repaired. Its input is the grayscale image of the occluded face image, the structure sketch of the occluded face image and the mask, and the output is the face structure sketch generated by the network. The overall framework of the face structure sketch generation network designed in this paper is as follows: Figure 6As shown in the figure, the upper part is the generator of the face structure sketch generation network, and the lower part is the discriminator of the face structure sketch generation network. The idea process of generating a face structure sketch first and then performing face restoration is similar to that when drawing a portrait. Usually, the face sketch is constructed first, and then colors and details are filled in on the sketch. This restoration method can better restore the missing areas of the face image and make the details and textures of the restored face image clear.

[0121] The generator of the structure sketch generation network is based on the encoder-decoder structure, mainly composed of three downsampling convolutional layers, seven residual blocks with dilated convolutions, three upsampling convolutional layers, and a feature fusion layer that jumps from the downsampling layer to the upsampling layer. In order to strengthen the generation ability of the structure sketch generation network generator for the face structure information in the area to be restored, skip connections are added between the downsampling layer and the upsampling layer of the structure sketch generation network generator for feature fusion. This can not only reuse the feature information of each stage collected in the downsampling process during the upsampling process, but also better transmit the gradient information from the shallow layer to the deep layer of the network, improving the training speed of the network.

[0122] Secondly, residual blocks with dilated convolutions are introduced after the downsampling layer to increase the receptive field of convolutional neurons. The introduction of dilated convolution can enable convolutional neurons to capture feature information in a larger range, so that the generator can collect more structural prior information of the face during the feature transfer process, and thus make the generated face structure sketch more in line with the face topology. Using residual blocks as the middle layer of the generator can not only avoid the problem of gradient disappearance in the training process of the GAN network, but also transfer more information after downsampling to the subsequent upsampling process, improving the training speed of the network and the generation quality of the face structure sketch.

[0123] Finally, instance normalization (IN) layers are added between each convolutional layer and activation layer during the downsampling and upsampling processes. The IN layer can not only make the network training more stable, but also accelerate the convergence speed of the network model during training.

[0124] The face restoration network is mainly used to generate a color image of the area to be restored on the face. Its input is the occluded face color image and the face structure sketch generated by the face structure sketch generation network, and the output is the restored face color image. The overall framework of the face restoration network designed in this paper is as Figure 7 shown. The upper part is the generator of the face restoration network, and the lower part is the discriminator of the face restoration network.

[0125] The generator of the face restoration network is also based on the encoder-decoder structure, mainly consisting of three downsampling layers, seven residual blocks with dilated convolutions, a long short-term attention layer, three upsampling layers, and a feature fusion layer that jumps from the downsampling layer to the upsampling layer. In order to make more effective use of the structural sketch information generated by the face structure sketch generation network, the face restoration network introduces an attention mechanism in the middle stage of the generator in addition to adding residual blocks with dilated convolutions. Among them, the purpose of introducing residual blocks with dilated convolutions in the face restoration network is similar to its role in the structural sketch generation network. Adding a long short-term attention layer (Self-attention Module) in the generator of the face restoration network can not only make the face restoration network pay more attention to the area to be restored during the restoration process, but also effectively connect the temporal feature maps, so as to ensure that the restored facial contour is more coordinated as much as possible and improve the facial texture details of the restored area.

[0126] In the face structure sketch generation network and the face restoration network, the discriminator both adopts the Patch-GAN, which mainly consists of four convolutional layers and a fully connected layer. Different from other discriminator network designs, the Patch-GAN first outputs an N×N matrix, and then calculates the mean value of the N×N matrix as the final discriminant output. This is essentially different from the output of traditional discriminators, which is only a true / false vector. Each position in the output matrix of the Patch-GAN can represent a receptive field of the generated image, and each receptive field corresponds to a part of the generated image. Therefore, using the Patch-GAN can more accurately distinguish the difference between the image generated by the generator and the real image, so as to better adjust the network gradient.

[0127] Secondly, in order to ensure that the discriminator pays as much attention as possible to the overall structure of the image and evaluates whether the generated image is consistent with the real image, the face structure sketch generation network and the face restoration network designed in this paper only use the global discriminator as the discriminator of the entire network. This is because when the local discriminator discriminates the difference between the generated image and the real image, it often only focuses on the restored area of the network. Although this satisfies the consistency of the restored area, it ignores the global nature of the overall image structure. And the face topology structure and texture information often need to satisfy the overall structure. Therefore, the global discriminator can better ensure the consistency between the regional structure and the overall structure, so that the generator can generate more realistic and vivid face images.

[0128] Finally, in order to prevent the situation of gradient explosion in the training process of the face structure sketch generation network and the face restoration network and make the training process proceed stably. Spectral Normalization (SN) is introduced in the discriminator to improve the training quality of the GAN network.

[0129] In this embodiment, the calculation of the loss function of the face structure sketch generation network includes: Let represent the real face image, represent the corresponding structure sketch, represent the corresponding grayscale image, be the mask to be repaired, 1 represents the missing area, and 0 represents the known area. Then the grayscale image of the known image can be expressed as , and the structure sketch of the known image can be expressed as . Therefore, the generator of the structure sketch generation network predicts that the face structure sketch can be expressed as:

[0130]

[0131] After the predicted face structure sketch , and the real structure sketch are input into the discriminator of the structure sketch generation network together, so as to distinguish the authenticity of the face structure sketch generated by the generator . In the structure sketch generation network, in order to better generate the face structure sketch of the area to be repaired, the structure sketch generation network introduces a feature matching loss on the basis of the generative adversarial loss for model training. Define the training objective of the entire structure sketch generation network as:

[0132] ;

[0133] In the above formula, and are regularization parameters.

[0134] Its adversarial loss is defined as follows:

[0135]

[0136] The feature matching loss constrains the generator by comparing the activation maps of the intermediate layers of the discriminator to produce results more similar to the real face structure sketch, thus stabilizing the training process. Its feature matching loss is defined as follows:

[0137]

[0138] In the above formula represents the discriminator The total number of convolutional layers, is the number of elements in each layer, denotes the discriminator The activation function output of the i-th layer.

[0139] The calculation of the face restoration network loss function includes: As can be seen from the above definition, the known-region color face image can be expressed as:

[0140] , the face structure sketch generated by the face structure generation network is , so the input to the face restoration network generator The structure sketch can be expressed as , so the face restoration network The output can be expressed as:

[0141] .

[0142] To ensure the rationality of the facial semantic information in the face restoration result, and to better reconstruct the facial contour structure and color texture of the face image in the area to be restored, the face restoration network introduces the loss, the generative adversarial loss , and jointly the perceptual loss and the style loss to train Equation , where The loss is defined as:

[0143]

[0144] In the formula, N is the number of masked pixel points after normalization.

[0145] The generative adversarial loss is defined as:

[0146]

[0147] The perceptual loss is used to constrain the reconstruction result of the face restoration network to be consistent with the original image in the semantic structure, and its definition is as follows:

[0148]

[0149] In the above formula is the activation output mapping of the th layer of the pre-trained VGG-19 model.

[0150] The definition of style loss is similar to that of perceptual loss. Both need to use the pre-trained VGG-19 network to map the input image to a high-dimensional feature space to extract high-level semantic structure feature information of the image. Different from perceptual loss, style loss first needs to construct a Gram matrix using the convolutional activation output features, calculate its correlation in the channel dimension, then calculate its Euclidean distance on the channel dimension correlation matrix, and finally model the texture style of the image. Its is defined as follows:

[0151]

[0152] In the above formula, is defined in the same way as in perceptual loss, and \(T\) represents matrix transpose.

[0153] Finally, the total loss of the face restoration network can be obtained as:

[0154]

[0155] In the above formula, , , and are respectively loss, generative adversarial loss , perceptual loss and style loss corresponding regularization parameters.

[0156] In this embodiment, 30,000 high-definition face images in the CelebA-HQ dataset are used to train and test the face structure sketch generation network and the face restoration network. Among them, 26,000 are used for the training set, 3,000 are used for the validation set, and 1,000 are used for the test set. There is no intersection between the training set, the validation set, and the test set. For the irregular mask dataset used in the experimental process, 26,000 in the dataset are used for the training set, 3,000 are used for the validation set, and 1,000 are used for the test set. The input image size is 256x256. During training, the face structure sketch generation network and the face restoration network are first trained separately, and then the face structure sketch generation network and the face restoration network are jointly trained. During the training process, the Adma optimizer with an exponential decay rate of \(\beta_1 = 0\), \(\beta_2 = 0.9\) is used for optimization. When training separately, the learning rate is set to \(10^{-4}\), and when jointly training, the learning rate is set to \(10^{-6}\). The batch size of the pictures is set to 8, and the number of iterations required to save the model is set to 1,000 times. The structure sketch used in the face structure sketch generation network is obtained by Gaussian blur and image division operation, and the Gaussian kernel size is set to (15,15). In the face structure sketch generation network, the regularization parameter of the loss function is set to = 1, = 10; In the face restoration network, the loss function regularization parameter is set to = 1, = = 0.1, = 250.

[0157] As Figure 8 shown is the model test result of this embodiment. Figure 8 In

[0158] From Figure 8 the test results, it can be seen that the structural sketch (e) generated by the face structure sketch generation network can not only maintain the face topology structure, but also accurately restore the facial semantic information of the face image under the mask occlusion. Its restored facial structure texture information is very rich, and at the same time, it can maintain the structural consistency at the boundary between the known facial area and the area to be restored. It is very difficult to distinguish the restoration traces with the naked eye, and finally it can restore a real and clear face structure sketch, which verifies the effectiveness of the face structure sketch generation network restoration performance designed by the present invention. Secondly, from Figure 8 the test results, it can be seen that under the guidance of the face structure sketch information predicted by the face structure sketch generation network, the face restoration network can well restore the facial color image (f) of the occluded image. The synthesized texture information can not only be consistent with the surrounding known areas, but also the facial contour is coordinated after restoration, the color distribution is uniform, the boundary information is coherent, and the restoration result is real and clear, which verifies the effectiveness of the face restoration network designed by the present invention.

[0159] An automatic segmentation and restoration management platform for face mask images integrates these two parts, that is, transmits an image blocked by a mask and outputs a restored face image. The design of the mask-blocked face detection and restoration software platform mainly uses the PyQt library in PyCharm for the development of the upper computer. PyQt implements the basic library functions of Qt using the Python language and integrates the basic functions of the Qt library. The mask face detection and restoration management software is divided into four major modules: the panel control area, the display area, the processing area, and the information printing area. Among them, the panel controller area is divided into four modules: the function selection area, the video control area, the image processing area, and the restoration quality evaluation area. The function selection area is divided into two parts: image processing and video processing; the video control area is divided into play, pause, recognition, previous frame, next frame, capture, and display of frame length, number of frames, and frame rate; the image processing area is divided into extraction, correction, restoration, evaluation mask, quality evaluation, and result saving; the restoration quality evaluation area is divided into SSIM (structural similarity), PSNR (peak signal-to-noise ratio), and FID distance. The software login interface is as Figure 9 shown, and the main interface is as Figure 1 shown.

[0160] In the software, by loading the image to be restored, then clicking the extraction button to automatically extract the mask, and then clicking the correction button to correct the mask. The main purpose is to remove some edge noise points through simple morphological image processing operations of erosion and dilation. Finally, click the restoration button to restore the face blocked by the mask, and finally obtain the restored face image, as shown in Figure 10.

[0161] By selecting the quality evaluation function in the software, and sending the standard image and the evaluation mask after superposition into the restoration software, the quality detection can be carried out. Among them, SSIM is the structural similarity. The larger the SSIM value, the better the image quality, and its value can better reflect the subjective feeling of the human eye; PSNR is the peak signal-to-noise ratio. The larger the PSNR, the smaller the image distortion, generally between 20 and 40 dB. FID represents the distance between the feature vectors of the generated image and the real image. The closer this distance is, the better the effect of the generation model, that is, the image has high clarity and rich diversity.

[0162] The present invention provides a face mask occlusion image automatic segmentation and restoration management platform and method. An improved DeepLabv3+ network model is used to automatically segment the face mask occlusion image. After automatic segmentation, a face image restoration network guided by face structure information is used to restore the image, obtaining a real face image not occluded by the mask. The entire process can be selected and controlled through the face mask occlusion image automatic segmentation and restoration management platform. Through the face mask occlusion image automatic segmentation and restoration management platform and method provided by the present invention, a large number of face mask occlusion images are processed with high accuracy, reconstructing the normal face image under the mask, which has a great improvement effect on image processing and recognition in public places such as subways where face recognition by swiping the face is required while wearing a mask.

[0163] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and the descriptions in the specification are only used to illustrate the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and all these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. An automatic segmentation and restoration method for face mask images, characterized in that, Including: Automatic segmentation of face mask images: Using an improved DeepLabv3+ network model for training to automatically segment face mask images; The backbone network of the improved DeepLabv3+ network model is the lightweight neural network MobileNetV2 network. The lightweight neural network MobileNetV2 network removes the classification layer in the original MobileNetV2 and retains it until the last Bottleneck for feature extraction; The atrous spatial pyramid pooling module adopted by the improved DeepLabv3+ network model is the DCB-ASPP module. The DCB-ASPP module changes the feature fusion method to a dense connection form without changing the structure of the ASPP module in the original Deeplabv3+ model. At the same time, a CBAM attention mechanism is introduced after the last layer of the ASPP module; The focal loss function is introduced into the loss function of the improved DeepLabv3+ network model for training; Face mask image restoration: Building a face image restoration network guided by face structure information, including a face structure sketch generation network and a face restoration network; The face structure sketch generation network generates a face structure sketch to guide the face restoration network to restore the image, and a feature matching loss is introduced into the loss function of the face structure sketch generation network; The face structure sketch generation network generates a face structure sketch of the area to be restored by learning the prior knowledge of the face structure, including a discriminator and a generator based on the encoder-decoder structure; The generator of the face structure sketch generation network also includes a downsampling convolutional layer, a residual block with dilated convolution, an upsampling convolutional layer, and a feature fusion layer that jumps from the downsampling layer to the upsampling layer; among them, a residual block with dilated convolution is introduced after the downsampling convolutional layer of the face structure sketch generation network, and an instance normalization layer is added between each convolutional layer and activation layer during the downsampling and upsampling processes; The generator of the face restoration network is based on the encoder-decoder structure. The generator of the face restoration network also includes a downsampling layer, a residual block with dilated convolution, a long short-term attention layer, an upsampling layer, and a feature fusion layer that jumps from the downsampling layer to the upsampling layer; The discriminators of both the face structure sketch generation network and the face restoration network are Markov discriminators.

2. The automatic segmentation and restoration method of a face mask image according to claim 1, characterized in that The output of any atrous convolutional layer in the DCB-ASPP module in the automatic segmentation of face mask images is: ； In the formula, represents the porosity of the nth layer, and k represents the size of the convolutional kernel. represents the output of all feature layers before cascading the nth layer; Calculating the value of the receptive field improved by the DCB-ASPP module based on the receptive field formula, and the receptive field formula is defined as: ； Among them, represents the receptive field of the n-th layer, represents the receptive field of the (n - 1)-th layer, represents the size of the convolutional kernel of the n-th layer.

3. The automatic segmentation and restoration method of a face mask image according to claim 1, characterized in that, The focal loss function introduced into the loss function of the improved DeepLabv3+ network model is defined as: ； Among them, the weight parameter and the weight factor are both adjustable parameters, , is the true label of the sample, is the predicted probability that the sample belongs to the corresponding sample category.

4. The automatic segmentation and restoration method of a face mask image according to claim 1, characterized in that, The training of the improved DeepLabv3+ network model includes the following steps: Step A1: Input the self-built training set of the face mask image dataset, set the number of loop training times, and initialize the network parameter weights; Step A2: Start the loop training process, load the labeled dataset, determine the current batch size, and adaptively adjust the learning rate; Step A3: Freeze the lightweight neural network MobileNetV2 network and train the DCB-ASPP module; Step A4: Calculate the focal loss and F1 score based on the predicted values and loss values; Step A5: Backpropagate and update the gradients according to the focal loss value; Step A6: Unfreeze the lightweight neural network MobileNetV2 network for training, and repeat Steps A4 and A5 until the entire network converges; Step A7: Save the model parameters and output the trained network parameter weights.

5. The automatic segmentation and repair method of a face mask image according to claim 1, characterized in that, The training objective of the entire face structure sketch generation network is defined as: ； wherein, and are regularization parameters; Adversarial loss It is defined as: ； Feature matching loss It is defined as: ； Among them, represents the real face image corresponding real structure sketch, represents the real face image corresponding grayscale image, is the generator, is at the face structure sketch predicted, L represents the discriminator total number of layers of the convolutional layer, is the number of elements in each layer, represents the discriminator activation function output of the i-th layer.

6. The automatic segmentation and restoration method of a face mask image according to claim 1, characterized in that The face image restoration network is guided by the face structure sketch generated by the face structure sketch generation network to restore the facial semantic structure and texture information of the face image.

7. A method for automatic segmentation and restoration of face mask images according to claim 1, characterized in that In the face restoration network, the face restoration network The loss is defined as: ； In the formula, represents the real face image, represents the output of the generator of the face restoration network, and N is the number of masked pixel points after normalization; Generative adversarial loss It is defined as: ； Among them, represents a structural sketch input to the generator of the face restoration network; Perceptual loss It is defined as: ； Among them, is the activation output map of the th layer of the pre-trained VGG-19 model; Style loss and perceptual loss both need to use the pre-trained VGG-19 model to map the input image to a high-dimensional feature space to extract high-level semantic structure feature information of the image; The total loss of the face restoration network is: ； Among them, , , and are respectively the loss, the generative adversarial loss , the perceptual loss and the style loss corresponding regularization parameters.

8. A face mask image automatic segmentation and restoration management platform for implementing the face mask image automatic segmentation and restoration method according to any one of claims 1-7, characterized in that It is implemented based on the PyQt library in PyCharm and includes a panel control module, a display module, a processing module, and an information printing module; The panel control module further includes: Function selection sub-module: Perform function selection, including image processing function selection and video processing function selection; Video control sub-module: Adjust the video to be processed, including playing, pausing, recognition, previous frame, next frame, as well as capturing and displaying the frame length, number of frames, and frame rate; Image processing sub-module: Process the image, including extraction, correction, restoration, evaluation mask, quality evaluation, and result saving; Restoration quality evaluation sub-module: Evaluate the restoration quality of the face mask image, including SSIM structural similarity, PSNR peak signal-to-noise ratio, and FID distance.

Images