Training method and apparatus for a synthetic model of facial mask images

Abstract

This invention provides a training method and apparatus for a synthesis model of facial mask images. The method includes: acquiring an image of a face without a mask and an image of a mask; training a generator to obtain spatial transformation coefficients for the mask image; deforming the mask image based on the spatial transformation coefficients; fusing the deformed mask image with the image of the face without a mask to obtain a preliminary facial mask image; judging the preliminary facial mask image using a discriminator to obtain a discrimination score W; if the discrimination score W is within a preset threshold range, the network parameters in the generator and discriminator are qualified, and the generator and discriminator are used as the synthesis model for the final facial mask image. The synthesis model in this application can quickly and massively synthesize facial mask images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a training method and apparatus for a synthetic model of a face mask image. Background Technology

[0002] Current deep learning-based face recognition methods achieve good recognition rates for faces without obstructions, but they can no longer accurately identify faces with large areas of occlusion. One of the main reasons for this is that face recognition neural network models were not trained using face data of people wearing masks. Therefore, to improve the recognition rate of face recognition systems when faces are masked, a large dataset of masked faces is needed. Summary of the Invention

[0003] Based on this, the purpose of this invention is to provide a synthesis model that integrates feature extraction from Re-WGAN (Wasserstein Generative Adversarial Nets) to achieve the synthesis of a large number of face mask images. This model can effectively learn the difference between the features of occluded and unoccluded faces, thereby generating a large number of accurate face mask images.

[0004] In a first aspect, this embodiment provides a training method for a synthesis model of a face mask image, the method comprising: step S102, acquiring an image of a face without a mask and an image of a mask; step S104, constructing a generator and training spatial transformation coefficients of the mask image based on the generator; wherein the generator includes generator network structure parameters; step S106, deforming the mask image based on the spatial transformation coefficients, and fusing the deformed mask image and the image of a face without a mask to obtain a preliminary face mask image; step S108, constructing a discriminator and using the discriminator to discriminate the preliminary face mask image to obtain a discrimination score W; wherein the discriminator includes discriminator network structure parameters; step S110, if the discrimination score W is within a preset threshold range, the network parameters in the generator and discriminator are qualified, and the generator and discriminator are used as the synthesis model of the final face mask image; step S112, if the discrimination score W is not within the preset threshold range, adjusting the generator network structure parameters and the discriminator network structure parameters.

[0005] Further, step S104 includes: constructing a generator; superimposing the acquired unmasked face image and masked image by channel number to obtain a superimposed image; inputting the superimposed image into the initially constructed generator for feature extraction to obtain spatial transformation coefficients.

[0006] Further, the step of inputting the superimposed image into the initially constructed generator for feature extraction to obtain spatial transformation coefficients includes: extracting features from the superimposed image using a 7*7 large convolution kernel in the generator network structure parameters to obtain target features; stacking the target features using a 3*3 intermediate convolution in the generator network structure parameters to extract information and obtain preliminary feature information of the superimposed image; and obtaining pooled features based on the preliminary feature information of the superimposed image using a 1*1 global adaptive smooth pooling in the generator network structure parameters.

[0007] Further, step S106 includes: using a preset face detector to detect preset facial key points and the position and pose of the deformed mask in the face image without a mask, and aligning them; in the aligned image, setting the pixel value of the mask area to 0 and the pixel value of the non-mask area to 1 to obtain a mask image; multiplying the face image without a mask and the deformed mask image by the mask image respectively to obtain a first mask image and a second mask image respectively; and performing weighted fusion on the first mask image and the second mask image to obtain a preliminary face mask image.

[0008] Further, step S108 includes: constructing a discriminator; inputting a pre-acquired real face image wearing a mask and a preliminary face mask image generated by the generator into the discriminator to obtain a discrimination score W, a loss function G of the generator, and a loss function D of the discriminator.

[0009] Further, step S108 also includes: inputting the pre-acquired real face image wearing a mask and the preliminary face mask image generated by the generator into the discriminator; extracting features from the real face image wearing a mask and the preliminary face mask image generated by the generator using a 7*7 large convolution kernel in the discriminator network structure parameters to obtain the first target feature and the second target feature respectively; stacking the first target feature and the second target feature using a 3*3 intermediate convolution in the discriminator network structure parameters to extract information, obtaining the first target information and the second target information; obtaining the first pooled feature and the second pooled feature by performing 1*1 global adaptive smooth pooling based on the first target information and the second target information respectively; obtaining the 5*5*1 feature matrix Pr by performing a 3*3 convolution on the first pooled feature; similarly, obtaining the 5*5*1 feature matrix Pg by performing a 3*3 convolution on the second pooled feature; calculating the discrimination score W based on the feature matrix Pr and the feature matrix Pg; and obtaining the loss function G of the generator and the loss function D of the discriminator when W(Pr, Pg) approaches 0.

[0010] Further, step S112 includes: optimizing the generator network structure parameters using the gradient ascent algorithm by calculating the gradient of the loss function G; similarly, optimizing the discriminator network structure parameters using the gradient descent algorithm by calculating the gradient of the loss function D; wherein the adjustment of the generator network structure parameters and the discriminator network structure parameters is performed alternately.

[0011] Secondly, embodiments of the present invention provide a training device for a synthetic model of a face mask image, the training device comprising: an image acquisition module, a spatial transformation coefficient confirmation module, a face mask image confirmation module, a discrimination score calculation module, a model confirmation module, and a parameter adjustment module.

[0012] The beneficial effects of the embodiments of the present invention are as follows:

[0013] The purpose of this application is to synthesize face images with masks based on images of faces without masks and images of faces with masks. This technology can enhance the realism of the synthesized images and can be used as a dataset to add to the training model for face recognition with mask occlusion, thereby improving the accuracy of the face recognition algorithm with mask occlusion.

[0014] Other features and advantages of the invention will be set forth in the following description, or some features and advantages may be inferred from the description or determined without doubt, or may be learned by practicing the techniques described above.

[0015] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating a training method for a synthetic model of a human face mask image, provided in an embodiment of the present invention. Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention. Example

[0020] This embodiment provides a training method for a face mask image synthesis model. The method is based on a Re-WGAN network to synthesize face mask images and includes the following steps.

[0021] Step 1: Obtain facial images of people without masks and images of people wearing masks.

[0022] The data mentioned above is the dataset required for training the Re_WGAN network (i.e., the synthetic model of this application). In this application, the Re_WGAN network includes a generator and a discriminator.

[0023] Step 2: Construct a generator and train it to obtain the spatial transformation coefficients of the mask image; wherein, the generator has generator network structure parameters.

[0024] Specifically, the principle of this step is a method based on the ResNet network structure (Residual Network). The second step is divided into the following steps.

[0025] 2-1: Construct the generator, where the generator network structure parameters are shown in Table 1.

[0026] Table 1

[0027]

[0028] 2-2: The acquired face images without masks and those with masks are superimposed by increasing the number of channels to obtain the superimposed image.

[0029] 2-3: Input the above superimposed image into the initially constructed generator for feature extraction to obtain spatial transformation coefficients.

[0030] Specifically, the ResNet_18 network structure is used here.

[0031] Steps 2-3 specifically include:

[0032] 2-3-1: The superimposed image extracts features from the superimposed image using a large 7*7 convolutional kernel in the generator network structure parameters to obtain the target features.

[0033] Specifically, the purpose of using a large 7*7 convolution kernel is to preserve as much information as possible from the original image while reducing computational cost.

[0034] 2-3-2: Information extraction is achieved by stacking the above target features through 3*3 intermediate convolutions in the generator network structure parameters, thus obtaining the preliminary feature information of the superimposed image.

[0035] 2-3-3: Based on the preliminary feature information of the superimposed image, the pooled features are obtained through 1*1 global adaptive smooth pooling in the generator network structure parameters, so as to reduce the feature dimension.

[0036] 2-3-4: Perform a fully connected operation on the pooled features to obtain the spatial transformation coefficients of the mask image.

[0037] Specifically, steps 2-3-3 and 2-3-4 use global adaptive smooth pooling to stretch all the features of the image into a 1*1 shape, that is, stretching the 1x512x7x7 input data into a 1x512x1x1 shape. Then, the output of the fully connected layer is connected, and softmax logistic regression is used to integrate the local information with class discrimination in the previous convolutional and pooling layers, and finally obtain the spatial transformation coefficients of the mask image.

[0038] Step 3: Based on the above spatial transformation coefficients, deform the mask image, and fuse the deformed mask image with the face image without a mask to obtain a preliminary face mask image.

[0039] Specifically, the basic principle of this step is to merge an image of a face without a mask and an image of a mask with a distorted mask into a face-mask image. The premise for the authenticity of the face-mask image synthesis is that the mask image must be consistent with the position, viewing angle and direction of the face image without a mask.

[0040] Specifically, the third step includes:

[0041] 3-1: Use a preset face detector to detect preset facial landmarks and the position and pose of the deformed mask in the face image of the person without a mask, and then align them.

[0042] 3-2: In the aligned image, set the pixel values ​​of the mask area to 0 and the pixel values ​​of the non-mask area to 1 to obtain a mask image.

[0043] 3-3: Multiply the face image without a mask and the image with a deformed mask by the mask image respectively to obtain the first mask image and the second mask image.

[0044] Specifically, the first mask image is the result of multiplying the image of an unmasked face by the mask image, and the second mask image is the result of multiplying the deformed mask image by the mask image. Here, the pixel values ​​of the masked areas in the second mask image become 0, while the pixel values ​​of the non-masked areas remain the original values ​​(i.e., the values ​​of the non-masked areas in the deformed mask image in step 3-1); while the pixel values ​​of the non-masked areas in the first mask image become 0, while the pixel values ​​of the masked areas remain the original values ​​(i.e., the values ​​of the masked areas detected in the unmasked face image using a preset face detector in step 3-1).

[0045] 3-4: Weighted fusion of the first mask image and the second mask image is performed to obtain a preliminary face mask image.

[0046] Step 4: Construct a discriminator and use the discriminator to discriminate the preliminary face mask image to obtain a discrimination score W; wherein, the discriminator has discriminator network structure parameters.

[0047] Specifically, the aforementioned discrimination score is used to represent the quality of the synthesized face mask image.

[0048] In this invention, the discriminator uses the objective function of WGAN as the optimization metric; Re_WGAN is the name of the entire network structure, which consists of a generator and a discriminator.

[0049] The fourth step includes:

[0050] 4-1: Construct the discriminator.

[0051] Specifically, the discriminator network structure is similar to the generator network structure, as shown in Table 2.

[0052] Specifically, unlike the generator, the discriminator does not use a fully connected layer at the end, but instead goes through a 3*3 convolution to obtain a 5*5*1 matrix.

[0053] Table 2

[0054]

[0055] Table 3

[0056]

[0057] Specifically, Table 3 shows the existing GAN network structures. The reason why Table 2 in this application differs from Table 3 is as follows: the discriminator network structure in the GAN network structure performs a real / fake binary classification task, so the last layer is a sigmoid function; however, the discriminator in this application performs an approximate fitting of the Wasserstein distance, which is a regression task. Therefore, the sigmoid function of the last layer in Table 3 is removed, resulting in Table 2. In this way, the feature matrix Pr of the real face image wearing a mask and the synthesized preliminary face mask image feature matrix Pg can be obtained from Table 2.

[0058] 4-2: Input the pre-acquired real face image of a person wearing a mask and the preliminary face image of a person wearing a mask generated by the generator into the discriminator to obtain the discrimination score W, the loss function G of the generator and the loss function D of the discriminator.

[0059] Specifically, this step uses the objective function of WGAN as the optimization metric and the Earth-mover distance (EM, also known as Wasserstein distance) as the evaluation criterion. The Wasserstein distance represents the minimum distance (or minimum transportation cost) between two distributions, and the Wasserstein distance is the discrimination score.

[0060] Step 4-2 is almost identical in principle to step 2-3, including:

[0061] 4-2-1: Input the real face image of wearing a mask obtained in advance and the preliminary face mask image generated by the generator into the discriminator. Extract the features of the real face image of wearing a mask and the preliminary face mask image generated by the generator through the 7*7 large convolution kernel in the discriminator network structure parameters, respectively, to obtain the first target feature and the second target feature.

[0062] 4-2-2: Information extraction is achieved by stacking the first target feature and the second target feature through 3*3 intermediate convolutions of the discriminator network structure parameters, thus obtaining the first target information and the second target information.

[0063] 4-2-3: Based on the first target information and the second target information, 1*1 global adaptive smooth pooling is used to obtain the first pooled feature and the second pooled feature respectively.

[0064] 4-2-4: The features after the first pooling are convolved by 3*3 to obtain a 5*5*1 feature matrix Pr; similarly, the features after the second pooling are convolved by 3*3 to obtain a 5*5*1 feature matrix Pg.

[0065] 4-2-5: Calculate the discrimination score W based on Pr and Pg.

[0066] The discriminant score, also known as the Wasserstein distance, reflects the difference between the image distribution generated by the generator and the real image distribution, thus better guiding the training of the discriminator. The specific calculation method is shown in Equation 1.

[0067]

[0068] in, For the expectation of Pr, Let Pg be the expectation, and D(x) represent the discriminator network without a sigmoid layer (i.e., the network structure in Table 2).

[0069] 4-2-6: When W(Pr, Pg) approaches 0, the loss function G of the generator and the loss function D of the discriminator are obtained.

[0070] Specifically, the generator's role is to approximately minimize the Wasserstein distance.

[0071] The specific calculation formulas are shown in Formulas 2 and 3:

[0072]

[0073] Step 5: If the discrimination score W is within the preset threshold range, then the network parameters in the generator and discriminator are qualified, and the generator and discriminator are used as the synthesis model of the final face mask image.

[0074] Step 6: If the discrimination score W is not within the preset threshold range, adjust the generator network structure parameters and the discriminator network structure parameters.

[0075] Specifically, the parameters of the generator and discriminator are optimized and updated by calculating the gradients of the loss function G and the loss function D respectively, thereby adjusting the structural parameters of the generator network and the discriminator network.

[0076] Specifically, the gradient of the loss function G is calculated, and the gradient ascent algorithm is used to optimize the generator network structure parameters, i.e., modifying the data in Table 1. Similarly, the gradient of the loss function D is calculated, and the gradient descent algorithm is used to optimize the discriminator network structure parameters, i.e., modifying the data in Table 2. The training (or parameter modification) of the generator and discriminator is performed alternately, that is, only the network structure parameters of one of the generator and discriminator models are updated at a time.

[0077] More specifically, the parameters that need to be adjusted can be any value in the third column of Table 1 or any value in the third column of Table 2; adjust these parameters in turn until the discrimination score W reaches the preset threshold range.

[0078] Specifically, by repeatedly training the overall adversarial generative training network (i.e., the parameters in the network of the generator and discriminator), the values ​​of the spatial transformation coefficients of the mask image are continuously optimized to optimize the projection transformation of the mask until the error of the discrimination score W is within a preset threshold range. The model parameters of the overall adversarial attack training network (i.e., the generator network structure parameters in Table 1 and the discriminator network structure parameters in Table 2) are saved, and the trained generator and discriminator are used as the final synthesis model of the face mask image.

[0079] Specifically, if the score W is not within the preset threshold range, then repeat steps two through six until the requirements are met.

[0080] The synthetic model described in this application can quickly and accurately generate a large number of relatively realistic face images with masks. These generated face images with masks can be used as a training set for subsequent face recognition algorithms, solving the problem of insufficient image data of faces wearing masks in existing face recognition datasets, and ultimately improving the accuracy of face recognition algorithms with mask occlusion.

[0081] Generative adversarial networks are currently widely used in the field of data augmentation, so they can effectively generate a large number of images of faces and masks to enrich the training set of feature extraction networks, thus avoiding the problem of inaccurate model algorithms caused by insufficient training set.

[0082] Example 2

[0083] This embodiment provides a training device for a synthetic model of a face mask image. The device includes: an image acquisition module, a spatial transformation coefficient confirmation module, a face mask image confirmation module, a discrimination score calculation module, a model confirmation module, and a parameter adjustment module.

[0084] The image acquisition module is used to acquire images of faces without masks and images of people wearing masks.

[0085] A spatial transformation coefficient confirmation module is used to construct a generator and train the spatial transformation coefficients of the mask image based on the generator; wherein, the generator has generator network structure parameters.

[0086] The face mask image confirmation module is used to deform the mask image based on the above spatial transformation coefficients, and then fuse the deformed mask image with the face image without a mask to obtain a preliminary face mask image.

[0087] The discrimination score calculation module is used to construct a discriminator and use the discriminator to discriminate the initial face mask image to obtain a discrimination score W; the discriminator includes discriminator network structure parameters.

[0088] The model verification module is used to determine whether the network parameters in the generator and discriminator are qualified if the discrimination score W is within a preset threshold range, and to use the generator and discriminator as the synthesis model of the final face mask image.

[0089] The parameter adjustment module is used to adjust the generator network structure parameters and the discriminator network structure parameters if the discrimination score W is not within the preset threshold range.

[0090] The training device for the synthetic model of face mask images provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned training method embodiment for the synthetic model of face mask images. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0091] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a synthetic model of facial mask images, characterized in that, The method includes: step S102, acquiring an image of a face without a mask and an image of a mask; Step S104: Construct a generator and train it to obtain the spatial transformation coefficients of the mask image; wherein, the generator includes generator network structure parameters; Step S106: Based on the spatial transformation coefficients, deform the mask image, and fuse the deformed mask image with the face image without a mask to obtain a preliminary face mask image; Step S108: Construct a discriminator and use the discriminator to discriminate the preliminary face mask image to obtain a discrimination score W; wherein, the discriminator includes discriminator network structure parameters; Step S110: If the discrimination score W is within the preset threshold range, then the network parameters in the generator and discriminator are qualified, and the generator and discriminator are used as the synthesis model of the final face mask image. Step S112: If the discrimination score W is not within the preset threshold range, adjust the generator network structure parameters and the discriminator network structure parameters. S104 includes: Build generators; The acquired images of faces without masks and images with masks are superimposed by adjusting the number of channels to obtain a superimposed image; The superimposed image extracts features from the superimposed image using a large 7*7 convolutional kernel in the generator network structure parameters to obtain the target features; Information is extracted by stacking the target features using 3x3 intermediate convolutions in the generator network structure parameters, thus obtaining the preliminary feature information of the superimposed image; Based on the preliminary feature information of the superimposed image, the pooled features are obtained through 1*1 global adaptive smooth pooling in the generator network structure parameters, and then the spatial transformation coefficients are obtained.

2. The method according to claim 1, characterized in that, Step S106 includes: Use a preset face detector to detect preset facial landmarks and the position and pose of the deformed mask in an image of a face without a mask, and then align them. In the aligned image, the pixel values ​​of the masked area are set to 0, and the pixel values ​​of the non-masked area are set to 1 to obtain the mask image; Multiply the face image without a mask and the image with a deformed mask by the mask image respectively to obtain the first mask image and the second mask image respectively; The first mask image and the second mask image are weighted and fused to obtain a preliminary face mask image.

3. The method according to claim 1, characterized in that, Step S108 includes: Construct a discriminator; The pre-acquired real face image of a person wearing a mask and the preliminary face image of a person wearing a mask generated by the generator are input into the discriminator to obtain the discrimination score W, the loss function G of the generator and the loss function D of the discriminator.

4. The method according to claim 3, characterized in that, Step S108 also includes: The pre-acquired real face image with mask and the preliminary face mask image generated by the generator are input into the discriminator. The features of the real face image with mask and the preliminary face mask image generated by the generator are extracted by the 7*7 large convolution kernel in the discriminator network structure parameters, respectively, to obtain the first target feature and the second target feature. Information is extracted by stacking the first target feature and the second target feature through a 3*3 intermediate convolution of the discriminator network structure parameters, thus obtaining the first target information and the second target information. Based on the first target information and the second target information, 1*1 global adaptive smooth pooling is used to obtain the first pooled features and the second pooled features respectively; The features after the first pooling are convolved by 3*3 to obtain a 5*5*1 feature matrix Pr; similarly, the features after the second pooling are convolved by 3*3 to obtain a 5*5*1 feature matrix Pg. Calculate the discrimination score W based on the feature matrix Pr and the feature matrix Pg; When W(Pr, Pg) approaches 0, we obtain the loss function G of the generator and the loss function D of the discriminator.

5. The method according to claim 3, characterized in that, Step S112 includes: The generator network structure parameters are optimized using the gradient ascent algorithm by calculating the gradient of the loss function G. Similarly, calculate the gradient of the loss function D, and use the gradient descent algorithm to optimize the discriminator network structure parameters; The generator network structure parameters and the discriminator network structure parameters are adjusted alternately.

6. A training device for a synthetic model of a human face mask image, characterized in that, The training device includes: The image acquisition module is used to acquire images of faces without masks and images of people wearing masks. A spatial transformation coefficient confirmation module is used to construct a generator and train the generator to obtain the spatial transformation coefficients of the mask image; wherein, the generator includes generator network structure parameters; The face mask image confirmation module is used to deform the mask image based on the spatial transformation coefficient, and then fuse the deformed mask image with the face image without a mask to obtain a preliminary face mask image. The discrimination score calculation module is used to construct a discriminator and use the discriminator to discriminate the initial face mask image to obtain a discrimination score W; wherein, the discriminator includes discriminator network structure parameters; The model verification module is used to determine whether the network parameters in the generator and discriminator are qualified if the discrimination score W is within a preset threshold range, and to use the generator and discriminator as the synthesis model of the final face mask image. The parameter adjustment module is used to adjust the generator network structure parameters and the discriminator network structure parameters if the discrimination score W is not within the preset threshold range. The spatial transformation coefficient confirmation module is also used to construct a generator; to superimpose the acquired unmasked face image and masked image by increasing the number of channels to obtain a superimposed image; to extract features from the superimposed image using a 7*7 large convolution kernel in the generator network structure parameters to obtain target features; to extract information by stacking the target features using a 3*3 intermediate convolution in the generator network structure parameters to obtain preliminary feature information of the superimposed image; and to obtain pooled features by using a 1*1 global adaptive smooth pooling in the generator network structure parameters, thereby obtaining spatial transformation coefficients.

Images