Method, system and equipment for identifying and removing glasses based on deep learning

Abstract

The invention discloses a deep learning-based glasses identification and removal method, system and device, and the method comprises the steps: inputting a face image set into a glasses dictionary for identification, and obtaining a glasses-wearing face image set and a glasses-free face image set; dividing the glasses-wearing face image set according to a first divider to obtain a non-transparent glasses face image set and a transparent glasses face image set; dividing the transparent glasses face image set according to a second divider to obtain a rimless glasses face image set, a half-frame glasses face image set and a full-frame glasses face image set; constructing a glasses removal model based on the generative adversarial network; and inputting the face image set into a corresponding glasses removal model for glasses removal to obtain a corresponding glasses-free face image set. According to the invention, glasses removal based on multiple features is realized, and the generated glasses-free face image keeps high consistency in identity features and visual effects.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision and image processing, and in particular to a method, system and device for glasses recognition and glasses removal based on deep learning. Background Art

[0002] In daily life, glasses, as a common accessory, often appear in people's facial photos, but this often brings technical challenges. First, the occlusion of glasses may not only obscure key facial features such as the eyes, but also cast shadows on the face. These factors significantly reduce the accuracy of technologies such as image-based face recognition, pupil localization, and 3D face reconstruction. In addition, for aesthetic reasons, many users expect to be able to remove glasses when editing portraits to achieve better visual effects.

[0003] To address this problem, it is particularly important to develop automatic portrait glasses removal technology. However, this technology faces several technical challenges. The first challenge is to restore the facial area occluded by glasses and ensure that it is consistent with the surrounding skin in terms of details and reflectivity; secondly, simply restoring the occluded area is not enough to achieve visual authenticity, because glasses also bring a series of complex lighting effects, such as shadows, reflections, and distortion of facial features.

[0004] Considering the popularity of face recognition in daily applications, the impact of glasses occlusion on the accuracy of face recognition systems cannot be ignored. Glasses occlusion often covers up key facial information, causing changes in facial feature values, which in turn reduces recognition accuracy. In particular, different types of glasses have different effects on face occlusion, which greatly affects the effect of glasses removal. Therefore, the development of face glasses removal technology is of great significance for improving the accuracy and user satisfaction of technologies such as face recognition technology. Summary of the invention

[0005] The purpose of the present invention is to generate a face image with glasses removed and keep it highly similar to the original face image with glasses. In order to achieve the above purpose, the present invention provides a method, system and device for glasses recognition and glasses removal based on deep learning.

[0006] In a first aspect, an embodiment of the present invention provides a glasses recognition and glasses removal method based on deep learning, comprising:

[0007] Inputting the face image set into the glasses dictionary for identification, obtaining a face image set of people wearing glasses and a face image set of people not wearing glasses, wherein the glasses dictionary is a first convolutional neural network for identifying whether there are glasses in the face image;

[0008] Dividing the set of face images of people wearing glasses according to a first divider to obtain a set of face images of people wearing opaque glasses and a set of face images of people wearing transparent glasses, wherein the first divider is a second convolutional neural network trained by the training set of face images of people wearing opaque glasses and the training set of face images of people wearing transparent glasses;

[0009] The transparent glasses face image set is divided according to a second divider to obtain a frameless glasses face image set, a half-frame glasses face image set and a full-frame glasses face image set, wherein the second divider is a third convolutional neural network trained with the frameless glasses face image training set, the half-frame glasses face image training set and the full-frame glasses face image training set;

[0010] Based on the generative adversarial network, an opaque glasses removal model, a frameless glasses removal model, a half-frame glasses removal model and a full-frame glasses removal model are respectively constructed, wherein the opaque glasses removal model comprises a glasses encoder, a face encoder, a decoder and a discriminator, the frameless glasses removal model comprises a first generator, a discriminator and a verifier, the half-frame glasses removal model comprises a second generator, a discriminator and a verifier, the full-frame glasses removal model comprises a third generator, a discriminator and a verifier, the first generator comprises at least a frameless glasses decoder, the second generator comprises at least a half-frame glasses decoder, and the third generator comprises at least a full-frame glasses decoder;

[0011] The opaque glasses face image set, frameless glasses face image set, half-frame glasses face image set and full-frame glasses face image set are respectively input into corresponding glasses removal models to remove glasses, and obtain corresponding glasses-free face image sets.

[0012] Preferably, before inputting the face image set into the glasses dictionary for identification to obtain the face image set of people wearing glasses and the face image set of people not wearing glasses, the method further comprises:

[0013] A glasses dictionary is constructed based on a training set of face images of people wearing glasses and a training set of face images of people not wearing glasses.

[0014] Preferably, the glasses dictionary is constructed based on a training set of face images of people wearing glasses and a training set of face images of people not wearing glasses, including:

[0015] Acquire multiple images of faces wearing glasses to form a training set of images of faces wearing glasses;

[0016] Acquire multiple face images without glasses to form a training set of face images without glasses;

[0017] The first convolutional neural network is trained according to the training set of face images of people wearing glasses and the training set of face images of people not wearing glasses to obtain a glasses dictionary.

[0018] Preferably, an opaque glasses removal model is constructed based on a generative adversarial network, including:

[0019] Building an initial opaque glasses removal model based on a generative adversarial network, wherein the initial opaque glasses removal model includes a glasses encoder, a face encoder, a decoder, and a discriminator;

[0020] Constructing a target loss function of the initial opaque glasses removal model based on multiple loss functions, wherein the multiple loss functions at least include a face image synthesis loss function, a glasses area loss function, a facial area loss function, and an adversarial loss function;

[0021] Acquire multiple face images without glasses to form a training set of face images without glasses;

[0022] The initial opaque glasses removal model is trained using the training set of face images without glasses and the training set of face images with opaque glasses to obtain an opaque glasses removal model, wherein the training includes optimizing the target loss function using the training set of face images without glasses and the training set of face images with opaque glasses.

[0023] Preferably, a frameless glasses removal model is constructed based on a generative adversarial network, including:

[0024] Constructing an initial rimless glasses removal model based on a generative adversarial network, wherein the initial rimless glasses removal model includes a first generator, a discriminator, and a verifier, wherein the first generator includes at least a rimless glasses decoder;

[0025] Constructing a target loss function of the initial frameless glasses removal model based on multiple loss functions, wherein the multiple loss functions at least include an identity consistency loss function, a pixel-level loss function, a glasses mask loss function, a glasses shadow mask loss function, and an adversarial loss function;

[0026] The initial rimless glasses removal model is trained using the rimless glasses face image training set to obtain a rimless glasses removal model, wherein the training includes optimizing the target loss function using the rimless glasses face image training set.

[0027] Preferably, a half-frame glasses removal model is constructed based on a generative adversarial network, including:

[0028] Constructing an initial half-frame glasses removal model based on a generative adversarial network, wherein the initial half-frame glasses removal model includes a second generator, a discriminator, and a verifier, wherein the second generator includes at least a half-frame glasses decoder;

[0029] Constructing a target loss function of the initial half-frame glasses removal model based on multiple loss functions, wherein the multiple loss functions at least include an identity consistency loss function, a pixel-level loss function, a glasses mask loss function, a glasses shadow mask loss function, and an adversarial loss function;

[0030] The initial half-frame glasses removal model is trained using the half-frame glasses face image training set to obtain a half-frame glasses removal model, wherein the training includes optimizing the target loss function using the half-frame glasses face image training set.

[0031] Preferably, a full-frame glasses removal model is constructed based on a generative adversarial network, including:

[0032] Constructing an initial full-frame glasses removal model based on a generative adversarial network, wherein the initial full-frame glasses removal model includes a third generator, a discriminator, and a verifier, wherein the third generator includes at least a full-frame glasses decoder;

[0033] Constructing a target loss function of the initial full-frame glasses removal model based on multiple loss functions, wherein the multiple loss functions at least include an identity consistency loss function, a pixel-level loss function, a glasses mask loss function, a glasses shadow mask loss function, and an adversarial loss function;

[0034] The initial full-frame glasses removal model is trained using the full-frame glasses face image training set to obtain a full-frame glasses removal model, wherein the training includes optimizing the target loss function using the full-frame glasses face image training set.

[0035] Preferably, the opaque glasses face image set, frameless glasses face image set, half-frame glasses face image set and full-frame glasses face image set are respectively input into corresponding glasses removal models to remove glasses, and obtain corresponding glasses-free face image sets, including:

[0036] Remove glasses from the opaque glasses face image set according to the opaque glasses removal model to obtain a first non-glasses face image set;

[0037] Performing glasses removal on the frameless glasses face image set according to the frameless glasses removal model to obtain a second frameless glasses face image set;

[0038] Performing glasses removal on the half-frame glasses face image set according to the half-frame glasses removal model to obtain a third non-glasses face image set;

[0039] The glasses are removed from the full-frame glasses face image set according to the full-frame glasses removal model to obtain a fourth non-glasses face image set.

[0040] In a second aspect, an embodiment of the present invention provides a glasses recognition and glasses removal system based on deep learning, comprising:

[0041] A glasses identification module, used to input a face image set into a glasses dictionary for identification, and obtain a face image set of people wearing glasses and a face image set of people not wearing glasses, wherein the glasses dictionary is a first convolutional neural network used to identify whether there are glasses in a face image;

[0042] A glasses segmentation module, used for segmenting the glasses-wearing face image set according to a first segmenter to obtain an opaque glasses face image set and a transparent glasses face image set, wherein the first segmenter is a second convolutional neural network trained by the opaque glasses face image training set and the transparent glasses face image training set;

[0043] a transparent glasses segmentation module, configured to segment the transparent glasses face image set according to a second segmenter to obtain a frameless glasses face image set, a half-frame glasses face image set, and a full-frame glasses face image set, wherein the second segmenter is a third convolutional neural network trained with the frameless glasses face image training set, the half-frame glasses face image training set, and the full-frame glasses face image training set;

[0044] A glasses removal model construction module, used to respectively construct an opaque glasses removal model, a frameless glasses removal model, a half-frame glasses removal model and a full-frame glasses removal model based on a generative adversarial network, wherein the opaque glasses removal model comprises a glasses encoder, a facial encoder, a decoder and a discriminator, the frameless glasses removal model comprises a first generator, a discriminator and a verifier, the half-frame glasses removal model comprises a second generator, a discriminator and a verifier, the full-frame glasses removal model comprises a third generator, a discriminator and a verifier, the first generator comprises at least a frameless glasses decoder, the second generator comprises at least a half-frame glasses decoder, and the third generator comprises at least a full-frame glasses decoder;

[0045] The glasses removal module is used to input the opaque glasses face image set, frameless glasses face image set, half-frame glasses face image set and full-frame glasses face image set into the corresponding glasses removal model to remove the glasses and obtain the corresponding glasses-free face image set.

[0046] In a third aspect, an embodiment of the present invention provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the glasses recognition and glasses removal method as described above when executing the computer program.

[0047] Compared with the prior art, the glasses recognition and glasses removal method, system and device based on deep learning in the embodiments of the present invention have the following beneficial effects: by optimizing the loss function and the model structure, the adaptability to glasses with different features is improved, and glasses removal based on multiple features is realized. The generated face images without glasses maintain a high degree of consistency in both identity features and visual effects, providing strong support for face recognition technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of a glasses recognition and glasses removal method based on deep learning according to an embodiment of the present invention;

[0049] Figure 2 is a schematic diagram of a process of constructing an opaque glasses removal model based on a generative adversarial network according to an embodiment of the present invention;

[0050] Figure 3 is a schematic structural diagram of an opaque glasses removal model according to an embodiment of the present invention;

[0051] Figure 4 It is a schematic diagram of a process of constructing a frameless glasses removal model based on a generative adversarial network according to an embodiment of the present invention;

[0052] Figure 5 is a schematic structural diagram of a frameless glasses removal model according to an embodiment of the present invention;

[0053] Figure 6 It is a schematic diagram of a process of constructing a half-frame glasses removal model based on a generative adversarial network according to an embodiment of the present invention;

[0054] Figure 7 is a structural schematic diagram of a half-frame glasses removal model according to an embodiment of the present invention;

[0055] Figure 8 It is a schematic diagram of a process of constructing a full-frame glasses removal model based on a generative adversarial network according to an embodiment of the present invention;

[0056] Fig. 9 is a structural schematic diagram of a full-frame glasses removal model according to an embodiment of the present invention;

[0057] Fig.10 is a schematic diagram of the process of removing glasses according to an embodiment of the present invention;

[0058] Fig.11 is a structural schematic diagram of a glasses recognition and glasses removal system based on deep learning provided by an embodiment of the present invention;

[0059] Fig.12 It is a schematic diagram of the structure of a terminal device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0061] like Figure 1 As shown, an embodiment of the present invention provides a glasses recognition and glasses removal method based on deep learning, comprising the steps of:

[0062] S1, inputting the face image set into the glasses dictionary for identification, and obtaining a face image set of people wearing glasses and a face image set of people not wearing glasses;

[0063] Specifically, the glasses dictionary is a first convolutional neural network used to identify whether there are glasses in the face image. Before step S1, it also includes:

[0064] A glasses dictionary is constructed based on a training set of face images of people wearing glasses and a training set of face images of people not wearing glasses.

[0065] Specifically, the process of building a glasses dictionary is as follows:

[0066] 1) obtaining multiple images of faces wearing glasses to form a training set of images of faces wearing glasses;

[0067] 2) obtaining multiple face images without glasses to form a training set of face images without glasses;

[0068] 3) The first convolutional neural network is trained according to the training set of face images with glasses and the training set of face images without glasses to obtain a glasses dictionary.

[0069] Specifically, the eye region of each face image with glasses in the training set of face images with glasses is annotated by an annotation tool, and then the eye region is converted into a binary mask, and the binary mask is stored in the eye mask database. The eye mask database can be used to locate the position of the eye in the face image with glasses, so as to extract the eye glasses with different features. The first convolutional neural network is trained with the obtained training set of face images with different features of glasses and without glasses to obtain the eye glasses dictionary.

[0070] S2, dividing the image set of people wearing glasses according to the first divider to obtain a set of people wearing opaque glasses and a set of people wearing transparent glasses;

[0071] The glasses mask database can be used to locate the glasses in the face image of a person wearing glasses, thereby extracting glasses with different features. The glasses with different features are divided into two categories, namely opaque glasses and transparent glasses. In order to effectively identify opaque glasses and transparent glasses, a first divider is designed in this step to divide opaque glasses and transparent glasses.

[0072] Specifically, the first divider is a second convolutional neural network trained by a training set of facial images of people wearing opaque glasses and a training set of facial images of people wearing transparent glasses.

[0073] Furthermore, the process of constructing the first divider is as follows:

[0074] 1) obtaining multiple images of people wearing opaque glasses to form a training set of people wearing opaque glasses;

[0075] 2) obtaining multiple transparent glasses face images to form a transparent glasses face image training set;

[0076] 3) The second convolutional neural network is trained according to the training set of face images of people wearing opaque glasses and the training set of face images of people wearing transparent glasses to obtain a first divider.

[0077] Specifically, the eye region of each opaque eyeglass face image in the opaque eyeglass face image training set and the eye region of each transparent eyeglass face image in the transparent eyeglass face image training set are annotated by an annotation tool, and then the eye region is converted into a binary mask, and the binary mask is stored in a eyeglass mask database. The eye positions in the opaque eyeglass face image and the transparent eyeglass face image can be located by the eyeglass mask database, so as to extract the opaque eyeglasses and the transparent eyeglasses. The second convolutional neural network is trained by the obtained opaque eyeglasses and transparent eyeglasses to obtain the first divider.

[0078] S3, dividing the transparent glasses face image set according to the second divider to obtain a frameless glasses face image set, a half-frame glasses face image set and a full-frame glasses face image set;

[0079] Since the degree of occlusion of the face by the glasses frame is different, the transparent glasses are further divided into frameless glasses, half-frame glasses and full-frame glasses. In order to effectively identify frameless glasses, half-frame glasses and full-frame glasses, a second classifier is designed in this step to classify frameless glasses, half-frame glasses and full-frame glasses.

[0080] Specifically, the second divider is a third convolutional neural network trained with a training set of face images of people with frameless glasses, a training set of face images of people with half-frame glasses, and a training set of face images of people with full-frame glasses.

[0081] Furthermore, the process of constructing the second divider is as follows:

[0082] 1) obtaining multiple face images of people with frameless glasses to form a training set of face images of people with frameless glasses;

[0083] 2) obtaining multiple half-frame glasses face images to form a half-frame glasses face image training set;

[0084] 3) obtaining multiple full-frame glasses face images to form a full-frame glasses face image training set;

[0085] 4) The third convolutional neural network is trained according to the training set of face images of frameless glasses, the training set of face images of half-frame glasses, and the training set of face images of full-frame glasses to obtain a second divider.

[0086] Specifically, the eyeglasses region of each frameless eyeglasses face image in the frameless eyeglasses face image training set, the eyeglasses region of each half-frame eyeglasses face image in the half-frame eyeglasses face image training set, and the eyeglasses region of each full-frame eyeglasses face image in the full-frame eyeglasses face image training set are annotated by an annotation tool, and then the eyeglasses region is converted into a binary mask, and the binary mask is stored in a eyeglasses mask database. The eyeglasses position in the frameless eyeglasses face image, the half-frame eyeglasses face image, and the full-frame eyeglasses face image can be located by the eyeglasses mask database, so as to extract the frameless eyeglasses, the half-frame eyeglasses, and the full-frame eyeglasses. The third convolutional neural network is trained by the obtained frameless eyeglasses, the half-frame eyeglasses, and the full-frame eyeglasses to obtain the second divider.

[0087] S4. Based on the generative adversarial network, opaque glasses removal model, frameless glasses removal model, half-frame glasses removal model and full-frame glasses removal model are constructed respectively;

[0088] Generative Adversarial Networks (GAN) is a deep learning model that learns through the mutual confrontation of two neural networks (generator and discriminator) to achieve the purpose of generating new data with a distribution similar to the real data.

[0089] An opaque glasses removal model is constructed based on a generative adversarial network, where the opaque glasses removal model includes a glasses encoder, a facial encoder, a decoder, and a discriminator. Specifically, the process of constructing an opaque glasses removal model based on a generative adversarial network is as follows: Figure 2 As shown, the steps include:

[0090] S401-A, constructing an initial opaque glasses removal model based on a generative adversarial network;

[0091] The initial opaque glasses removal model includes glasses encoder, face encoder, decoder and discriminator. Specifically, the description of each neural network of the model is as follows:

[0092] 1) Glasses encoder and face encoder: Both the glasses encoder and the face encoder are composed of several convolutional layers and residual blocks. Since the face area is divided into two parts, the glasses area and the non-glasses area (i.e., the face area), the glasses area is first covered with a mask, and then the face encoder is used to encode the face area. Finally, the glasses encoder is used to encode the glasses area using the mask;

[0093] 2) Decoder: The decoder is also composed of convolutional layers, and a refinement block is added at the end. The decoder is used to decode the combination of the glasses area code and the face area code to generate a face image with the glasses removed;

[0094] 3) Discriminator: The discriminator uses a fully convolutional model, which distinguishes the decoder’s generated images from the original images while ensuring that the generated images are more realistic.

[0095] S401-B, constructing the target loss function of the initial opaque glasses removal model based on multiple loss functions;

[0096] The multiple loss functions include at least a face image synthesis loss function, a glasses area loss function, a facial area loss function, and an adversarial loss function. Specifically, the multiple loss functions are as follows:

[0097] 1) Face image synthesis loss function: The opaque glasses removal model synthesizes the facial region encoding of the opaque glasses face image and the glasses region encoding of the face image without glasses through the decoder, thereby generating a face image with glasses removed, that is, ensuring that the synthesized image is similar to the opaque glasses face image. This embodiment uses the L1 regression loss function to represent the face image synthesis loss function;

[0098] 2) Glasses area loss function: In order to strengthen the model's attention to the glasses area, the glasses area loss function is introduced. This embodiment uses a binary cross entropy loss function to represent the glasses area loss function;

[0099] 3) Facial region loss function: In order to strengthen the model's attention to the facial region, the facial region loss function is introduced. This embodiment uses a binary cross entropy loss function to represent the facial region loss function;

[0100] 4) Adversarial loss function: In order to make the synthesized image more similar to the original image, that is, the synthesized image is more realistic, an adversarial loss function is introduced;

[0101] 5) Dice loss function: In image segmentation tasks, the prediction result is usually represented as a binary image, where the pixel value is 0 or 1. The Dice loss function measures the similarity between the prediction result and the binary image of the true label by comparing them. The smaller the Dice loss, the higher the similarity between the prediction result and the true label;

[0102] 6) Focal loss function: This function is a loss function widely used in target detection and image segmentation tasks. It is specifically used to deal with category imbalance problems and can improve the model's prediction effect for fewer categories.

[0103] The above six loss functions are weighted summed to obtain the target loss function of the initial opaque glasses removal model. This embodiment uses the following formula to represent the target loss function:

[0104] L GR =ρ C L C +ρ G L G +ρ F L F +ρ AD L AD +ρ DICE L DICE +ρ FOCAL L FOCAL

[0105] Among them, L GR represents the target loss function, ρ C , G , F , AD , DICE , FOCAL Represents the corresponding weight coefficient, L C represents the face image synthesis loss function, L G represents the glasses area loss function, L F represents the facial region loss function, L AD represents the adversarial loss function, L DICE represents the Dice loss function, L FOCAL Represents the Focal loss function.

[0106] S401-C, obtaining multiple images of faces without glasses to form a training set of images of faces without glasses;

[0107] S401-D. Train the initial opaque glasses removal model using a training set of face images without glasses and a training set of face images with opaque glasses to obtain an opaque glasses removal model.

[0108] The training includes optimizing the target loss function through the training set of face images without glasses and the training set of face images with opaque glasses. The initial opaque glasses removal model is trained through the training set of face images without glasses and the training set of face images with opaque glasses, and the target loss function is continuously optimized to obtain the opaque glasses removal model. The structure of the opaque glasses removal model can be found in Figure 3 .

[0109] A rimless glasses removal model is constructed based on a generative adversarial network, wherein the rimless glasses removal model includes a first generator, a discriminator, and a verifier, and the first generator includes at least a rimless glasses decoder. Specifically, the process of constructing a rimless glasses removal model based on a generative adversarial network is as follows: Figure 4 As shown, the steps include:

[0110] S402-A, constructing an initial frameless glasses removal model based on a generative adversarial network;

[0111] The initial rimless glasses removal model includes a first generator, a discriminator and a verifier, and the first generator includes at least a rimless glasses decoder. Specifically, the description of each neural network of the model is as follows:

[0112] 1) The first generator: The first generator uses a U-Net structure, which can greatly reduce information loss, including an encoder and a decoder. At the same time, a jump connection is used between the encoder and the decoder. The first generator is used to generate a face image with glasses removed based on a face image without frameless glasses;

[0113] Encoder: The encoder is mainly constructed using convolutional layer components, and its purpose is to extract features from and encode face images of frameless glasses;

[0114] Decoder: The decoder is also constructed by convolutional layer components, which includes a face decoder, a frameless glasses decoder, and a glasses shadow decoder. The frameless glasses decoder and the glasses shadow decoder generate binary masks of the glasses area and the glasses shadow area, respectively. Due to the jump connection between the encoder and the decoder, the feature information can be shared among the three decoders. The frameless glasses decoder and the glasses shadow decoder can guide the face decoder to remove the glasses and the glasses shadows in turn, thereby generating a face image with the glasses removed;

[0115] 2) Discriminator: The discriminator is a full convolution model including multiple convolution blocks, and its function is to measure the difference between the generated face image of the first generator and the original face image. Specifically, the discriminator of this embodiment is a full convolution model including 5 convolution blocks, which extracts the features of the generated face image and the original face image, and uses the cosine distance to measure the difference in facial features in the feature space. The authenticity of the generated face image is judged according to a preset threshold, that is, the probability that the generated face image is the original face image is calculated through the sigmoid function output by the last convolution block of the discriminator, and it is judged whether the probability is higher than the preset threshold. If so, the generated face image is judged to be authentic;

[0116] 3) Verifier: The verifier calculates the Euclidean distance between the identity feature vectors of the face image without rimless glasses and the face image with glasses removed and minimizes the distance, thereby ensuring that the face image with glasses removed generated by the first generator is consistent with the original face image in terms of identity features.

[0117] S402-B, constructing a target loss function of the initial rimless glasses removal model based on multiple loss functions;

[0118] The multiple loss functions include at least identity consistency loss function, pixel-level loss function, glasses mask loss function, glasses shadow mask loss function and adversarial loss function. Specifically, the multiple loss functions are as follows:

[0119] 1) Identity consistency loss function: The identity consistency loss function is constructed by comparing the features of the generated face image and the original face image. Specifically, this embodiment uses a pre-trained identity feature extractor to extract the features of the generated face image and the original face image, and then calculates the Euclidean distance between them. By minimizing the Euclidean distance between the generated face image and the original face image, the two images are forced to share the same identity information;

[0120] 2) Pixel-level loss function: The goal of the first generator is to generate a synthetic image that is close to the original image. To achieve this goal, a pixel-level loss function can be used to quantify the difference between the generated image and the original image. This embodiment uses the L1 regression loss function to represent the pixel-level loss function. By calculating the L1 distance between the generated image and the original image, it is helpful to enhance the glasses removal capability of the first generator;

[0121] 3) Glasses mask loss function: This embodiment uses a binary cross entropy loss function to represent the glasses mask loss function;

[0122] 4) Glasses shadow mask loss function: This embodiment uses a binary cross entropy loss function to represent the glasses shadow mask loss function;

[0123] 5) Adversarial loss function: This embodiment uses the cross entropy loss function to represent the adversarial loss function, which is used to measure the probability that the generated image is judged as false. The goal of the first generator is to minimize this loss function, while the goal of the discriminator is to maximize this loss function;

[0124] 6) Dice loss function: In image segmentation tasks, the prediction result is usually represented as a binary image, where the pixel value is 0 or 1. The Dice loss function measures the similarity between the prediction result and the binary image of the true label by comparing them. The smaller the Dice loss, the higher the similarity between the prediction result and the true label;

[0125] 7) Focal loss function: This function is a loss function widely used in target detection and image segmentation tasks. It is specifically used to deal with category imbalance problems and can improve the model's prediction effect for fewer categories.

[0126] The above seven loss functions are weighted and summed to obtain the target loss function of the initial frameless glasses removal model. This embodiment uses the following formula to represent the target loss function:

[0127] L ER_1 =ρ ID_1 L ID_1 +ρ P_1 L P_1 +ρ E_1 L E_1 +ρ s_1 L s_1 +ρ ADV_1 L ADV_1 +ρ DICE_1 L DICE_1 +ρ FOCAL_1 L FOCAL_1

[0128] Among them, L ER_1 represents the target loss function, ρ ID_1 , P_1 , E_1 , S_1 , ADV_1 , DICE_1 , FOCAL_1 Represents the corresponding weight coefficient, L ID_1 represents the identity consistency loss function, L P_1 represents the pixel-level loss function, L E_1 represents the glasses mask loss function, L S_1 represents the glasses shadow mask loss function, L ADV_1 represents the adversarial loss function, L DICE_1 represents the Dice loss function, L FOCAL_1 Represents the Focal loss function.

[0129] S402-C, training the initial rimless glasses removal model using the rimless glasses face image training set to obtain a rimless glasses removal model.

[0130] The training includes optimizing the target loss function through the training set of rimless glasses face images. The initial rimless glasses removal model is trained through the rimless glasses face image training set, and the target loss function is continuously optimized to obtain the rimless glasses removal model. The structure of the rimless glasses removal model can be found in Figure 5 .

[0131] A half-frame glasses removal model is constructed based on a generative adversarial network, wherein the half-frame glasses removal model includes a second generator, a discriminator, and a verifier, and the second generator includes at least a half-frame glasses decoder. Specifically, the process of constructing a half-frame glasses removal model based on a generative adversarial network is as follows: Figure 6 As shown, the steps include:

[0132] S403-A, constructing an initial half-frame glasses removal model based on a generative adversarial network;

[0133] The initial half-frame glasses removal model includes a second generator, a discriminator and a verifier, and the second generator includes at least a half-frame glasses decoder. Specifically, the description of each neural network of the model is as follows:

[0134] 1) Second Generator: The second generator uses a U-Net structure, which can greatly reduce information loss, including encoder and decoder. At the same time, a jump connection is used between the encoder and decoder. The second generator is used to generate a face image with glasses removed based on a face image with half-frame glasses;

[0135] Encoder: The encoder is mainly constructed using convolutional layer components, and its purpose is to extract features from half-frame glasses face images and encode them;

[0136] Decoder: The decoder is also constructed by convolutional layer components, which includes a face decoder, a half-frame glasses decoder, and a glasses shadow decoder. The half-frame glasses decoder and the glasses shadow decoder generate binary masks of the glasses area and the glasses shadow area, respectively. Due to the jump connection between the encoder and the decoder, the feature information can be shared among the three decoders. The half-frame glasses decoder and the glasses shadow decoder can guide the face decoder to remove the glasses and the glasses shadows in turn, thereby generating a face image with the glasses removed;

[0137] 2) Discriminator: The discriminator is a full convolution model including multiple convolution blocks, and its function is to measure the difference between the generated face image of the second generator and the original face image. Specifically, the discriminator of this embodiment is a full convolution model including 5 convolution blocks, which extracts the features of the generated face image and the original face image, and uses the cosine distance to measure the difference in facial features in the feature space. The authenticity of the generated face image is judged according to a preset threshold, that is, the probability that the generated face image is the original face image is calculated through the sigmoid function output by the last convolution block of the discriminator, and it is judged whether the probability is higher than the preset threshold. If so, the generated face image is judged to be authentic;

[0138] 3) Verifier: The verifier calculates the Euclidean distance between the identity feature vectors of the face image with half-frame glasses and the face image with glasses removed and minimizes the distance, thereby ensuring that the face image with glasses removed generated by the second generator is consistent with the original face image in terms of identity features.

[0139] S403-B, constructing a target loss function for the initial half-frame glasses removal model based on multiple loss functions;

[0140] The multiple loss functions include at least identity consistency loss function, pixel-level loss function, glasses mask loss function, glasses shadow mask loss function and adversarial loss function. Specifically, the multiple loss functions are as follows:

[0141] 1) Identity consistency loss function: The identity consistency loss function is constructed by comparing the features of the generated face image and the original face image. Specifically, this embodiment uses a pre-trained identity feature extractor to extract the features of the generated face image and the original face image, and then calculates the Euclidean distance between them. By minimizing the Euclidean distance between the generated face image and the original face image, the two images are forced to share the same identity information;

[0142] 2) Pixel-level loss function: The goal of the second generator is to generate a synthetic image that is close to the original image. To achieve this goal, a pixel-level loss function can be used to quantify the difference between the generated image and the original image. This embodiment uses the L1 regression loss function to represent the pixel-level loss function. By calculating the L1 distance between the generated image and the original image, it is helpful to enhance the glasses removal capability of the second generator;

[0143] 3) Glasses mask loss function: This embodiment uses a binary cross entropy loss function to represent the glasses mask loss function;

[0144] 4) Glasses shadow mask loss function: This embodiment uses a binary cross entropy loss function to represent the glasses shadow mask loss function;

[0145] 5) Adversarial loss function: This embodiment uses the cross entropy loss function to represent the adversarial loss function, which is used to measure the probability that the generated image is judged as false. The goal of the second generator is to minimize this loss function, while the goal of the discriminator is to maximize this loss function;

[0146] 6) Dice loss function: In image segmentation tasks, the prediction result is usually represented as a binary image, where the pixel value is 0 or 1. The Dice loss function measures the similarity between the prediction result and the binary image of the true label by comparing them. The smaller the Dice loss, the higher the similarity between the prediction result and the true label;

[0147] 7) Focal loss function: This function is a loss function widely used in target detection and image segmentation tasks. It is specifically used to deal with category imbalance problems and can improve the model's prediction effect for fewer categories.

[0148] The above seven loss functions are weighted and summed to obtain the target loss function of the initial half-frame glasses removal model. This embodiment uses the following formula to represent the target loss function:

[0149] L ER_2 =ρ ID_2 L ID_2 +ρ P_2 L P_2 +ρ E_2 L E_2 +ρ S_2 L S_2 +ρ ADV_2 L ADV_2 +ρ DICE_2 L DICE_2 +ρ FOCAL_2 L FOCAL_2

[0150] Among them, L ER_2 represents the target loss function, ρ ID_2 , P_2 , E_2 , S_2 , ADV_2 , DICE_2 , FOCAL_2 Represents the corresponding weight coefficient, L ID_2 represents the identity consistency loss function, L P_2 represents the pixel-level loss function, L E_2 represents the glasses mask loss function, L S_2 represents the glasses shadow mask loss function, L ADV_2 represents the adversarial loss function, L DICE_2 represents the Dice loss function, L FOCAL_2 Represents the Focal loss function.

[0151] S403-C. Train the initial half-frame glasses removal model using the half-frame glasses face image training set to obtain a half-frame glasses removal model.

[0152] The training includes optimizing the target loss function through the training set of half-frame glasses face images. The initial half-frame glasses removal model is trained through the training set of half-frame glasses face images, and the target loss function is continuously optimized to obtain the half-frame glasses removal model. The structure of the half-frame glasses removal model can be found in Figure 7 .

[0153] A full-frame glasses removal model is constructed based on a generative adversarial network, wherein the full-frame glasses removal model includes a third generator, a discriminator, and a verifier, and the third generator includes at least a full-frame glasses decoder. Specifically, the process of constructing a full-frame glasses removal model based on a generative adversarial network is as follows: Figure 8 As shown, the steps include:

[0154] S404-A, constructing an initial full-frame glasses removal model based on a generative adversarial network;

[0155] The initial full-frame glasses removal model includes a third generator, a discriminator, and a verifier, and the third generator includes at least a full-frame glasses decoder. Specifically, the description of each neural network of the model is as follows:

[0156] 1) The third generator: The third generator uses a U-Net structure, which can greatly reduce information loss, including encoders and decoders. At the same time, a jump connection is used between the encoder and the decoder. The third generator is used to generate a face image with glasses removed based on a full-frame glasses face image;

[0157] Encoder: The encoder is mainly constructed using convolutional layer components, and its purpose is to extract features from full-frame glasses face images and encode them;

[0158] Decoder: The decoder is also constructed by convolutional layer components, which includes a face decoder, a full-frame glasses decoder, and a glasses shadow decoder. The full-frame glasses decoder and the glasses shadow decoder generate binary masks of the glasses area and the glasses shadow area, respectively. Due to the jump connection between the encoder and the decoder, the three decoders can share feature information. The full-frame glasses decoder and the glasses shadow decoder can guide the face decoder to remove the glasses and the glasses shadows in turn, thereby generating a face image with the glasses removed;

[0159] 2) Discriminator: The discriminator is a full convolution model including multiple convolution blocks, and its function is to measure the difference between the generated face image of the third generator and the original face image. Specifically, the discriminator of this embodiment is a full convolution model including 5 convolution blocks, which extracts the features of the generated face image and the original face image, and uses the cosine distance to measure the difference in facial features in the feature space. The authenticity of the generated face image is judged according to a preset threshold, that is, the probability that the generated face image is the original face image is calculated through the sigmoid function output by the last convolution block of the discriminator, and it is judged whether the probability is higher than the preset threshold. If so, the generated face image is judged to be authentic;

[0160] 3) Verifier: The verifier calculates the Euclidean distance between the identity feature vectors of the face image with full-frame glasses and the face image with glasses removed, and minimizes the distance, thereby ensuring that the face image with glasses removed generated by the third generator is consistent with the original face image in terms of identity features.

[0161] S404-B, constructing the target loss function of the initial full-frame glasses removal model based on multiple loss functions;

[0162] The multiple loss functions include at least identity consistency loss function, pixel-level loss function, glasses mask loss function, glasses shadow mask loss function and adversarial loss function. Specifically, the multiple loss functions are as follows:

[0163] 1) Identity consistency loss function: The identity consistency loss function is constructed by comparing the features of the generated face image and the original face image. Specifically, this embodiment uses a pre-trained identity feature extractor to extract the features of the generated face image and the original face image, and then calculates the Euclidean distance between them. By minimizing the Euclidean distance between the generated face image and the original face image, the two images are forced to share the same identity information;

[0164] 2) Pixel-level loss function: The goal of the third generator is to generate a synthetic image that is close to the original image. To achieve this goal, a pixel-level loss function can be used to quantify the difference between the generated image and the original image. This embodiment uses the L1 regression loss function to represent the pixel-level loss function. By calculating the L1 distance between the generated image and the original image, it is helpful to enhance the glasses removal capability of the third generator;

[0165] 3) Glasses mask loss function: This embodiment uses a binary cross entropy loss function to represent the glasses mask loss function;

[0166] 4) Glasses shadow mask loss function: This embodiment uses a binary cross entropy loss function to represent the glasses shadow mask loss function;

[0167] 5) Adversarial loss function: This embodiment uses the cross entropy loss function to represent the adversarial loss function, which is used to measure the probability that the generated image is judged as false. The goal of the third generator is to minimize this loss function, while the goal of the discriminator is to maximize this loss function;

[0168] 6) Dice loss function: In image segmentation tasks, the prediction result is usually represented as a binary image, where the pixel value is 0 or 1. The Dice loss function measures the similarity between the prediction result and the binary image of the true label by comparing them. The smaller the Dice loss, the higher the similarity between the prediction result and the true label;

[0169] 7) Focal loss function: This function is a loss function widely used in target detection and image segmentation tasks. It is specifically used to deal with category imbalance problems and can improve the model's prediction effect for fewer categories.

[0170] The above seven loss functions are weighted and summed to obtain the target loss function of the initial full-frame glasses removal model. This embodiment uses the following formula to represent the target loss function:

[0171] L ER_3 =ρ ID_3 L ID_3 +ρ P_3 L P_3 +ρ E_3 L E_3 +ρ S_3 L S_3 +ρ ADV_3 L ADV_3 +ρ DICE_3 L DICE_3 +ρ FOCAL_3 L FOCAL_3

[0172] Among them, L ER_3 represents the target loss function, ρ ID_3 , P_3 , E_3 , S_3 , ADV_3 , DICE_3 , FOCAL_3 Represents the corresponding weight coefficient, L ID_3 represents the identity consistency loss function, L P_3 represents the pixel-level loss function, L E_3 represents the glasses mask loss function, L S_3 represents the glasses shadow mask loss function, L ADV_3 represents the adversarial loss function, L DICE_3 represents the Dice loss function, L FOCAL_3 Represents the Focal loss function.

[0173] S404-C. Train the initial full-frame glasses removal model using the full-frame glasses face image training set to obtain a full-frame glasses removal model.

[0174] The training includes optimizing the target loss function through the full-frame glasses face image training set. The initial full-frame glasses removal model is trained through the full-frame glasses face image training set, and the target loss function is continuously optimized to obtain the full-frame glasses removal model. The structure of the full-frame glasses removal model can be found in Fig. 9 .

[0175] S5. Input the opaque glasses face image set, the frameless glasses face image set, the half-frame glasses face image set and the full-frame glasses face image set into the corresponding glasses removal model respectively to remove the glasses, and obtain the corresponding glasses-free face image set.

[0176] Specifically, Fig.10 As shown, step S5 includes:

[0177] S501, removing glasses from an opaque glasses face image set according to an opaque glasses removal model to obtain a first non-glasses face image set;

[0178] The opaque glasses removal model processes a set of face images with opaque glasses and a set of face images without glasses to synthesize a set of face images with opaque glasses removed. Specifically, the glasses encoder encodes the set of face images without glasses to obtain the glasses region codes of the set of face images without glasses; the face encoder encodes the set of face images with opaque glasses to obtain the face region codes of the set of face images with opaque glasses. The decoder combines the glasses region codes and the face region codes to obtain a synthesized image set, i.e., the set of face images with opaque glasses removed, i.e., the first set of face images without glasses. Furthermore, a discriminator is used to distinguish between the original image set and the synthesized image set. This model can effectively remove the opaque glasses from the face images with opaque glasses, thereby obtaining realistic face images without glasses.

[0179] S502, removing glasses from the face image set without rimless glasses according to the rimless glasses removal model to obtain a second face image set without glasses;

[0180] The rimless glasses removal model processes the rimless glasses face image set to synthesize the face image set with rimless glasses removed. Specifically, the encoder extracts features from the rimless glasses face image set and encodes them, generates a binary mask of the glasses area through the rimless glasses decoder, and generates a binary mask of the glasses shadow area through the glasses shadow decoder, and then the rimless glasses decoder and the glasses shadow decoder guide the face decoder to remove the glasses and glasses shadows in turn, thereby generating a face image set with rimless glasses removed, that is, the second face image set without glasses. Furthermore, a discriminator is used to ensure that the generated second face image set without glasses is more realistic by measuring the difference between the second face image set without glasses and the original face image set; and a verifier is used to improve the similarity between the second face image set without glasses and the original face image set by minimizing the Euclidean distance between the two.

[0181] S503, removing glasses from the half-frame glasses face image set according to the half-frame glasses removal model to obtain a third set of face images without glasses;

[0182] The half-frame glasses removal model processes the half-frame glasses face image set to synthesize the face image set with the half-frame glasses removed. Specifically, the encoder extracts features from the half-frame glasses face image set and encodes them, generates a binary mask of the glasses area through the half-frame glasses decoder, and generates a binary mask of the glasses shadow area through the glasses shadow decoder, and then the half-frame glasses decoder and the glasses shadow decoder guide the face decoder to remove the glasses and the glasses shadows in turn, thereby generating a face image set with the half-frame glasses removed, that is, the third face image set without glasses. Furthermore, a discriminator is used to ensure that the generated third face image set without glasses is more realistic by measuring the difference between the third face image set without glasses and the original face image set; and a verifier is used to improve the similarity between the third face image set without glasses and the original face image set by minimizing the Euclidean distance between the two.

[0183] S504 , removing glasses from the full-frame glasses face image set according to the full-frame glasses removal model to obtain a fourth non-glasses face image set.

[0184] The full-frame glasses removal model processes the full-frame glasses face image set to synthesize the face image set with full-frame glasses removed. Specifically, the encoder extracts features from the full-frame glasses face image set and encodes them, generates a binary mask of the glasses area through the full-frame glasses decoder, generates a binary mask of the glasses shadow area through the glasses shadow decoder, and then the full-frame glasses decoder and the glasses shadow decoder guide the face decoder to remove the glasses and glasses shadows successively, thereby generating a face image set with full-frame glasses removed, that is, the fourth face image set without glasses. Furthermore, a discriminator is used to ensure that the generated fourth face image set without glasses is more realistic by measuring the difference between the fourth face image set without glasses and the original face image set; and a verifier is used to improve the similarity between the fourth face image set without glasses and the original face image set by minimizing the Euclidean distance between the two. Through the frameless glasses removal model, the half-frame glasses removal model and the full-frame glasses removal model, transparent glasses can be removed more accurately to generate more realistic face images.

[0185] The embodiment of the present invention is a glasses recognition and glasses removal method based on deep learning. By optimizing the loss function and the model structure, the adaptability to glasses with different features is improved, and glasses removal based on multiple features is realized. The generated face images without glasses maintain a high degree of consistency in identity features and visual effects, providing strong support for face recognition technology.

[0186] Based on the above glasses recognition and glasses removal methods, such as Fig.11 As shown, an embodiment of the present invention provides a glasses recognition and glasses removal system based on deep learning, including:

[0187] Glasses identification module 1, used for inputting a face image set into a glasses dictionary for identification, and obtaining a face image set of people wearing glasses and a face image set of people not wearing glasses, wherein the glasses dictionary is a first convolutional neural network used for identifying whether there are glasses in a face image;

[0188] Glasses segmentation module 2, used for segmenting the image set of people wearing glasses according to the first segmentation device to obtain an opaque glasses face image set and a transparent glasses face image set, wherein the first segmentation device is a second convolutional neural network trained by the opaque glasses face image training set and the transparent glasses face image training set;

[0189] A transparent glasses segmentation module 3 is used to segment the transparent glasses face image set according to a second segmenter to obtain a frameless glasses face image set, a half-frame glasses face image set and a full-frame glasses face image set, wherein the second segmenter is a third convolutional neural network trained with the frameless glasses face image training set, the half-frame glasses face image training set and the full-frame glasses face image training set;

[0190] A glasses removal model construction module 4 is used to respectively construct an opaque glasses removal model, a frameless glasses removal model, a half-frame glasses removal model and a full-frame glasses removal model based on a generative adversarial network, wherein the opaque glasses removal model includes a glasses encoder, a facial encoder, a decoder and a discriminator, the frameless glasses removal model includes a first generator, a discriminator and a verifier, the half-frame glasses removal model includes a second generator, a discriminator and a verifier, and the full-frame glasses removal model includes a third generator, a discriminator and a verifier, the first generator includes at least a frameless glasses decoder, the second generator includes at least a half-frame glasses decoder, and the third generator includes at least a full-frame glasses decoder;

[0191] The glasses removal module 5 is used to input the opaque glasses face image set, the frameless glasses face image set, the half-frame glasses face image set and the full-frame glasses face image set into the corresponding glasses removal model to remove the glasses and obtain the corresponding glasses-free face image set.

[0192] It should be noted that each module in the above-mentioned glasses recognition and glasses removal system based on deep learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules. For the specific definition of a glasses recognition and glasses removal system based on deep learning, please refer to the definition of a glasses recognition and glasses removal method based on deep learning above. The two have the same functions and effects, which will not be repeated here.

[0193] An embodiment of the present invention further provides a terminal device, the terminal device comprising:

[0194] processor, memory, and bus;

[0195] The bus is used to connect the processor and the memory;

[0196] The memory is used to store operation instructions;

[0197] The processor is used to call the operation instruction, and the executable instruction enables the processor to perform operations corresponding to the glasses recognition and glasses removal method based on deep learning as described above in the present invention.

[0198] In an optional embodiment, a terminal device is provided, such as Fig.12 As shown, Fig.12 The terminal device 5000 shown includes: a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, such as through a bus 5002. Optionally, the terminal device 5000 may also include a transceiver 5004. It should be noted that in actual applications, the transceiver 5004 is not limited to one, and the structure of the terminal device 5000 does not constitute a limitation on the embodiments of the present invention.

[0199] Processor 5001 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 5001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0200] The bus 5002 may include a path to transmit information between the above components. The bus 5002 may be a PCI bus or an EISA bus, etc. The bus 5002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.12 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0201] The memory 5003 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an EEPROM, a CD-ROM or other optical disk storage, an optical disk storage (including a compressed optical disk, a laser disk, an optical disk, a digital versatile disk, a Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0202] The memory 5003 is used to store application code for executing the solution of the present invention, and the execution is controlled by the processor 5001. The processor 5001 is used to execute the application code stored in the memory 5003 to implement the content shown in any of the above method embodiments.

[0203] In summary, the embodiments of the present invention provide a method, system and device for glasses recognition and glasses removal based on deep learning, which improves the adaptability to glasses with different features by optimizing the loss function and model structure, realizes glasses removal based on multiple features, and generates glasses-free face images that maintain a high degree of consistency in identity features and visual effects, providing strong support for face recognition technology.

[0204] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0205] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention. These improvements and substitutions should also be regarded as the scope of protection of the present invention.

Claims

1. A glasses recognition and glasses removal method based on deep learning, characterized in that: include: Inputting the face image set into the glasses dictionary for identification, obtaining a face image set of people wearing glasses and a face image set of people not wearing glasses, wherein the glasses dictionary is a first convolutional neural network for identifying whether there are glasses in the face image; Dividing the set of face images of people wearing glasses according to a first divider to obtain a set of face images of people wearing opaque glasses and a set of face images of people wearing transparent glasses, wherein the first divider is a second convolutional neural network trained by the training set of face images of people wearing opaque glasses and the training set of face images of people wearing transparent glasses; The transparent glasses face image set is divided according to a second divider to obtain a frameless glasses face image set, a half-frame glasses face image set and a full-frame glasses face image set, wherein the second divider is a third convolutional neural network trained with the frameless glasses face image training set, the half-frame glasses face image training set and the full-frame glasses face image training set; Based on the generative adversarial network, an opaque glasses removal model, a frameless glasses removal model, a half-frame glasses removal model and a full-frame glasses removal model are respectively constructed, wherein the opaque glasses removal model comprises a glasses encoder, a face encoder, a decoder and a discriminator, the frameless glasses removal model comprises a first generator, a discriminator and a verifier, the half-frame glasses removal model comprises a second generator, a discriminator and a verifier, the full-frame glasses removal model comprises a third generator, a discriminator and a verifier, the first generator comprises at least a frameless glasses decoder, the second generator comprises at least a half-frame glasses decoder, and the third generator comprises at least a full-frame glasses decoder; The opaque glasses face image set, frameless glasses face image set, half-frame glasses face image set and full-frame glasses face image set are respectively input into corresponding glasses removal models for glasses removal to obtain corresponding glasses-free face image sets.

2. The glasses recognition and glasses removal method according to claim 1, characterized in that: Before inputting the face image set into the glasses dictionary for identification to obtain the face image set of people wearing glasses and the face image set of people not wearing glasses, the method further includes: A glasses dictionary is constructed based on a training set of face images of people wearing glasses and a training set of face images of people not wearing glasses.

3. The glasses identification and glasses removal method according to claim 2, characterized in that: The method of constructing a glasses dictionary based on a training set of face images of people wearing glasses and a training set of face images of people not wearing glasses comprises: Acquire multiple images of faces wearing glasses to form a training set of images of faces wearing glasses; Acquire multiple face images without glasses to form a training set of face images without glasses; The first convolutional neural network is trained according to the training set of face images of people wearing glasses and the training set of face images of people not wearing glasses to obtain a glasses dictionary.

4. The glasses recognition and glasses removal method according to claim 1, characterized in that: Construct an opaque glasses removal model based on a generative adversarial network, including: Building an initial opaque glasses removal model based on a generative adversarial network, wherein the initial opaque glasses removal model includes a glasses encoder, a face encoder, a decoder, and a discriminator; Constructing a target loss function of the initial opaque glasses removal model based on multiple loss functions, wherein the multiple loss functions at least include a face image synthesis loss function, a glasses area loss function, a facial area loss function, and an adversarial loss function; Acquire multiple face images without glasses to form a training set of face images without glasses; The initial opaque glasses removal model is trained using the training set of face images without glasses and the training set of face images with opaque glasses to obtain an opaque glasses removal model, wherein the training includes optimizing the target loss function using the training set of face images without glasses and the training set of face images with opaque glasses.

5. The glasses identification and glasses removal method according to claim 1, characterized in that: Build a frameless glasses removal model based on a generative adversarial network, including: Constructing an initial rimless glasses removal model based on a generative adversarial network, wherein the initial rimless glasses removal model includes a first generator, a discriminator, and a verifier, wherein the first generator includes at least a rimless glasses decoder; Constructing a target loss function of the initial frameless glasses removal model based on multiple loss functions, wherein the multiple loss functions at least include an identity consistency loss function, a pixel-level loss function, a glasses mask loss function, a glasses shadow mask loss function, and an adversarial loss function; The initial rimless glasses removal model is trained using the rimless glasses face image training set to obtain a rimless glasses removal model, wherein the training includes optimizing the target loss function using the rimless glasses face image training set.

6. The glasses identification and glasses removal method according to claim 1, characterized in that: A half-frame glasses removal model is built based on a generative adversarial network, including: Constructing an initial half-frame glasses removal model based on a generative adversarial network, wherein the initial half-frame glasses removal model includes a second generator, a discriminator, and a verifier, wherein the second generator includes at least a half-frame glasses decoder; Constructing a target loss function of the initial half-frame glasses removal model based on multiple loss functions, wherein the multiple loss functions at least include an identity consistency loss function, a pixel-level loss function, a glasses mask loss function, a glasses shadow mask loss function, and an adversarial loss function; The initial half-frame glasses removal model is trained using the half-frame glasses face image training set to obtain a half-frame glasses removal model, wherein the training includes optimizing the target loss function using the half-frame glasses face image training set.

7. The glasses recognition and glasses removal method according to claim 1, characterized in that: Build a full-frame glasses removal model based on a generative adversarial network, including: Constructing an initial full-frame glasses removal model based on a generative adversarial network, wherein the initial full-frame glasses removal model includes a third generator, a discriminator, and a verifier, wherein the third generator includes at least a full-frame glasses decoder; Constructing a target loss function of the initial full-frame glasses removal model based on multiple loss functions, wherein the multiple loss functions at least include an identity consistency loss function, a pixel-level loss function, a glasses mask loss function, a glasses shadow mask loss function, and an adversarial loss function; The initial full-frame glasses removal model is trained using the full-frame glasses face image training set to obtain a full-frame glasses removal model, wherein the training includes optimizing the target loss function using the full-frame glasses face image training set.

8. The glasses identification and glasses removal method according to claim 1, characterized in that: The opaque glasses face image set, frameless glasses face image set, half-frame glasses face image set and full-frame glasses face image set are respectively input into corresponding glasses removal models to remove glasses, and the corresponding glasses-free face image set is obtained, including: Remove glasses from the opaque glasses face image set according to the opaque glasses removal model to obtain a first non-glasses face image set; Performing glasses removal on the frameless glasses face image set according to the frameless glasses removal model to obtain a second frameless glasses face image set; Performing glasses removal on the half-frame glasses face image set according to the half-frame glasses removal model to obtain a third non-glasses face image set; The glasses are removed from the full-frame glasses face image set according to the full-frame glasses removal model to obtain a fourth non-glasses face image set.

9. A glasses recognition and glasses removal system based on deep learning, characterized in that: include: A glasses identification module, used to input a face image set into a glasses dictionary for identification, and obtain a face image set of people wearing glasses and a face image set of people not wearing glasses, wherein the glasses dictionary is a first convolutional neural network used to identify whether there are glasses in a face image; A glasses segmentation module, used for segmenting the glasses-wearing face image set according to a first segmenter to obtain an opaque glasses face image set and a transparent glasses face image set, wherein the first segmenter is a second convolutional neural network trained by the opaque glasses face image training set and the transparent glasses face image training set; a transparent glasses segmentation module, configured to segment the transparent glasses face image set according to a second segmenter to obtain a frameless glasses face image set, a half-frame glasses face image set, and a full-frame glasses face image set, wherein the second segmenter is a third convolutional neural network trained with the frameless glasses face image training set, the half-frame glasses face image training set, and the full-frame glasses face image training set; A glasses removal model construction module, used to respectively construct an opaque glasses removal model, a frameless glasses removal model, a half-frame glasses removal model and a full-frame glasses removal model based on a generative adversarial network, wherein the opaque glasses removal model comprises a glasses encoder, a facial encoder, a decoder and a discriminator, the frameless glasses removal model comprises a first generator, a discriminator and a verifier, the half-frame glasses removal model comprises a second generator, a discriminator and a verifier, the full-frame glasses removal model comprises a third generator, a discriminator and a verifier, the first generator comprises at least a frameless glasses decoder, the second generator comprises at least a half-frame glasses decoder, and the third generator comprises at least a full-frame glasses decoder; The glasses removal module is used to input the opaque glasses face image set, frameless glasses face image set, half-frame glasses face image set and full-frame glasses face image set into the corresponding glasses removal model to remove the glasses and obtain the corresponding glasses-free face image set.

10. A terminal device, characterized in that: The invention comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the glasses recognition and glasses removal method according to any one of claims 1 to 8 is implemented.

Images