Image glasses elimination method, device and equipment based on diffusion model

Abstract

The invention discloses an image glasses elimination method, device and equipment based on a diffusion model, and the method comprises the steps: inputting a to-be-eliminated glasses wearing image into a glasses frame divider, obtaining a glasses frame mask, and fusing the glasses frame mask with the glasses wearing image, thereby obtaining a smeared image; respectively inputting the smeared image into an image encoder and a context encoder to obtain latent code features and global context features; inputting the latent code features, the global context features, the down-sampled mirror frame mask and the latent noise into a double-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code; and inputting the denoised latent code into an image decoder to obtain a glasses elimination result image. The glasses in the image can be effectively eliminated, the natural and real face image with rich details is restored, and the requirement of a user for a high-quality image is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image glasses elimination method, device and equipment based on a diffusion model. BACKGROUND

[0002] In daily life and work, people often need to wear glasses due to visual needs or dressing needs. However, glasses will block the key area of the face and interfere with the face recognition algorithm. In addition, some certificate submission websites require the submission of a non-glasses portrait, and in some photography, glasses are often removed for aesthetic requirements. Therefore, the method of automatically eliminating glasses is of great significance to improve the visual effect of the face in the image, and is widely used in personal photo editing, social media picture optimization and other occasions requiring clear face images.

[0003] The existing image glasses elimination algorithm such as the PCA-based method can only provide an approximate glasses elimination image through face subspace projection, and the performance is poor. Although the face attribute editing method using the GAN mechanism can disentangle the face image identity and attribute vector to realize face attribute editing, it has instability and may cause loss of identity information and easy generation of artifacts. In addition, the GAN-based inpainting method is prone to generate false content and is not real and clear due to the lack of strong prior knowledge. SUMMARY

[0004] Therefore, the purpose of the present application is to provide an image glasses elimination method, device and equipment based on a diffusion model, aiming to solve the problems of poor glasses elimination effect and unrealistic generated content of the existing image glasses elimination method.

[0005] To achieve the above purpose, the present application provides an image glasses elimination method based on a diffusion model, which is realized based on a pre-trained glasses elimination model. The network structure of the glasses elimination model includes a frame segmenter, an image encoder, a context encoder, a double-branch UNet module and an image decoder. The method comprises the following steps:

[0006] The glasses-wearing image to be eliminated is input into the frame segmenter, and after obtaining the frame mask, the glasses-wearing image is fused to obtain a smearing image;

[0007] The smearing image is input into the image encoder and the context encoder respectively to obtain latent code features and global context features;

[0008] The latent code features, global context features, down-sampled frame mask and latent noise are input into the double-branch UNet module for feature fusion and iterative denoising to generate denoised latent code;

[0009] Input the denoising latent code into an image decoder to obtain a glasses removal result image.

[0010] Preferably, the double-branch UNet module comprises a Unet image feature extractor and a Unet denoiser; the feature fusion and iterative denoising of the latent code feature, the global context feature, the down-sampled frame mask and the potential noise input into the double-branch UNet module to generate a denoising latent code comprises:

[0011] The latent code feature is spliced with the down-sampled frame mask and the potential noise, and the global context feature is input into the Unet image feature extractor to obtain an image feature;

[0012] The image feature is fused into the Unet denoiser in a feature insertion manner through weight control to obtain the denoising latent code.

[0013] Preferably, the training process of the glasses removal model comprises:

[0014] Collecting face images including images with glasses and images without glasses for image construction and preprocessing to obtain a paired data set;

[0015] Inputting the paired data set into the network structure of the glasses removal model and performing network training based on a preset loss function; wherein the preset loss function comprises noise loss, L1 loss and high receptive field perception loss, the noise loss is L2 loss of predicted noise of the UNet denoiser and target noise, and the L1 loss comprises image L1 loss and smearing area L1 loss.

[0016] Preferably, the face images including images with glasses and images without glasses are collected for image construction and preprocessing to obtain a training data set, comprising:

[0017] Randomly extracting frame data of each image with glasses and pasting it into the corresponding image without glasses to generate a glasses pasting image, and using the image without glasses and the glasses pasting image as the first paired data;

[0018] Extracting reflections and shadows in the face image as materials;

[0019] Adjusting the eyeless to the eye state in the face image, and constructing a small degree of eye scaling pair through liquefaction technology to obtain the second paired data;

[0020] Constructing the deformation of the face shape behind the lens through liquefaction technology to obtain the third paired data;

[0021] The first paired data, the material, the second paired data and the third paired data are used as the paired data set;

[0022] The paired data set is preprocessed, including adding lens color disturbance with a random probability by positive film superposition technology, adding reflective and shadow materials within the lens range, smearing the image of wearing glasses by a frame mask, performing random flipping, translation, rotation, scaling geometric transformation and brightness noise processing with a random probability, to obtain the training data set.

[0023] Preferably, the image encoder is constructed using a VAE encoder.

[0024] Preferably, the context encoder sampling is constructed using a Vision Transformers structure.

[0025] Preferably, the image decoder is constructed using a VAE decoder.

[0026] Preferably, the Unet image feature extractor and the Unet denoiser have the same network structure, and the Unet module of Stable Diffusion is used for construction.

[0027] To achieve the above-mentioned purpose, the application provides an image glasses elimination device based on a diffusion model, which is realized based on a pre-trained glasses elimination model, and the network structure of the glasses elimination model includes a frame segmenter, an image encoder, a context encoder, a double-branch UNet module, and an image decoder. The device includes:

[0028] The segmentation unit is used for inputting the wearing glasses image to be eliminated into the frame segmenter, fusing the frame mask obtained after the frame segmenter with the wearing glasses image, and obtaining a smearing image.

[0029] The encoding unit is used for inputting the smearing image into the image encoder and the context encoder respectively, to obtain latent code features and global context features.

[0030] The denoising unit is used for inputting the latent code features, the global context features, the down-sampled frame mask, and the latent noise into the double-branch UNet module for feature fusion and iterative denoising, to generate a denoised latent code.

[0031] The decoding unit is used for inputting the denoised latent code into the image decoder, to obtain a glasses elimination result image.

[0032] To achieve the above-mentioned purpose, the application further provides an image glasses elimination device based on a diffusion model, which includes a processor, a memory, and a computer program stored in the memory. The computer program is executed by the processor to implement the steps of the image glasses elimination method based on the diffusion model as described in the above-mentioned embodiments.

[0033] To achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement steps of the image glasses elimination method based on a diffusion model.

[0034] To achieve the above object, the application further provides a computer program product, comprising computer programs / instructions, which are executed by a processor to implement steps of the image glasses elimination method based on a diffusion model.

[0035] Advantages:

[0036] The above scheme is realized by a pre-trained glasses elimination model, can effectively eliminate the glasses in the face in the image, restore a natural and real face image with rich details, and meet the demand of users for high-quality images. The frame segmenter can accurately segment the frame region, provide accurate region information for subsequent image processing, and improve the accuracy and efficiency of processing. The context encoder is introduced to encode the context features of the non-painted region in the image, thereby better guiding the generation of the content of the painted region and providing rich information for subsequent image reconstruction. The dual-branch UNet module combines latent code features, global context features, down-sampled frame masks and latent noise, performs feature fusion and iterative denoising, and generates denoised latent codes, which can better process the information of the frame region and the non-frame region and improve the quality and reality of image reconstruction.

[0037] The model adopts a dual-branch network architecture, which can better distinguish between painted and non-painted regions and perform better image repair. After the latent code features are spliced with the down-sampled frame mask and the latent noise by the Unet image feature extractor, the global context features are inputted, richer image features can be extracted, which can provide more accurate guidance for subsequent denoising operations. The image features are fused by the Unet denoiser in a feature insertion manner through weight control, which can more accurately remove noise and generate high-quality denoised latent codes, so that the denoising process is more efficient and accurate, and the quality of the final image is improved.

[0038] In addition to the noise loss function of the diffusion model, a global and mask area L1 loss, and a high receptive field perception loss are added to better help the network iterate correctly. In particular, by analyzing the shortcomings of previous removal effects and user requirements, a hybrid dataset is constructed to reduce data acquisition while further enriching the diversity of the dataset to meet the removal requirements. Through the eyewear removal method of the case, the removal effect is more natural and clean, and the character features, especially the eye area, are better preserved; the generation of the occluded eyebrows and hair is also more natural and clear, and the non-painting areas such as reflections and shadows can also be automatically removed cleanly, and the refracted face shape can also be well restored and fused after removing the eyewear.

[0039] The image encoder is constructed using a VAE encoder, which can compress the spatial features of the image into latent space features, reducing the amount of calculation while preserving important information, providing an efficient and accurate feature representation for subsequent image processing. The context encoder is constructed using a Vision Transformers (ViT) structure, which can effectively encode the context information of the non-painting area, and reuse this feature to guide the reconstruction of the painting area in the iterative denoising process, improving the quality and consistency of the image reconstruction. The image decoder is constructed using a VAE decoder, which can decode the latent code features back to the image space features, restoring a clear and natural face image, ensuring the final image quality. The Unet image feature extractor and the Unet denoiser have the same network structure, and the Unet module of Stable Diffusion is used for construction, so that the two modules can share part of the network structure and parameters, improving the training efficiency and performance of the model; at the same time, the weights of the Unet denoiser are frozen to provide strong reconstruction capability, while the weights of the Unet image feature extractor are trainable, which can better reconstruct the painting area, further improving the image processing effect. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0041] Figure 1 A flowchart of an image eyewear removal method based on a diffusion model provided by an embodiment of the present application.

[0042] Figure 2 A schematic diagram of the overall processing of the eyewear removal model provided by an embodiment of the present application.

[0043] Figure 3 Fig. 1 is a structural schematic diagram of an image glasses elimination device based on a diffusion model according to an embodiment of the present application.

[0044] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions of the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0046] The content of the present application will be described in detail below in combination with embodiments.

[0047] Reference Figure 1 Fig. 1 is a structural schematic diagram of an image glasses elimination device based on a diffusion model according to an embodiment of the present application.

[0048] In the present embodiment, the method is implemented based on a pre-trained glasses elimination model, the network structure of the glasses elimination model includes a frame segmentor, an image encoder, a context encoder, a double-branch UNet module and an image decoder, and the method includes:

[0049] S11, inputting a to-be-eliminated glasses-wearing image into the frame segmentor, obtaining a frame mask, and fusing the frame mask with the glasses-wearing image to obtain a smearing image;

[0050] S12, inputting the smearing image into the image encoder and the context encoder respectively to obtain latent code features and global context features;

[0051] S13, inputting the latent code features, the global context features, the down-sampled frame mask and the latent noise into the double-branch UNet module for feature fusion and iterative denoising to generate denoised latent code;

[0052] S14, inputting the denoised latent code into the image decoder to obtain a glasses elimination result image.

[0053] Furthermore, the dual-branch UNet module includes a UNet image feature extractor and a UNet denoiser; in step S13, the process of inputting the latent code features, global context features, downsampled frame mask, and latent noise into the dual-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code includes:

[0054] S13-1, The latent code features are concatenated with the downsampled frame mask and latent noise, and then combined with the global context features and input into the Unet image feature extractor to obtain image features;

[0055] S13-2, The image features are incorporated into the Unet denoiser through feature insertion and weight control to obtain the denoising latent code.

[0056] In this embodiment, the image of people wearing glasses to be removed is input into the glasses removal model, and the result image of glasses removal is output. The network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a Unet image feature extractor, a Unet denoiser, and an image decoder. (Refer to...) Figure 2 As shown, the image of the person wearing glasses to be removed is first processed by a pre-trained frame segmenter to obtain a frame mask. Specifically, the frame segmenter includes some refraction at the frame edges to address this challenging situation. Then, the frame mask is fused with the image of the person wearing glasses to obtain the smeared image. The smeared image is then fed into an image encoder to obtain the corresponding latent features, and simultaneously into a context encoder to obtain a global context representation (global context feature). This efficiently encodes the global information needed for the smeared area, guiding pixel generation in the smeared area and ensuring that the generated content is consistent with the overall portrait features. The latent features are then concatenated with the downsampled frame mask and latent noise (Noisy Latent) to form a single feature, which is then fed into the Unet image feature extractor, focusing on extracting features from the smeared image. Simultaneously, the noise... Latenty is also fed into the Unet denoiser to focus on generation; the Unet image feature extractor is a copy of the Unet denoiser, in which the weights of the Unet denoiser are frozen, providing powerful reconstruction capabilities, while the weights of the Unet image feature extractor are trainable, enabling better reconstruction of smeared areas. The output of the Unet image feature extractor is incorporated into the Unet denoiser through weight control via feature insertion to obtain the denoising latent code; finally, it is fed into the image decoder to output the glasses-removed image.

[0057] Furthermore, the image encoder is constructed using a VAE encoder.

[0058] Further, the context encoder sampling is constructed by using a Vision Transformers structure.

[0059] Further, the context encoder sampling is constructed by using a Vision Transformers structure.

[0060] Further, the Unet image feature extractor has the same network structure as the Unet denoiser, and is constructed by using the Unet module of Stable Diffusion.

[0061] In the embodiment, the mirror frame segmenter is a pre-trained model (such as a model of a Segformer framework), which outputs a mirror frame mask and is fused with a wearing glasses image to obtain a smearing image. The formula can be expressed as:

[0062]

[0063] wherein, X glass represents a wearing glasses image, the smearing image X masked is the output of the mirror frame segmenter fused with the wearing glasses image, and blend represents a fusion process.

[0064] The image encoder compresses image space features into latent space features to reduce the amount of calculation, and is constructed by using a vae encoder. The formula can be expressed as:

[0065] Z glass = vae enc (X masked );

[0066] wherein, Z glass represents the latent code feature corresponding to the smearing image X masked , and vae enc () represents the image encoder.

[0067] The context encoder adopts a vit (Vision Transformers) structure, effectively encodes the context information given by the non-smearing area, and repeatedly uses the global context feature to guide the reconstruction of the smearing area in the iterative denoising. The formula can be expressed as:

[0068]

[0069] wherein, represents the extracted global context feature, and ∈ vitnet represents the context encoder.

[0070] The Unet image feature extractor and the Unet denoiser have the same network structure, which is constructed along with the Unet module of Stable Diffusion, for step-by-step removal of noise in the latent space, through multiple time step iteration denoising operations, thereby realizing efficient image generation. The Unet image feature extractor receives the guidance of the global context features to reconstruct the smearing area, and is fed into the Unet denoiser through the feature insertion mode of the zero convolution layer. Its formula can be expressed as:

[0071]

[0072] wherein, ∈ θ (Z t ,t) i represents the feature of the i-th layer in the Unet denoising network ∈ θ , i ∈ [1, n], n is the number of layers of the unet, represents the feature of the Unet image feature extractor, which is composed of the latent code feature Z t of the t-th step (i.e., the latent noise map corresponding to the t-th step), the latent code feature Z glass corresponding to the smearing image, the down-sampled mirror frame segmentation map mask down and the global context feature , and σ represents the zero convolution operation, and ω represents the fusion weight. The double-branch structure can better distinguish between smearing and non-smearing, and achieve better repair effect. When iterating to the 0th step, the denoised latent code feature is output:

[0073] Z t-1 =Z t -∈ θ (Z t ,t);

[0074] wherein, ∈ θ (·) represents the noise predicted by the UNet denoiser, Z t is the latent code at the current step t, Z t-1 is the latent code feature of the previous step before the current t, and Z0 is the denoised latent code output after the t-step iteration is completed.

[0075] The image decoder decodes the latent code feature back to the image space feature, which is constructed by the vae decoder. Its formula can be expressed as:

[0076]

[0077] wherein, represents the final glasses elimination result map, and vae dec represents the image decoder.

[0078] Further, the training process of the glasses removal model comprises:

[0079] The image construction and preprocessing of the collected images including wearing glasses images and non-wearing glasses images are performed to obtain a paired dataset;

[0080] The paired dataset is input into the network structure of the glasses removal model and network training is performed based on a preset loss function; wherein the preset loss function comprises a noise loss, an L1 loss and a high receptive field perception loss, the noise loss is an L2 loss of the predicted noise of the UNet denoiser and the target noise, and the L1 loss comprises an image L1 loss and a smearing area L1 loss.

[0081] The image L1 loss is an L1 loss of the elimination target image and the glasses removal result image, and the smearing area L1 loss is an L1 loss of the elimination target image mask and the glasses removal result image mask.

[0082] Further, the image construction and preprocessing of the collected images including wearing glasses images and non-wearing glasses images are performed to obtain a training dataset, comprising:

[0083] The frame data of each wearing glasses image is randomly extracted and pasted into the corresponding non-wearing glasses image to generate a glasses pasting image, and the non-wearing glasses image and the glasses pasting image are used as the first paired data;

[0084] The reflections and shadows in the face image are extracted as materials;

[0085] The eyeless state in the face image is adjusted to the eye state, and a small degree of eye scaling pairing is constructed through liquefaction technology to obtain the second paired data;

[0086] The deformation of the face shape behind the lens is constructed through liquefaction technology to obtain the third paired data;

[0087] The first paired data, the materials, the second paired data and the third paired data are used as the paired dataset;

[0088] The paired dataset is preprocessed, including adding lens color disturbance with random probability through positive film superimposition technology, adding reflection and shadow materials in the lens range, smearing the wearing glasses image through the frame mask, and performing geometric transformation of random flipping, translation, rotation and scaling and brightness noise processing with random probability to obtain the training dataset.

[0089] In this embodiment, the effect of eliminating glasses not only needs to remove the glasses area and generate natural content, but also needs to remove the reflection, shadow and other sundries caused by glasses, restore the hair, eyebrows, facial contours and other parts blocked by glasses, and remove the original light and shadow structure, skin texture direction after removal, and also need to process the eye light after removing the lens to make the eyes more lively. Therefore, a batch of customized data sets are constructed, and then the model constructed is trained based on the batch of data sets, so as to achieve the purpose of elimination.

[0090] Due to the difficulty and high cost of obtaining paired data, a mixed data method is adopted for the construction of the data set. Specifically: a batch of data of wearing glasses and not wearing glasses is collected respectively and cut and aligned according to the unified standard, and then the data is divided into 20 groups according to the calculation of the facial pose angle. By randomly extracting the frame data of each wearing glasses group and pasting it into the corresponding not wearing glasses group, the original not wearing glasses image and the glasses pasted image form the preliminary paired data. For the task requirements of reflection and shadow, a batch of reflection and shadow in the image are extracted as materials and randomly added in the data preprocessing. For the eye light task requirement, the paired data from no spirit to spirit state is constructed by processing the eye part such as Meitu Xiu Xiu, and the small degree of eye zooming paired is constructed by liquefaction technology to meet the physical deformation of the eyes after removing the glasses. Similarly, for the deformation of the face shape behind the lens, liquefaction technology is also adopted to construct the paired data.

[0091] After obtaining the above paired data set, various processing needs to be added in the training to meet the requirements of elimination task. First, the positive film superimposition technology is used to add lens color disturbance with random probability to meet the elimination task of transparent color glasses, and then the collected reflection and shadow materials are added in the lens range. Then the wearing glasses image is painted by the mirror frame mask, and finally random flip, translation, rotation, scaling and other geometric transformation processing and brightness noise disturbance processing are performed with random probability to form the final network input training data set.

[0092] img i multiply =multiply(img i ori );

[0093] img i sha_ref =shadow_reflection(img i multiply );

[0094] img i blend =blend(img isha_ref , mask i );

[0095] t_img i inp ,t_img i target =T(img i blend ,img i target );

[0096] where img i ori represents the constructed wearing glasses image (i.e., the eyeglasses map image with the frame attached), img i target represents the corresponding non-wearing glasses image, i.e., the elimination target image, img i multiply represents the image after the positive film superimposition disturbance multiply(), img i sha_ref represents the image after the reflection and shadow disturbance shadow_reflection(), img i blend represents the wearing glasses image after the blending operation blend, t_img i target , t_img i inp represents the paired data after the geometric operation, i represents the ith image in the sample set, T represents the geometric operation, and the brightness and noise disturbance set. In particular, only the frame part is painted to retain more information around the eyes and maintain the key features of the person. In addition, a certain probability of loss is also adopted for the frame painted area to deal with the residual frame caused by inaccurate frame segmentation during inference.

[0097] The network training is the same as the training steps of the diffusion model, and the prediction of noise is constrained by a noise loss function. The noise loss can be represented as:

[0098]

[0099] where ∈ represents the target noise (Noisy Latent), which conforms to the Gaussian normal distribution, and ∈ θ (·) represents the noise predicted by the UNet denoiser, and the L2 loss between the two guides the network to update parameters.

[0100] In addition, image L1 loss (i.e., L1 loss between the elimination target image and the eyeglasses elimination result image) and L1 loss of the painted area (i.e., L1 loss between the elimination target image mask and the eyeglasses elimination result image mask) are also added:

[0101]

[0102] where t_img target denotes the non-glasses image after geometric operation, denotes the final glasses removal result image, and mask denotes the frame mask.

[0103] The understanding of the global structure is helpful for the repair task, so a base network using the fast growing receptive field is also added as the high receptive field perception loss. The formula is represented as:

[0104]

[0105] represent the intra-layer mean and inter-layer mean calculation, represent the high receptive field network.

[0106] The final total loss is:

[0107] L total =λ1L noise +λ2L1+λ3L h_lpips ;

[0108] Wherein, λ1, λ2, λ3 are loss weights.

[0109] According to the above description, the preprocessed paired data is input into the network, and the loss is trained according to the above description, the Adam optimizer is used, and the learning rate is set to 0.0002.

[0110] Referring to Figure 3 Fig. 1 shows a structure schematic diagram of an image glasses removal device based on a diffusion model according to an embodiment of the present application.

[0111] In the embodiment, the device is realized based on a pre-trained glasses removal model, and the network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a double-branch UNet module, and an image decoder. The device 20 includes:

[0112] The segmentation unit 21 is configured to input a glasses-wearing image to be removed into the frame segmenter, fuse the frame mask obtained by the frame segmenter with the glasses-wearing image, and obtain a smearing image.

[0113] The encoding unit 22 is configured to input the smearing image into the image encoder and the context encoder respectively, and obtain latent code features and global context features.

[0114] The denoising unit 23 is configured to input the latent code feature, the global context feature, the down-sampled frame mask and the potential noise into a double-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code.

[0115] The decoding unit 24 is configured to input the denoised latent code into an image decoder to obtain a glasses elimination result image.

[0116] The various unit modules of the device 20 can respectively perform the corresponding steps in the above method embodiments, and thus will not be described here in detail. For details, please refer to the description of the corresponding steps above.

[0117] The embodiment of the application also provides an image glasses elimination device based on a diffusion model, which comprises the image glasses elimination device based on a diffusion model as described above. Figure 3 The structure of the embodiment can correspondingly perform the technical solutions of the method embodiments. Figure 1 The technical solutions of the method embodiments are similar in implementation principle and technical effects, and details can be referred to the related description in the above embodiments, which will not be described here in detail.

[0118] The device can include a memory, a processor, an input unit, a display unit, a power supply and the like.

[0119] The memory can be used to store software programs and modules, and the processor can execute various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs (such as image playing function, etc.) required by at least one function, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device or other volatile solid-state memory device. Correspondingly, the memory can also include a memory controller to provide access to the memory for the processor and the input unit.

[0120] The input unit can be used to receive input digital or character or image information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit of the embodiment can include a camera, a touch-sensitive surface (such as a touch display screen) and other input devices in addition to the camera.

[0121] The display unit can be used to display information input by the user or provided to the user and various graphical user interfaces of the device, which can be composed of graphics, text, icons, video and any combination thereof. The display unit can include a display panel, which can be configured in the form of an LCD (Liquid Crystal Display), an OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface can cover the display panel, and when the touch-sensitive surface detects a touch operation thereon or nearby, it transmits to the processor to determine the type of touch event, and then the processor provides corresponding visual output on the display panel according to the type of touch event.

[0122] The embodiment of the present application further provides a computer readable storage medium, which can be the computer readable storage medium contained in the memory in the above-mentioned embodiment, or can exist separately and not be assembled into the device. The computer readable storage medium stores at least one instruction, which is loaded and executed by the processor to realize the method of eliminating image glasses based on a diffusion model shown in the above-mentioned embodiment. Figure 1 The computer readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0123] The embodiment of the present application further provides a computer program product, which includes a computer program / instruction, which is loaded and executed by the processor to realize the method of eliminating image glasses based on a diffusion model shown in the above-mentioned embodiment. Figure 1 The computer program / instruction can be stored in the computer readable storage medium.

[0124] It should be noted that each embodiment in the present specification adopts a progressive description manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts of each embodiment can be referred to each other. For the device embodiment, the equipment embodiment and the storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0125] In addition, in this document, the term "comprising" or "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0126] The foregoing description has shown and described preferred embodiments of the application, but it will be understood that the application is not limited to the particular embodiments disclosed, and it will be understood that various other modifications, applications and embodiments can be made without departing from the spirit and scope of the application, and that the scope of the application will be measured by the claims that follow.

Claims

1. A method for eliminating image glasses based on a diffusion model, characterized in that, The method is implemented based on a pre-trained glasses removal model, a network structure of the glasses removal model includes a frame segmentor, an image encoder, a context encoder, a double-branch UNet module, and an image decoder, and the method includes: inputting a to-be-removed glasses-wearing image into the frame segmentor, fusing the frame mask obtained after the frame segmentor with the glasses-wearing image to obtain a smearing image; inputting the smearing image into the image encoder and the context encoder respectively to obtain latent code features and global context features; inputting the latent code features, the global context features, the down-sampled frame mask, and potential noise into the double-branch UNet module for feature fusion and iterative denoising to generate denoised latent code; inputting the denoised latent code into the image decoder to obtain a glasses removal result image.

2. The image glasses elimination method based on diffusion model according to claim 1, wherein, The double-branch UNet module includes a UNet image feature extractor and a UNet denoiser; the step of inputting the latent code features, the global context features, the down-sampled frame mask, and the potential noise into the double-branch UNet module for feature fusion and iterative denoising to generate denoised latent code includes: concatenating the latent code features, the down-sampled frame mask, and the potential noise, and inputting the result into the UNet image feature extractor in combination with the global context features to obtain image features; fusing the image features into the UNet denoiser in a feature insertion manner through weight control to obtain the denoised latent code.

3. The image glasses elimination method based on diffusion model according to claim 1, characterized in that, The training process of the glasses removal model includes: collecting images including glasses-wearing images and non-glasses-wearing images for image construction and preprocessing to obtain a paired data set; inputting the paired data set into the network structure of the glasses removal model and performing network training based on a preset loss function; wherein the preset loss function includes noise loss, L1 loss, and high receptive field perception loss, the noise loss is L2 loss of predicted noise of the UNet denoiser and target noise, and the L1 loss includes image L1 loss and smearing area L1 loss.

4. The image glasses removal method based on diffusion model according to claim 3, wherein, The training process of the glasses removal model includes: collecting images including glasses-wearing images and non-glasses-wearing images for image construction and preprocessing to obtain a paired data set; randomly extracting frame data of each glasses-wearing image and pasting it into the corresponding non-glasses-wearing image to generate a glasses pasting image, using the non-glasses-wearing image and the glasses pasting image as first paired data; extracting reflections and shadows in the face image as materials; adjusting the eyeless state to the eye state in the face image, and constructing a small degree of eye scaling pair through liquefaction technology to obtain second paired data; constructing the deformation of the face shape behind the lens through liquefaction technology to obtain third paired data; using the first paired data, the materials, the second paired data, and the third paired data as the paired data set; preprocessing the paired data set, including adding lens color disturbance with a random probability through positive film superimposition technology, adding reflection and shadow materials within the lens range, smearing the glasses-wearing image through the frame mask, and performing geometric transformation such as random flipping, translation, rotation, and scaling and brightness noise processing with a random probability to obtain the training data set.

5. The image glasses removal method based on diffusion model according to claim 1, wherein, The image encoder is constructed by using a VAE encoder.

6. The image glasses removal method based on diffusion model according to claim 1, wherein, The context encoder sampling is constructed by using a Vision Transformers structure.

7. The image glasses removal method based on diffusion model according to claim 1, wherein, The image decoder is constructed by using a VAE decoder.

8. The image glasses removal method based on diffusion model according to claim 2, wherein, The Unet image feature extractor has the same network structure as the Unet denoiser, and is constructed by using the Unet module of Stable Diffusion.

9. A diffusion model based image eyeglasses removal apparatus, characterized by, The device is implemented based on a pre-trained glasses removal model, and the network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a double-branch UNet module, and an image decoder. The device includes: The segmentation unit is configured to input a to-be-removed glasses image into the frame segmenter, obtain a frame mask, and fuse the frame mask with the glasses image to obtain a smearing image. The encoding unit is configured to input the smearing image into the image encoder and the context encoder respectively to obtain latent code features and global context features. The denoising unit is configured to input the latent code features, the global context features, the down-sampled frame mask, and the latent noise into the double-branch UNet module for feature fusion and iterative denoising to generate denoised latent codes.

10. An image eyeglass canceling apparatus based on a diffusion model, characterized by, The decoding unit is configured to input the denoised latent codes into the image decoder to obtain a glasses removal result image. The device includes a processor, a memory, and a computer program stored in the memory, and the computer program is executed by the processor to implement the steps of the image glasses removal method based on the diffusion model according to any one of claims 1 to 8.

Images