An image glasses elimination method, device and equipment based on a diffusion model

Abstract

The application discloses a kind of based on diffusion model's image glasses elimination method, device and equipment, comprising: the input frame separator of wearing glasses graph to be eliminated, after obtaining frame mask, with the wearing glasses graph is fused, and obtains smearing image;The smearing image is respectively input image encoder and context encoder, and obtains latent code feature and global context feature;The latent code feature, global context feature, frame mask after down-sampling and latent noise are input double-branch UNet module and are iteratively denoised, and generate denoising latent code;The denoising latent code is input image decoder, and obtains glasses elimination result graph.Effectively eliminate the glasses in image, restore natural real and detailed face image, meet the demand of high-quality image to user.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image glasses removal method, apparatus, and device based on a diffusion model. Background Technology

[0002] In daily life and work, people often need to wear glasses due to vision requirements or fashion sense. However, glasses can obscure key facial areas, interfering with facial recognition algorithms. Furthermore, some document submission websites require portraits taken without glasses, and in some photographs, glasses are often removed for aesthetic reasons. Therefore, methods for automatically removing glasses are crucial for improving the visual appearance of faces in images and are widely used in personal photo editing, social media image optimization, and other situations requiring clear facial images.

[0003] Existing image glasses removal algorithms, such as PCA-based methods, can only provide approximate glasses-removed images through facial subspace projection, resulting in poor performance. While GAN-based facial attribute editing methods can decipher the identity and attribute vectors of a face image to achieve facial attribute editing, they suffer from instability, potentially leading to the loss of identity information and the generation of artifacts. Furthermore, GAN-based inpainting methods, lacking robust prior knowledge, are prone to generating erroneous and unrealistic content. Summary of the Invention

[0004] In view of this, the purpose of this invention is to propose an image glasses removal method, apparatus, and device based on a diffusion model, which aims to solve the problems of poor glasses removal effect and unrealistic generated content in existing image glasses removal methods.

[0005] To achieve the above objectives, this invention provides an image glasses removal method based on a diffusion model. The method is implemented based on a pre-trained glasses removal model. The network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a dual-branch UNet module, and an image decoder. The method includes:

[0006] The image of the person wearing glasses to be removed is input into the frame segmenter. After obtaining the frame mask, it is fused with the image of the person wearing glasses to obtain the smeared image.

[0007] The smeared image is input into the image encoder and the context encoder respectively to obtain latent code features and global context features;

[0008] The latent code features, global context features, downsampled frame mask, and latent noise are input into the dual-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code.

[0009] The denoising latent code is input into the image decoder to obtain the glasses removal result image.

[0010] Preferably, the dual-branch UNet module includes a UNet image feature extractor and a UNet denoiser; the step of inputting the latent code features, global context features, downsampled frame mask, and latent noise into the dual-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code includes:

[0011] The latent code features are concatenated with the downsampled frame mask and latent noise, and then combined with the global context features and input into the Unet image feature extractor to obtain image features;

[0012] The image features are incorporated into the Unet denoiser using a feature insertion method with weight control to obtain the denoising latent code.

[0013] Preferably, the training process of the glasses elimination model includes:

[0014] We collected images of people wearing glasses and those not wearing glasses, performed image construction and preprocessing, and obtained a paired dataset.

[0015] The paired dataset is input into the network structure of the glasses removal model and the network is trained based on a preset loss function; wherein, the preset loss function includes noise loss, L1 loss and high receptive field perception loss, the noise loss is the L2 loss of the predicted noise of the UNet denoiser and the target noise, and the L1 loss includes image L1 loss and smeared region L1 loss.

[0016] Preferably, the collection includes face images of people wearing glasses and not wearing glasses, which are then used for image construction and preprocessing to obtain a training dataset, including:

[0017] Randomly extract the frame data of each image with glasses and paste it into the corresponding image without glasses to generate a glasses texture image. Use the image without glasses and the glasses texture image as the first pairing data.

[0018] Extract reflections and shadows from facial images as source material;

[0019] Adjust the eyes in a facial image from dull to bright, and use liquefaction technology to construct a small degree of eye scaling pairing to obtain the second pairing data;

[0020] The deformation of the face behind the lens was constructed using liquefaction technology to obtain third-pair data;

[0021] The first pairing data, the material, the second pairing data, and the third pairing data are used as the pairing dataset;

[0022] The paired dataset is preprocessed, including adding lens color perturbation with random probability using positive overlay technology, adding reflective and shadow materials within the lens area, smearing the image of wearing glasses with a frame mask, and performing geometric transformations such as random flipping, translation, rotation, and scaling with random probability, as well as processing brightness noise, to obtain the training dataset.

[0023] Preferably, the image encoder is constructed using a VAE encoder.

[0024] Preferably, the context encoder sampling is constructed using a Vision Transformers structure.

[0025] Preferably, the image decoder is constructed using a VAE decoder.

[0026] Preferably, the network structure of the Unet image feature extractor is the same as that of the Unet denoiser, and it is constructed using the Unet module of Stable Diffusion.

[0027] To achieve the above objectives, the present invention provides an image glasses removal device based on a diffusion model. The device is implemented based on a pre-trained glasses removal model. The network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a dual-branch UNet module, and an image decoder. The device includes:

[0028] The segmentation unit is used to input the image of wearing glasses to be removed into the frame segmenter, obtain a frame mask, and then fuse it with the image of wearing glasses to obtain a smeared image.

[0029] The encoding unit is used to input the smeared image into the image encoder and the context encoder respectively to obtain latent code features and global context features;

[0030] The denoising unit is used to input the latent code features, global context features, downsampled frame mask and latent noise into the dual-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code.

[0031] The decoding unit is used to input the denoising latent code into the image decoder to obtain the glasses removal result image.

[0032] To achieve the above objectives, the present invention also proposes an image glasses removal device based on a diffusion model, comprising a processor, a memory, and a computer program stored in the memory, wherein the computer program is executed by the processor to implement the steps of an image glasses removal method based on a diffusion model as described in the above embodiments.

[0033] To achieve the above objectives, the present invention also proposes a computer-readable storage medium storing a computer program that is executed by a processor to implement the steps of an image glasses removal method based on a diffusion model as described in the above embodiments.

[0034] To achieve the above objectives, the present invention also proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of an image glasses removal method based on a diffusion model as described in the above embodiments.

[0035] Beneficial effects:

[0036] The above solution, implemented through a pre-trained glasses removal model, effectively removes glasses from images, restoring natural, realistic, and detailed facial images to meet users' demands for high-quality images. The frame segmenter accurately segments the frame region, providing precise regional information for subsequent image processing, improving accuracy and efficiency. By introducing a context encoder, the contextual features of non-smeared regions in the image are encoded, better guiding the generation of smeared content and providing rich information for subsequent image reconstruction. The dual-branch UNet module combines latent code features, global contextual features, downsampled frame masks, and latent noise for feature fusion and iterative denoising, generating a denoised latent code. This better handles information from both frame and non-frame regions, improving the quality and realism of image reconstruction.

[0037] The model employs a dual-branch network architecture, which better distinguishes between smeared and unsmeared areas, resulting in improved image inpainting. By concatenating latent code features with the downsampled frame mask and latent noise using the Unet image feature extractor, and combining this with global contextual feature input, richer image features can be extracted, providing more accurate guidance for subsequent denoising operations. The Unet denoiser integrates image features through feature insertion with weighted control, enabling more precise noise removal and generating high-quality denoised latent codes. This makes the denoising process more efficient and accurate, improving the final image quality.

[0038] In addition to the noise loss function of the diffusion model, L1 loss for the global and masked regions, as well as high receptive field perceptual loss, were added to better assist the network in correct iteration. Specifically, by analyzing the shortcomings of previous removal effects and user expectations, a hybrid dataset was constructed, reducing data acquisition barriers while further enriching the dataset's diversity to meet removal requirements. The glasses removal method used in this case achieves cleaner and more natural removal results, better preserving facial features; especially in the eye area, it conforms to physical changes, and the generation of occluded eyebrows and hair is more natural and clear. It can also automatically and cleanly remove non-smeared areas such as reflections and shadows, and the refracted facial shape can be well restored and blended after removing the glasses.

[0039] The image encoder is constructed using a VAE encoder, which compresses the spatial features of the image into latent space features, reducing computational cost while retaining important information, providing efficient and accurate feature representations for subsequent image processing. The context encoder is constructed using a Vision Transformers (ViT) structure, effectively encoding contextual information of non-smeared regions. This feature is reused repeatedly in iterative denoising to guide the reconstruction of smeared regions, improving the quality and consistency of image reconstruction. The image decoder is constructed using a VAE decoder, which decodes latent features back to image spatial features, restoring clear and natural face images and ensuring final image quality. The Unet image feature extractor and Unet denoiser share the same network structure, using the Stable Diffusion Unet module, allowing the two modules to share some network structure and parameters, improving model training efficiency and performance. Simultaneously, the weights of the Unet denoiser are frozen, providing powerful reconstruction capabilities, while the weights of the Unet image feature extractor are trainable, enabling better reconstruction of smeared regions, further improving image processing results. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating an image glasses removal method based on a diffusion model, provided in an embodiment of the present invention.

[0042] Figure 2 This is a schematic diagram of the overall processing of the glasses elimination model provided in an embodiment of the present invention.

[0043] Figure 3 This is a schematic diagram of an image glasses elimination device based on a diffusion model, provided in an embodiment of the present invention.

[0044] The realization of the invention's objective, its functional characteristics, and advantages will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The present invention will be described in detail below with reference to the embodiments.

[0047] Reference Figure 1 The diagram shown is a flowchart of an image glasses removal method based on a diffusion model according to an embodiment of the present invention.

[0048] In this embodiment, the method is based on a pre-trained glasses removal model. The network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a dual-branch UNet module, and an image decoder. The method includes:

[0049] S11, Input the image of wearing glasses to be removed into the frame segmenter, obtain the frame mask, and then fuse it with the image of wearing glasses to obtain the smeared image;

[0050] S12, the smeared image is input into the image encoder and the context encoder respectively to obtain latent code features and global context features;

[0051] S13, the latent code features, global context features, downsampled frame mask and latent noise are input into the dual-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code;

[0052] S14, input the denoising latent code into the image decoder to obtain the glasses removal result image.

[0053] Furthermore, the dual-branch UNet module includes a UNet image feature extractor and a UNet denoiser; in step S13, the process of inputting the latent code features, global context features, downsampled frame mask, and latent noise into the dual-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code includes:

[0054] S13-1, The latent code features are concatenated with the downsampled frame mask and latent noise, and then combined with the global context features and input into the Unet image feature extractor to obtain image features;

[0055] S13-2, The image features are incorporated into the Unet denoiser through feature insertion and weight control to obtain the denoising latent code.

[0056] In this embodiment, the image of people wearing glasses to be removed is input into the glasses removal model, and the result image of glasses removal is output. The network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a Unet image feature extractor, a Unet denoiser, and an image decoder. (Refer to...) Figure 2 As shown, the image of the person wearing glasses to be removed is first processed by a pre-trained frame segmenter to obtain a frame mask. Specifically, the frame segmenter includes some refraction at the frame edges to address this challenging situation. Then, the frame mask is fused with the image of the person wearing glasses to obtain the smeared image. The smeared image is then fed into an image encoder to obtain the corresponding latent features, and simultaneously into a context encoder to obtain a global context representation (global context feature). This efficiently encodes the global information needed for the smeared area, guiding pixel generation in the smeared area and ensuring that the generated content is consistent with the overall portrait features. The latent features are then concatenated with the downsampled frame mask and latent noise (Noisy Latent) to form a single feature, which is then fed into the Unet image feature extractor, focusing on extracting features from the smeared image. Simultaneously, the noise... Latenty is also fed into the Unet denoiser to focus on generation; the Unet image feature extractor is a copy of the Unet denoiser, in which the weights of the Unet denoiser are frozen, providing powerful reconstruction capabilities, while the weights of the Unet image feature extractor are trainable, enabling better reconstruction of smeared areas. The output of the Unet image feature extractor is incorporated into the Unet denoiser through weight control via feature insertion to obtain the denoising latent code; finally, it is fed into the image decoder to output the glasses-removed image.

[0057] Furthermore, the image encoder is constructed using a VAE encoder.

[0058] Furthermore, the context encoder sampling is constructed using a Vision Transformers architecture.

[0059] Furthermore, the context encoder sampling is constructed using a Vision Transformers architecture.

[0060] Furthermore, the network structure of the Unet image feature extractor is the same as that of the Unet denoiser, and it is constructed using the Unet module of Stable Diffusion.

[0061] In this embodiment, the frame segmenter is a pre-trained model (such as a model based on the Segformer framework), which outputs a frame mask and fuses it with the image of someone wearing glasses to obtain a smeared image. Its formula can be expressed as:

[0062]

[0063] Among them, X glass To represent an image of someone wearing glasses, erase the X in the image. masked It is a picture frame divider The output of is the result of fusing the image of the person wearing glasses; blend indicates the fusion process.

[0064] Image encoders compress image spatial features into latent space features to reduce computational cost, and are constructed using a VAE encoder. The formula can be expressed as:

[0065] Z glass =vae enc (X masked );

[0066] Among them, Z glass Indicates smearing image X masked The corresponding latent code feature, vae enc () indicates an image encoder.

[0067] The context encoder employs a Vision Transformers (vit) structure to effectively encode the contextual information provided by the non-smeared regions. During iterative denoising, this global contextual feature is repeatedly used to guide the reconstruction of the smeared regions. Its formula can be expressed as:

[0068]

[0069] in, Represents the extracted global contextual features, ∈ vitnet Represents the context encoder.

[0070] The Unet image feature extractor and Unet denoiser share the same network structure, built using the Unet module from Stable Diffusion. They progressively remove noise from the latent space through iterative denoising operations over multiple time steps, thereby achieving efficient image generation. The Unet image feature extractor receives global context features. The smeared region is reconstructed under the guidance of the algorithm and fed into the UNet denoiser via feature insertion using a zero-convolutional layer. The formula can be expressed as:

[0071]

[0072] Where, ∈ θ (Z t ,t) i Indicates the Unet denoising network ∈ θ The features of the i-th layer, i∈[1,n], where n is the layer number of the unet. The features of the Unet image feature extractor are represented by the latent code features Z at step t. t (i.e., the latent noise map corresponding to step t), the latent code feature Z corresponding to the smeared image. glass , downsampled frame segmentation mask down and global context features The structure consists of two branches, where σ represents zero convolution and ω represents the fusion weights. The dual-branch structure better distinguishes between smeared and unsmeared areas, achieving better restoration results. At iteration 0, the denoised latent code features are output:

[0073] Z t-1 =Z t -∈ θ (Z t ,t);

[0074] Where, ∈ θ (·) represents the noise predicted by the UNet denoiser, Z t Z is the latent code at the current step size t. t-1 The latent code features are the features of the previous step t. After t iterations, the denoised latent code Z0 is finally output.

[0075] The image decoder decodes latent features back into image spatial features, which are then constructed using a VAE decoder. The formula can be expressed as:

[0076]

[0077] in, This image shows the final result of the glasses removal process. dec This indicates an image decoder.

[0078] Furthermore, the training process of the glasses elimination model includes:

[0079] We collected images of people wearing glasses and those not wearing glasses, performed image construction and preprocessing, and obtained a paired dataset.

[0080] The paired dataset is input into the network structure of the glasses removal model and the network is trained based on a preset loss function; wherein, the preset loss function includes noise loss, L1 loss and high receptive field perception loss, the noise loss is the L2 loss of the predicted noise of the UNet denoiser and the target noise, and the L1 loss includes image L1 loss and smeared region L1 loss.

[0081] Wherein, the image L1 loss is the L1 loss between the target image and the glasses removal result image, and the smeared region L1 loss is the L1 loss between the target image mask and the glasses removal result image mask.

[0082] Furthermore, the collected face images, including images of people wearing glasses and images of people not wearing glasses, are used for image construction and preprocessing to obtain a training dataset, including:

[0083] Randomly extract the frame data of each image with glasses and paste it into the corresponding image without glasses to generate a glasses texture image. Use the image without glasses and the glasses texture image as the first pairing data.

[0084] Extract reflections and shadows from facial images as source material;

[0085] Adjust the eyes in a facial image from dull to bright, and use liquefaction technology to construct a small degree of eye scaling pairing to obtain the second pairing data;

[0086] The deformation of the face behind the lens was constructed using liquefaction technology to obtain third-pair data;

[0087] The first pairing data, the material, the second pairing data, and the third pairing data are used as the pairing dataset;

[0088] The paired dataset is preprocessed, including adding lens color perturbation with random probability using positive overlay technology, adding reflective and shadow materials within the lens area, smearing the image of wearing glasses with a frame mask, and performing geometric transformations such as random flipping, translation, rotation, and scaling with random probability, as well as processing brightness noise, to obtain the training dataset.

[0089] In this embodiment, achieving effective glasses removal requires not only generating natural content after removing the glasses area, but also removing reflections, shadows, and other clutter caused by the glasses, restoring hair, eyebrows, facial contours, and other parts obscured by the glasses, without affecting the original light and shadow structure or skin texture. Furthermore, it's necessary to process the catchlight in the eyes after removing the lenses to make the eyes more expressive. Therefore, a customized dataset was constructed, and the model was trained based on this dataset to achieve the removal goal.

[0090] Due to the high difficulty and cost of acquiring paired data, a hybrid data approach was adopted for dataset construction. Specifically, a batch of images with and without glasses were collected and cropped and aligned according to a uniform standard. Then, based on the calculated facial pose angle, the data was divided into 20 groups. The frame data of glasses was randomly extracted from each group of images with glasses and pasted onto the corresponding image in the group without glasses. Thus, the original image without glasses and the image with the glasses frame pasted on formed the initial paired data. For the task requirements regarding reflections and shadows, reflections and shadows were extracted from a batch of images and randomly added during data preprocessing. For the task requirements regarding catchlights, eye processing techniques such as Meitu Xiu Xiu were used to construct paired data ranging from lifeless to expressive. Simultaneously, liquefaction techniques were used to construct paired data with slight eye scaling to accommodate the physical deformation of the eyes after removing glasses. Similarly, liquefaction techniques were also used to construct paired data for facial deformation behind lenses.

[0091] After obtaining the paired dataset described above, various processing steps need to be added during training to meet the requirements of the elimination task. First, a positive overlay technique is used to add lens color perturbations with random probability to meet the task of eliminating transparent glasses. Then, collected reflection and shadow materials are added within the lens area. Next, the image of wearing glasses is smoothed using a frame mask. Finally, geometric transformations such as random flipping, translation, rotation, and scaling, as well as perturbations such as brightness noise, are performed with random probability to form the final network input training dataset.

[0092] img i multiply =multiply(img i ori );

[0093] img i sha_ref = shadow_reflection(img i multiply );

[0094] img i blend =blend(img isha_ref mask i );

[0095] t_img i inp ,t_img i target =T(img i blend ,img i target );

[0096] Among them, img i ori The image represents a constructed image of someone wearing glasses (i.e., a texture image of glasses with frames applied), img i target This represents the corresponding image of someone not wearing glasses, which is the target image to be eliminated. (img) i multiply The image represents the result after multiply() perturbation. i sha_ref The image after `shadow_reflection()` represents the reflection and shadow perturbations. i blend This represents an image of someone wearing glasses after the blending process has been applied. (t_img) i target t_img i inp The data represents the paired data after geometric operations, where i represents the i-th image in the sample set, and T represents the geometric operations, brightness, and noise perturbation set. Specifically, smoothing only the frame area preserves more information around the eyes, maintaining key features of the person. Furthermore, a certain probability of losing the smoothed frame area is applied to address frame remnants caused by inaccurate frame segmentation during inference.

[0097] The network training process is the same as that for diffusion models, using a noise loss function to constrain noise prediction. The noise loss can be expressed as:

[0098]

[0099] Where ∈ represents the target noise (latent noise), which follows a Gaussian normal distribution. θ (·) represents the noise predicted by the UNet denoiser, and the L2 loss of both guides the network to update its parameters.

[0100] Additionally, image L1 loss (i.e., L1 loss between the target image and the glasses removal result image) and L1 loss of the smeared region (i.e., L1 loss between the target image mask and the glasses removal result image mask) were added:

[0101]

[0102] In the formula, t_img target This represents the image without glasses after geometric operations. This image shows the final result of the glasses removal process; mask represents the eyeglass frame mask.

[0103] Understanding the global structure is helpful for the repair task; therefore, a base network with a rapidly growing receptive field is incorporated as a high receptive field perceptual loss. The formula is expressed as:

[0104]

[0105] This represents the calculation of intra-layer and inter-layer means. This represents a high-sensory-field network.

[0106] The final total loss is:

[0107] L total =λ1L noise +λ2L1+λ3L h_lpips ;

[0108] Where λ1, λ2, and λ3 are the loss weights.

[0109] Based on the above description, the preprocessed paired data is fed into the network, trained using the loss function described above, and the Adam optimizer is used with a learning rate of 0.0002.

[0110] Reference Figure 3 The diagram shown is a schematic representation of an image glasses removal device based on a diffusion model according to an embodiment of the present invention.

[0111] In this embodiment, the device is implemented based on a pre-trained glasses removal model. The network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a dual-branch UNet module, and an image decoder. The device 20 includes:

[0112] The segmentation unit 21 is used to input the image of wearing glasses to be removed into the frame segmenter, obtain a frame mask, and then fuse it with the image of wearing glasses to obtain a smeared image;

[0113] Encoding unit 22 is used to input the smeared image into the image encoder and the context encoder respectively to obtain latent code features and global context features;

[0114] The denoising unit 23 is used to perform feature fusion and iterative denoising on the latent code features, global context features, downsampled frame mask and latent noise input into the dual-branch UNet module to generate a denoised latent code;

[0115] Decoding unit 24 is used to input the denoising latent code into the image decoder to obtain the glasses removal result image.

[0116] Each unit module of the device 20 can execute the corresponding steps in the above method embodiment, so the details of each unit module will not be elaborated here. Please refer to the description of the corresponding steps above for details.

[0117] This invention also provides an image glasses removal device based on a diffusion model, which includes the image glasses removal apparatus based on a diffusion model as described above, wherein the image glasses removal apparatus based on a diffusion model can employ... Figure 3 The structure of the embodiment, correspondingly, can be executed Figure 1 The technical solutions of the method embodiments shown are similar in implementation principle and technical effect. For details, please refer to the relevant records in the above embodiments, which will not be repeated here.

[0118] The device includes: a mobile phone, digital camera, or tablet computer, or other device with a camera function; or a device with an image processing function; or a device with an image display function. The device may include components such as a memory, processor, input unit, display unit, and power supply.

[0119] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as image playback function), etc.; the data storage area can store data created according to the use of the device. In addition, the memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory can also include a memory controller to provide access to the memory for the processor and input units.

[0120] The input unit can be used to receive input numerical, character, or image information, and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control. Specifically, in addition to a camera, the input unit of this embodiment may also include a touch-sensitive surface (e.g., a touch screen) and other input devices.

[0121] The display unit can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. The display unit may include a display panel, optionally configured as an LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), or other similar display panel. Furthermore, a touch-sensitive surface may cover the display panel. When the touch-sensitive surface detects a touch operation on or near it, it transmits the information to the processor to determine the type of touch event. Subsequently, the processor provides corresponding visual output on the display panel based on the type of touch event.

[0122] This invention also provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory described in the above embodiments; or it may be a standalone computer-readable storage medium not assembled into a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement... Figure 1 The illustrated image glasses removal method is based on a diffusion model. The computer-readable storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0123] This invention also provides a computer program product, including a computer program / instructions, which are loaded and executed by a processor to achieve [the desired result]. Figure 1 This illustrates an image glasses removal method based on a diffusion model.

[0124] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the device embodiments, equipment embodiments, and storage medium embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions in the method embodiments.

[0125] Furthermore, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0126] The foregoing description illustrates and describes preferred embodiments of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept by means of the foregoing teachings or techniques or knowledge in related fields. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. An image glasses removal method based on a diffusion model, characterized in that, The method is based on a pre-trained glasses removal model. The network structure of the glasses removal model includes a frame segmenter, an image encoder, a context encoder, a dual-branch UNet module, and an image decoder. The method includes: The image of the person wearing glasses to be removed is input into the frame segmenter. After obtaining the frame mask, it is fused with the image of the person wearing glasses to obtain the smeared image. The smeared image is input into the image encoder and the context encoder respectively to obtain latent code features and global context features; The latent code features, global context features, downsampled lens mask, and latent noise are input into a dual-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code. The dual-branch UNet module includes a UNet image feature extractor and a UNet denoiser. The process of inputting the latent code features, global context features, downsampled lens mask, and latent noise into the dual-branch UNet module for feature fusion and iterative denoising to generate the denoised latent code includes: The latent code features are concatenated with the downsampled frame mask and latent noise, and then combined with the global context features and input into the Unet image feature extractor to obtain image features; The image features are incorporated into the Unet denoiser using a feature insertion method with weight control to obtain the denoising latent code; The denoising latent code is input into the image decoder to obtain the glasses removal result image; The training process of the glasses elimination model includes: We collected images of people wearing glasses and people not wearing glasses, performed image construction and preprocessing, and obtained a training dataset. The training dataset is input into the network structure of the glasses removal model and the network is trained based on a preset loss function; wherein, the preset loss function includes noise loss, L1 loss and high receptive field perception loss, the noise loss is the L2 loss of the predicted noise of the UNet denoiser and the target noise, and the L1 loss includes the image L1 loss and the smeared region L1 loss. The collection includes image construction and preprocessing of images with and without glasses to obtain a training dataset, including: Randomly extract the frame data of each image with glasses and paste it into the corresponding image without glasses to generate a glasses texture image. Use the image without glasses and the glasses texture image as the first pairing data. Extract reflections and shadows from facial images as source material; Adjust the eyes in a facial image from dull to bright, and use liquefaction technology to construct a small degree of eye scaling pairing to obtain the second pairing data; The deformation of the face behind the lens was constructed using liquefaction technology to obtain third-pair data; The first pairing data, the material, the second pairing data, and the third pairing data are used as the pairing dataset; The paired dataset is preprocessed, including adding lens color perturbation with random probability using positive overlay technology, adding reflective and shadow materials within the lens area, smearing the image of wearing glasses with a frame mask, and performing geometric transformations such as random flipping, translation, rotation, and scaling with random probability, as well as processing brightness noise, to obtain the training dataset.

2. The image glasses removal method based on a diffusion model according to claim 1, characterized in that, The image encoder is constructed using a VAE encoder.

3. The image glasses removal method based on a diffusion model according to claim 1, characterized in that, The context encoder sampling is constructed using the Vision Transformers architecture.

4. The image glasses removal method based on a diffusion model according to claim 1, characterized in that, The image decoder is constructed using a VAE decoder.

5. The image glasses removal method based on a diffusion model according to claim 1, characterized in that, The Unet image feature extractor has the same network structure as the Unet denoiser, and is constructed using the Stable Diffusion Unet module.

6. An image glasses removal device based on a diffusion model, characterized in that, The image glasses removal method based on a diffusion model according to any one of claims 1-5, wherein the device is implemented based on a pre-trained glasses removal model, the network structure of which includes a frame segmenter, an image encoder, a context encoder, a dual-branch UNet module, and an image decoder, the device comprising: The segmentation unit is used to input the image of wearing glasses to be removed into the frame segmenter, obtain a frame mask, and then fuse it with the image of wearing glasses to obtain a smeared image. The encoding unit is used to input the smeared image into the image encoder and the context encoder respectively to obtain latent code features and global context features; The denoising unit is used to input the latent code features, global context features, downsampled frame mask and latent noise into the dual-branch UNet module for feature fusion and iterative denoising to generate a denoised latent code. The decoding unit is used to input the denoising latent code into the image decoder to obtain the glasses removal result image.

7. An image glasses removal device based on a diffusion model, characterized in that, The device includes a processor, a memory, and a computer program stored in the memory, the computer program being executed by the processor to implement the steps of a diffusion-based image glasses removal method as described in any one of claims 1 to 5.

Images