Image light spot removing method

Abstract

The invention provides an image light spot removal method, which comprises the following steps of: constructing a visual generation model VQ-GAN, and training the VQ-GAN; performing local de-noising processing on the image of which the light spots are to be removed; extracting global structure information of the image of which the light spots are to be removed by using the trained visual generative model VQ-GAN; and carrying out weighted fusion processing on the image after local denoising processing and the image reconstructed based on the global structure information to obtain an image without light spots. According to the method, local filtering and global reconstruction are combined, so that local details can be effectively reserved, and high-quality recovery of the overall structure of the image can be realized; while light spots and noise are removed, the problem of detail blurring or overall distortion possibly caused by pure local or global processing is effectively avoided. According to the method, a learnable mapping layer is introduced in the encoding and decoding processes, so that the repairing effect of a highlight spot area is remarkably improved; by setting an adjustable fusion weight, the proportion of a local denoising result and a global reconstruction result in a final output image is flexibly controlled.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for removing light spots from images of highly reflective material surfaces. Background Technology

[0002] In the fields of industrial inspection and image processing, especially in the imaging of metal surfaces, the problem of light spots (or light reflection) has always been a pressing issue. Due to the special characteristics of metal surfaces and lighting conditions, local overexposure or high-brightness areas are prone to occur. This not only affects the overall image quality but also leads to the loss of detailed information, thus posing difficulties for subsequent tasks such as automated inspection and defect identification.

[0003] Traditional methods primarily rely on image preprocessing techniques, such as threshold-based segmentation, morphological operations, and filtering. Thresholding segmentation and filtering involve segmenting the image using a fixed or adaptive threshold to detect overly bright areas, followed by smoothing the bright spots using median filtering or bilateral filtering. This method works well for simple bright spots, but it is insufficient for complex backgrounds and detail preservation, easily leading to blurred edges.

[0004] Morphological processing involves extracting the spot region through morphological operations such as dilation and erosion, and then filling it using image inpainting or interpolation algorithms. This method can restore local structure to some extent, but it has high requirements for spot morphology and brightness distribution, and it is difficult to adapt to varying spot conditions.

[0005] Secondly, existing technologies also remove image glare using image inpainting techniques. Image inpainting techniques use PDE diffusion methods or texture synthesis to blend the glare area with the surrounding background. The PDE diffusion method uses partial differential equations to diffuse the image, automatically filling in the pixel information of the glare area. Although this method can restore local continuity well, it often fails to retain detailed textures in highlight areas and involves a large computational load. Texture synthesis methods extract textures from glare-free areas and then synthesize them into the glare areas to remove glare. This method is more effective when dealing with backgrounds with relatively simple structures, but its effect on restoring complex textures is limited.

[0006] Furthermore, due to their excellent performance in image inpainting and enhancement tasks, deep convolutional neural networks (CNNs) and generative adversarial networks (GANs) have been applied to glare removal. CNNs, employing structures such as U-NET or ResNet, learn the mapping from glare-prone images to glare-free images using a large amount of labeled data, thus achieving automated glare detection and removal. This method can effectively suppress glare while preserving image details, but it is highly dependent on training data, and its robustness across different scenes needs improvement. GANs, on the other hand, achieve high-quality image inpainting through adversarial training between a generator and a discriminator. The generator produces the inpainting result based on the input image, while the discriminator judges whether the inpainting effect is natural. GAN methods have advantages in restoring image details and textures, but their training process is complex, and model stability and convergence still face challenges.

[0007] Although the above methods can solve the spot problem to some extent, they generally have the following shortcomings:

[0008] (1) Insufficient detail recovery: Traditional image processing methods and partial restoration techniques are prone to causing edge blurring or loss of local details, affecting the overall image quality.

[0009] (2) Poor adaptability: Existing deep learning methods perform differently under different spot shapes, sizes and brightness distributions, and often require a large amount of labeled data for training, making them difficult to apply in real-time or resource-constrained environments.

[0010] (3) High computational complexity: In particular, deep learning-based image restoration methods have high hardware requirements during the training process, and there are difficulties in deployment in practical applications.

[0011] In summary, while current spot removal techniques each have their advantages and disadvantages, they still fall short of achieving ideal results when dealing with complex spot problems such as those on metal surfaces. Therefore, there is an urgent need for a method that can efficiently and accurately detect and remove spots while preserving image details and textures and reducing computational complexity. Summary of the Invention

[0012] To address the shortcomings of existing technologies, this invention provides an image spot removal method. This invention achieves high-quality restoration of image details and overall structure by combining local and global processing strategies.

[0013] The technical solution of the present invention is: an image spot removal method, comprising the following steps:

[0014] S1) Construct the visual generative model VQ-GAN and train it;

[0015] S2) Perform local denoising processing on the image to be removed to obtain a locally denoised image;

[0016] S3) Use the trained visual generative model VQ-GAN to extract the global structural information of the image to be removed, and obtain the image reconstructed based on the global structural information;

[0017] S4) The image after local denoising and the image reconstructed based on global structural information are weighted and fused to obtain an image with the light spots removed.

[0018] Preferably, in step S1), the visual generative model VQ-GAN includes an encoder and a decoder, as well as a learnable encoder weighted mapping layer MapEnc and a decoder weighted mapping layer MapDec; the encoder receives the input image of the light spots to be removed. It generates a preliminary latent variable representation; the decoder is used to reconstruct the discretized latent variables back to the original graph domain; the encoder weighted mapping layer MapEnc and the decoder weighted mapping layer MapDec perform shape mapping and weighting operations on the latent variables at both ends of the encoder and decoder. By adding trainable weights at these two ends, the latent variables can better retain the overall structural information while undergoing shape transformation.

[0019] Preferably, in step S1), the visual generation model VQ-GAN is represented as:

[0020] VQ{Enc,Codebook,MapEnc,MapDec,Dec};

[0021] In the formula, Enc represents the encoder; Codebook is the code that quantizes continuous latent variables into discrete codes; Dec is the decoder; MapEnc is the encoder weighted mapping layer; and MapDec is the decoder weighted mapping layer.

[0022] Preferably, in step S2), local denoising processing is performed on the image to be removed from the light spots, specifically including the following steps:

[0023] S21) The image of the light spot to be removed Divide into N local patches; that is:

[0024]

[0025] In the formula P i This represents the i-th image patch; H, W, and C represent the length, width, and channel dimensions, respectively.

[0026] S22), for each image block P i Local denoising is performed using bilateral filtering, i.e.:

[0027]

[0028] In the formula, flocal is the local denoising operator; This represents the i-th image patch after filtering at the local (x,y) location; flocal(P i (x,y) represents performing bilateral filtering at position (x,y) of the i-th image patch; W p Ω(x,y) represents the normalization factor, ensuring the sum of the weights is 1; Ω(x,y) represents the neighborhood window centered at (x,y); (x′,y′) represents the local position of the neighborhood window; P i (x′,y′) represents the i-th image patch centered at the local location (x′,y′);

[0029] S23) All the denoised image blocks are stitched together into a locally denoised image I through a recombination operation. local ,Right now:

[0030]

[0031] In the formula, This is a restructuring operation.

[0032] Preferably, in step S22), the normalization factor W p w(x,y,x′,y′) are respectively represented as:

[0033]

[0034] ΔI(x,y,x′,y′)=∣P i (x,y)-P i (x′,y′)∣;

[0035] In the formula, Let be a spatial Gaussian function, and let its standard deviation σ be... s Control the influence range of neighboring spaces; A Gaussian function with a range and a standard deviation σ r The influence of pixel value similarity is controlled; d(x,x′,y,y′) represents the distance between positions (x,y) and (x′,y′); ΔI(x,y,x′,y′) represents the absolute value of the difference between the patch variables.

[0036] Preferably, in step S3), the global structural information of the image to be freed of light spots is extracted using the trained visual generation model VQ-GAN, specifically including the following steps:

[0037] S31) Given the input image First, the latent variables are obtained through the encoder (Encoder) and the encoder weighted mapping layer (MapEnc), namely:

[0038]

[0039] In the formula, Z′ is the latent variable; Z is the shape mapping of the latent variable; Enc represents the encoder;

[0040] S32) The shape mapping Z of the latent variable is quantized into a discrete code space through a discretization operation, that is:

[0041]

[0042] In the formula, Discrete code; Codebook represents the discretization operation; A set representing discrete codes;

[0043] S33) Using a decoder to convert discrete code The shape is mapped back to the same shape as the latent variable Z, and then the reconstructed image is obtained through the decoder weighted mapping layer MapDec and the decoder Decoder.

[0044]

[0045] I global =Dec(Z) ′* );

[0046] In the formula, I global This represents an image reconstructed based on global structural information; Dec represents the decoder; the decoder weighted mapping layer MapDec quantizes the image Z with shape H′×W′×1. * Map back to the representation Z′ of shape H′×W′×D * And in this process, a more complete global information is recovered through trainable weighted operations, where H′×W′×D is Z′. * The length, width, and passageway.

[0047] Preferably, in step S4), a fusion weight α∈[0,1] is introduced to perform a weighted fusion of the local and global denoising results, that is:

[0048]

[0049] In the formula, I local The image after local denoising; I global An image reconstructed based on global structural information; This is the image after weighted fusion.

[0050] Preferably, the total loss function L of the visual generative model VQ-GAN is expressed as:

[0051] L=λ rec Lrec +λ local L local +λ global L global +λ VQ L VQ ;

[0052] In the formula, λ rec , λ local , λ global , λ VQ L represents the weight hyperparameters of the corresponding loss function; rec For reconstruction losses; L local For local consistency loss; L global To preserve the loss for the global structure; L VQ This refers to the loss associated with VQGAN.

[0053] The beneficial effects of this invention are as follows:

[0054] 1. This invention combines local filtering with a global reconstruction model, which can effectively preserve local details and achieve high-quality restoration of the overall image structure. This dual processing mechanism removes light spots and noise while effectively avoiding the problems of detail blurring or overall distortion that may be caused by simple local or global processing.

[0055] 2. This invention introduces a learnable mapping layer during the encoding and decoding process, which enhances the preservation of global structural features through additional trainable parameters, and locks some weights during the inference stage to prevent overfitting or structural loss, thereby significantly improving the restoration effect of high-brightness spot areas.

[0056] 3. This invention flexibly controls the local denoising result I by setting an adjustable fusion weight α. local Compared with global reconstruction results I global The proportion in the final output image; this strategy not only enhances the adaptability of the invention, but also enables different needs for local detail and overall structure restoration to be met in different scenarios;

[0057] 4. This invention employs a multi-constraint design of overall reconstruction loss, local consistency loss, global structure preservation loss, and VQGAN-related loss to ensure that the model achieves the best balance in terms of detail preservation, structure restoration, and image realism. Attached Figure Description

[0058] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the framework of the method of the present invention. Detailed Implementation

[0059] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0060] like Figure 1 As shown, this embodiment provides a method for removing image smudges, including the following steps:

[0061] S1) Construct the visual generative model VQ-GAN and train it;

[0062] The visual generative model VQ-GAN includes an encoder and a decoder, as well as a learnable encoder weighted mapping layer MapEnc and a decoder weighted mapping layer MapDec; the encoder receives the input image of the light spots to be removed. It generates a preliminary latent variable representation; the decoder is used to reconstruct the discretized latent variables back to the original graph domain; the encoder weighted mapping layer MapEnc and the decoder weighted mapping layer MapDec perform shape mapping and weighting operations on the latent variables at both ends of the encoder and decoder. By adding trainable weights at these two ends, the latent variables can better retain the overall structural information while undergoing shape transformation.

[0063] In this embodiment, the visual generative model VQ-GAN is represented as:

[0064] VQ{Enc,Codebook,MapEnc,Mapdec,Dec};

[0065] In the formula, Enc represents the encoder; Codebook is the code that quantizes continuous latent variables into discrete codes; Dec is the decoder; MapEnc is the encoder weighted mapping layer; and MapDec is the decoder weighted mapping layer.

[0066] In this embodiment, the training process of the visual generative model VQ-GAN is as follows:

[0067] S2) Perform local denoising on the image to be removed from the light spots; specifically including the following steps:

[0068] S21) The image of the light spot to be removed Divide into N local patches; that is:

[0069]

[0070] In the formula P i H represents the i-th image patch; H, W, and C represent the length, width, and channel dimensions of image I, respectively; h×w×C is the P-value. i The three dimensions of length, width, and channel;

[0071] S22), for each image block P i Local denoising is performed using bilateral filtering, i.e.:

[0072]

[0073] In the formula, flocal is the local denoising operator; This represents the i-th image patch after filtering at the local location (x, y); flocal(P i (x,y) represents performing bilateral filtering at position (x,y) of the i-th image patch; W p Ω(x,y) represents the normalization factor, ensuring the sum of the weights is 1; Ω(x,y) represents the neighborhood window centered at (x,y); (x′,y′) represents the local position of the neighborhood window; P i (x′,y′) represents the i-th image patch centered at the local location (x′,y′);

[0074] The normalization factor W o w(x,y,x′,y′) are respectively represented as:

[0075]

[0076] ΔI(x,y,x′,y′)=∣P i (x,y)-P i (x′,y′)∣;

[0077] In the formula, Let be a spatial Gaussian function, and let its standard deviation σ be... s Control the influence range of neighboring spaces; A Gaussian function with a range and a standard deviation σ r The influence of pixel value similarity is controlled; d(x,x′,y,y′) represents the distance between positions (x,y) and (x′,y′); ΔI(x,y,x′,y′) represents the absolute value of the difference between the patch variables.

[0078] S23) All the denoised image blocks are stitched together into a locally denoised image I through a recombination operation. local ,Right now:

[0079]

[0080] In the formula, This is a restructuring operation.

[0081] S3) Extract global structural information of the image to be cleaned using the trained visual generative model VQ-GAN; specifically including the following steps:

[0082] S31) Given the input image First, the latent variables are obtained through the encoder (Encoder) and the encoder weighted mapping layer (MapEnc), namely:

[0083]

[0084] In the formula, Z′ is the latent variable; Z is the shape mapping of the latent variable; Enc represents the encoder;

[0085] S32) The shape mapping Z of the latent variable is quantized into a discrete code space through a discretization operation, that is:

[0086]

[0087] In the formula, Discrete code; Codebook represents the discretization operation; A set representing discrete codes;

[0088] S33) Using a decoder to convert discrete code The shape is mapped back to the same shape as the latent variable Z, and then the reconstructed image is obtained through the decoder weighted mapping layer MapDec and the decoder Decoder.

[0089]

[0090] I global =Dec(Z′) * );

[0091] In the formula, I global This represents an image reconstructed based on global structural information; Dec represents the decoder; the decoder weighted mapping layer MapDec quantizes the image Z with shape H′×W′×1. * Map back to the quantized representation Z′ of shape H′×W′×D * And in the process, more complete global information can be recovered through trainable weighted operations.

[0092] S4) The image after local denoising and the image reconstructed based on global structural information are weighted and fused to obtain an image with the light spots removed;

[0093] This embodiment introduces a fusion weight α∈[0,1] to perform weighted fusion of the local and global denoising results, that is:

[0094]

[0095] In the formula, I local The image after local denoising; I global An image reconstructed based on global structural information; This is the image after weighted fusion.

[0096] The fusion weight α is a number between 0 and 1 used to control the local denoising of the image I. local and the image I reconstructed based on global structural information global In the final output image The contribution ratio in the process; the choice of fusion weight α depends on the specific application scenario and requirements:

[0097] When local details of an image are particularly important, a higher fusion weight α can be selected to retain more local information; conversely, if it is necessary to emphasize the overall structural coherence of the image, a lower fusion weight α should be selected to enhance the impact of the global processing result.

[0098] In this preferred embodiment, a set of multiple loss functions is designed to train the visual generative model VQ-GAN, ensuring that the model achieves the best balance between local detail preservation, global structure restoration, and overall visual quality. The total loss function L of the visual generative model VQ-GAN is expressed as:

[0099] L=λ rec L rec +λ local L local +λ global L global +λ VQ L VQ ;

[0100] In the formula, λ rec , λ local , λ global , λ VQ L represents the weight hyperparameters of the corresponding loss function; rec For reconstruction losses; L local For local consistency loss; L global To preserve the loss for the global structure; L VQ This refers to the loss associated with VQGAN.

[0101] This embodiment aims to ensure the final fused image. The image I containing the light spots to be removed maintains high pixel-level consistency with the input image I; the reconstruction loss L... rec express;

[0102]

[0103] In the formula, I is the image of the light spot to be removed; This is the image after weighted fusion.

[0104] In this embodiment, to preserve the detailed information of each local region, the local consistency loss L... local For each local image patch P iThe result after bilateral filtering Constraints are applied between them, expressed as follows:

[0105]

[0106] In the formula, P i Represents the i-th image patch; This represents the i-th image block after bilateral filtering.

[0107] In this embodiment, the global branch reconstructs image I using the visual generative model VQ-GAN and a mapping layer. global To better recover global structural information, a pre-trained feature extraction network φ(·) is used to extract high-level semantic features, and a global structure preservation loss L is designed. global for:

[0108]

[0109] In the formula, I is the image of the light spot to be removed.

[0110] Preferably, the VQGAN related loss L VQ Including quantization loss L quant and combat losses L adv ;Right now:

[0111] L VQ =L quant +L adv ;

[0112] Among them, the quantization loss L quant Used to constrain the matching of encoder output with discrete codebook; adversarial loss L adv Used to enhance the realism of globally reconstructed images.

[0113] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A method for removing image smudges, characterized in that, Includes the following steps: S1) Construct the visual generative model VQ-GAN and train it; S2) Perform local denoising processing on the image to be removed to obtain a locally denoised image; S3) Use the trained visual generative model VQ-GAN to extract the global structural information of the image to be removed, and obtain the image reconstructed based on the global structural information; S4) The image after local denoising and the image reconstructed based on global structural information are weighted and fused to obtain an image with the light spots removed.

2. The image spot removal method according to claim 1, characterized in that: In step S1), the visual generative model VQ-GAN includes an encoder and a decoder, as well as a learnable encoder weighted mapping layer MapEnc and a decoder weighted mapping layer MapDec; the encoder receives the input image of the light spots to be removed. It generates a preliminary latent variable representation; the decoder is used to reconstruct the discretized latent variables back to the original graph domain; the encoder weighted mapping layer MapEnc and the decoder weighted mapping layer MapDec perform shape mapping and weighting operations on the latent variables at both ends of the encoder and decoder. By adding trainable weights at these two ends, the latent variables can better retain the overall structural information while undergoing shape transformation.

3. The image spot removal method according to claim 2, characterized in that: In step S1), the visual generative model VQ-GAN is represented as: VQ{Enc,Codebook,MapEnc,MapDec,Dec}; In the formula, Enc represents the encoder; Codebook is the code that quantizes continuous latent variables into discrete codes; Dec is the decoder; MapEnc is the encoder weighted mapping layer; MapDec is the weighted mapping layer for the decoder.

4. The image spot removal method according to claim 1, characterized in that: In step S2), local denoising processing is performed on the image to be denoised, specifically including the following steps: S21) The image of the light spot to be removed Divide into N local patches; that is: In the formula P i This represents the i-th image patch; G, W, and C represent the length, width, and channel dimensions, respectively. S22), for each image block P i Local denoising is performed using bilateral filtering, i.e.: In the formula, flocal is the local denoising operator; This represents the i-th image patch after filtering at the local location (x, y); flocal(P i (x,y) represents performing bilateral filtering at position (x,y) of the i-th image patch; W p Ω(x,y) represents the normalization factor, ensuring the sum of the weights is 1; Ω(x,y) represents the neighborhood window centered at (x,y); (x′,y′) represents the local position of the neighborhood window; P i (x′,y′) represents the i-th image patch centered at the local location (x′,y′); The normalization factor W p w(x,y,x′,y′) are respectively represented as: ΔI(x,y,x′,y′)=∣P i (x,y)-P i (x′,y′)∣; In the formula, Let be a spatial Gaussian function, and let its standard deviation σ be... s Control the influence range of neighboring spaces; A Gaussian function with a range and a standard deviation σ r The influence of pixel value similarity is controlled; d(x,x′,y,y′) represents the distance between positions (x,y) and (x′,y′); ΔI(x,y,x′,y′) represents the absolute value of the difference between the patch variables; S23) All the denoised image blocks are stitched together into a locally denoised image I through a recombination operation. local ,Right now: In the formula, This is a restructuring operation.

5. The image spot removal method according to claim 1, characterized in that: In step S3), the trained visual generative model VQ-GAN is used to extract the global structural information of the image to be cleaned of light spots, specifically including the following steps: S31) Given the input image First, the latent variables are obtained through the encoder (Encoder) and the encoder weighted mapping layer (MapEnc), namely: In the formula, Z′ is a latent variable; Z is the shape mapping of the latent variable Z′; Enc represents the encoder; S32) The shape mapping Z of the latent variable is quantized into a discrete code space through a discretization operation, that is: In the formula, Discrete code; Codebook represents the discretization operation; A set representing discrete codes; S33) Using a decoder to convert discrete code The shape is mapped back to the same shape as the latent variable Z, and then the reconstructed image is obtained through the decoder weighted mapping layer MapDec and the decoder Decoder. I global =Dec(Z′ * ); In the formula, I global This represents an image reconstructed based on global structural information; Dec represents the decoder. The decoder weighted mapping layer MapDec quantizes the representation Z of shape H′×W′×1. * Map back to Z′ of shape H′×W′×D * And in the process, more complete global information can be recovered through trainable weighted operations.

6. The image spot removal method according to claim 1, characterized in that: In step S4), a fusion weight α∈[0,1] is introduced to perform a weighted fusion of the local and global denoising results, that is: In the formula, I local The image after local denoising; I global An image reconstructed based on global structural information; This is the image after weighted fusion.

7. The image spot removal method according to claim 1, characterized in that: The total loss function L of the visual generative model VQ-GAN is expressed as: L=λ rec L rec +λ local L local +λ global L global +λ VQ L VQ ; In the formula, λ rec , λ local , λ global , λ VQ L represents the weight hyperparameters of the corresponding loss function; rec For reconstruction losses; L local For local consistency loss; L global To preserve the loss for the global structure; L VQ This refers to the loss associated with VQGAN.

8. The image spot removal method according to claim 7, characterized in that: The reconstruction loss L rec express; In the formula, I is the image of the light spot to be removed; This is the image after weighted fusion.

9. The image spot removal method according to claim 7, characterized in that: The local consistency loss L local For each local image patch P i The result after bilateral filtering Constraints are applied between them, expressed as follows: In the formula, P i Represents the i-th image patch; This represents the i-th image block after bilateral filtering. The global branch reconstructs image I using the visual generative model VQ-GAN and a mapping layer. global To better recover global structural information, a pre-trained feature extraction network φ(·) is used to extract high-level semantic features, and a global structure preservation loss L is designed. global for: In the formula, I is the image of the light spot to be removed.

10. The image spot removal method according to claim 8, characterized in that: The VQGAN related loss L VQ Including quantization loss L quant and combat losses L adv ;Right now: L VQ L quant +L adv ; Among them, the quantization loss L quant Used to constrain the matching of encoder output with discrete codebook; adversarial loss L adv Used to enhance the realism of globally reconstructed images.

Images