Image stray light removal method, device, equipment and storage medium

Abstract

The present invention is mainly used in the field of image processing technology. The present invention discloses a method, device, equipment and storage medium for removing stray light from an image. The method includes: obtaining an input image contaminated by stray light and compressing it to obtain an input vector; based on the resolution of the input image, extracting features from the input vector to obtain multiple hierarchical features; combining each hierarchical feature with the target intermediate layer in the denoising network of the pre-trained latent diffusion model to adjust the parameters of the pre-trained latent diffusion model; obtaining the input vector after noise processing, and denoising the input vector after noise processing through the denoising network of the pre-trained latent diffusion model after parameter adjustment to generate a denoised output vector; decompressing the denoised output vector to obtain a restored image with stray light removed. The present application effectively removes stray light from an image by improving the image processing capability of a deep learning model.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for removing stray light from an image. Background Art

[0002] In the field of image processing technology, there are two ways to remove stray light in images. One is to reduce the reflection between the lens surfaces by adding an anti-reflection layer to the lens before taking the image, thereby reducing the stray light in the captured image. The other is to remove stray light through artificial intelligence, but due to the limitations of the image processing capabilities of deep learning models, it is impossible to effectively remove stray light in images. Summary of the invention

[0003] The present invention provides a method, device, equipment and storage medium for removing stray light from an image, which effectively removes stray light in an image by improving the image processing capability of a deep learning model.

[0004] In a first aspect, the present invention provides a method for removing stray light from an image, the method comprising:

[0005] An input image contaminated by stray light is obtained and compressed to obtain an input vector;

[0006] Based on the resolution of the input image, feature extraction is performed on the input vector to obtain multiple levels of features;

[0007] Combining each of the hierarchical features with a target intermediate layer in a denoising network of a pre-trained latent diffusion model to adjust parameters of the pre-trained latent diffusion model;

[0008] Acquire a noise-processed input vector, and perform denoising on the noise-processed input vector through a denoising network of a pre-trained latent diffusion model with adjusted parameters to generate a denoised output vector;

[0009] The output vector after the denoising process is decompressed to obtain a restored image with stray light removed.

[0010] Optionally, decompressing the output vector after the denoising process to obtain a restored image with stray light removed includes:

[0011] Performing feature fusion processing on the output vector after the denoising process and the input vector to obtain a fused feature vector;

[0012] The fused feature vector is decompressed to obtain a restored image with stray light removed.

[0013] Optionally, the performing feature fusion processing on the output vector after the denoising process and the input vector to obtain a fused feature vector includes:

[0014] Get the brightness gradient mask;

[0015] Based on the brightness gradient mask, adjusting a preset attention value;

[0016] Based on the adjusted preset attention value, the denoising output vector is convolved with the input vector to obtain the fused feature vector.

[0017] Optionally, obtaining a brightness gradient mask includes:

[0018] Converting the input image into a matrix of brightness pixels;

[0019] The brightness pixel matrix is ​​subjected to pooling processing and function activation processing to obtain an extended brightness pixel matrix as the brightness gradient mask.

[0020] Optionally, the method further comprises:

[0021] Get real images;

[0022] Comparing the restored image with the real image and obtaining a reconstruction loss value;

[0023] The pre-trained latent diffusion model is updated based on the reconstruction loss value.

[0024] Optionally, the method further comprises:

[0025] Obtain a light source image dataset and a natural image dataset;

[0026] According to a preset screening rule, screening out a first image from the light source image data set, and screening out a second image from the natural image data set;

[0027] According to a preset fitting rule, the input image and the real image are generated based on the first image and the second image.

[0028] Optionally, the method further comprises:

[0029] Data enhancement processing is performed on the input image and the real image respectively, wherein the data enhancement processing includes image cropping, image flipping and image rotation.

[0030] In a second aspect, the present invention provides an image stray light removal device, the device comprising:

[0031] An input module, used for acquiring an input image polluted by stray light and compressing it to obtain an input vector;

[0032] An extraction module, configured to perform feature extraction on the input vector based on the resolution of the input image to obtain multiple levels of features;

[0033] A guiding module, for combining each of the hierarchical features with a target intermediate layer in a denoising network of a pre-trained latent diffusion model to adjust parameters of the pre-trained latent diffusion model;

[0034] A denoising module, used for obtaining a denoised input vector, and denoising the denoised input vector through a denoising network of a pre-trained latent diffusion model with adjusted parameters to generate a denoised output vector;

[0035] The output module is used to decompress the output vector after the denoising process to obtain a restored image with stray light removed.

[0036] In a third aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the image stray light removal method as described in any one of the above items when executing the computer program.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image stray light removal method as described in any one of the above items is implemented.

[0038] The present invention has at least the following beneficial effects: the technical solution of the present application utilizes a latent diffusion model to remove stray light, and therefore the input image contaminated by the stray light must first be compressed into an input vector to be input into the latent diffusion model, and then, by extracting multiple hierarchical features of different resolutions and combining each hierarchical feature with a target intermediate layer in a denoising network of a pre-trained latent diffusion model, the parameters of the pre-trained latent diffusion model can be adjusted to provide multi-scale structural guidance when the latent diffusion model generates an image, thereby improving the image processing capability of the deep learning model, and finally, the input vector is denoised and denoised using the pre-trained latent diffusion model after parameter adjustment, and the denoised output vector is decompressed into a restored image, thereby effectively removing the stray light in the input image and obtaining a restored image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0040] Figure 1 is a flow chart of steps of a method for removing stray light from an image;

[0041] Figure 2 It is a schematic diagram of the steps of extracting multiple levels of features of an image stray light removal method;

[0042] Figure 3 yes Figure 1 The step flow chart of step S500 in FIG.

[0043] Figure 4 yes Figure 3 The step flow chart of step S510 in FIG.

[0044] Figure 5 It is a schematic diagram of the feature fusion processing steps of an image stray light removal method;

[0045] Figure 6 It is a data flow diagram of an image stray light removal method;

[0046] Figure 7 It is a structural schematic diagram of an image stray light removal device;

[0047] Figure 8 It is a schematic diagram of the hardware structure of an electronic device. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0049] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0050] Stray light refers to non-imaging light that is incident on the image plane from outside the field of view of the optical system. Stray light affects the image quality, reduces the image signal-to-noise ratio, and thus affects the performance of downstream tasks. Stray light is mainly divided into two types: stray light caused by scattering and stray light caused by reflection. When the light emitted by the light source reaches the camera lens, water droplets, fingerprints, scratches, etc. on the lens will scatter the incident light to different positions on the image plane, resulting in the generation of scattered stray light. At the same time, the lens of a modern camera is composed of a combination of multiple lenses. When refracting the incident light, reflections will occur between the mirror surfaces of different lenses, which will also converge the incident light to different positions on the image plane, resulting in the generation of reflected stray light. The stray light generated by scattering usually appears as a beam-like bright spot, while the stray light generated by reflection mostly appears as circular or polygonal artifacts. In the field of computer vision, stray light removal technology has a long history of research. Early stray light removal mainly relied on pre-shooting processing, that is, by adding an anti-reflection layer to the lens to reduce the reflection between the lens surfaces. However, this method cannot remove all stray light, has poor stability, and cannot be processed after shooting.

[0051] In recent years, with the continuous development of artificial intelligence, deep learning-based stray light removal methods have become the mainstream method for removing stray light. It regards stray light removal as a local image restoration task, and attempts to restore a stray light-free image from an observed image contaminated by stray light. Most of the previous deep learning methods were trained from scratch, that is, a neural network was used to fit the mapping from an image contaminated by stray light to an image contaminated by stray light. However, such methods have high training costs and are unable to combine the natural image priors contained in the pre-trained large model, resulting in unsatisfactory restoration results, which often produce overly sharp edges and more obvious chromatic aberrations near the restored light source. Limited by the image processing capabilities of the deep learning model, it is impossible to effectively remove stray light from the image.

[0052] People think of improving the image processing capabilities of deep learning models by adjusting the parameters of pre-trained large models. In order to complete specific downstream tasks while utilizing the natural image priors contained in pre-trained large models, researchers have proposed various parameter fine-tuning techniques for pre-trained large models. Low-rank adaptation (LoRA) and control network (ControlNet) are commonly used large model parameter fine-tuning methods, most of which follow the idea of ​​freezing the parameters of the pre-trained network first, and then adding a trainable bypass network to achieve fine-tuning to adapt to downstream tasks. Most of these methods are aimed at image generation tasks, aiming to add precise control of the structure or shape of the generated image. However, so far, there are relatively few studies on the application of pre-trained large models to image restoration tasks, and there is no application for stray light removal tasks.

[0053] Existing stray light removal methods based on deep learning have the following limitations: On the one hand, large models need to be trained from scratch, and the natural image priors contained in the pre-trained large models are not combined, resulting in unsatisfactory image restoration results. On the other hand, the existing image restoration methods combined with the pre-trained latent diffusion model (LDM) will cause information loss during the image restoration process due to the randomness of the sampling process and the compression of the latent space. Therefore, they can only be used for global image restoration tasks such as image super-resolution and dehazing, and cannot be directly transferred to the local image restoration task of stray light removal. Therefore, the image processing ability of removing image stray light is weak.

[0054] In order to solve the above problems, the technical solution of the present application uses a latent diffusion model to remove stray light. Therefore, the input image contaminated by stray light must first be compressed into an input vector input to the latent diffusion model. Then, by extracting multiple hierarchical features of different resolutions and combining each hierarchical feature with the target intermediate layer in the denoising network of the pre-trained latent diffusion model, the parameters of the pre-trained latent diffusion model can be adjusted to provide multi-scale structural guidance when the latent diffusion model generates an image, thereby improving the image processing capability of the deep learning model. Finally, the input vector is denoised and denoised using the pre-trained latent diffusion model after parameter adjustment, and the denoised output vector is decompressed into a restored image, thereby effectively removing the stray light in the input image and obtaining a restored image. The following are various embodiments provided by the technical solution of the present application.

[0055] Please refer to Figure 1 , Figure 1 The present invention is a flowchart of the steps of a method for removing stray light from an image.

[0056] In a first aspect, an embodiment of the present application provides a method for removing stray light from an image, the method comprising the following steps:

[0057] S100, obtaining an input image polluted by stray light and compressing it to obtain an input vector.

[0058] S200 , based on the resolution of the input image, extract features from the input vector to obtain features at multiple levels.

[0059] S300, combining each level feature with the target intermediate layer in the denoising network of the pre-trained latent diffusion model to adjust the parameters of the pre-trained latent diffusion model.

[0060] S400, obtaining a denoised input vector, and denoising the denoised input vector through a denoising network of a pre-trained latent diffusion model with adjusted parameters to generate a denoised output vector.

[0061] S500 , decompressing the output vector after denoising to obtain a restored image with stray light removed.

[0062] In step S100 of some embodiments, the input image is compressed into a latent space by a pre-trained vector quantization generative adversarial network (VQ-GAN) to obtain an input vector z of the latent space i .

[0063] Please refer to Figure 2 In step S300 of some embodiments, the input vector is divided into multiple hierarchical features through step S200, and the resolution of each hierarchical feature is different. The latent space input z is extracted from the high level to the low level through the normalization layer. i If we extract the hierarchical features of the Lth layer , the hierarchical features Combined with the target intermediate layer (the denoising network of the pre-trained latent diffusion model, which can be a convolutional layer or a pooling layer). The combination method is:

[0064]

[0065] in, and Convolutional layers in a denoising network for a pre-trained latent diffusion model.

[0066] In addition, in some other embodiments, the extracted multiple hierarchical features of different resolutions are used as semantic graphs (matrices), which are combined with the target intermediate layer (the target intermediate layer is located in the denoising network UNet of the pre-trained latent diffusion model) by affine transformation through a spatially-adaptive normalization (SPADE) layer.

[0067] In step S400 of some embodiments, the method of obtaining the noise-processed input vector may be to perform noise processing on the input vector through the pre-trained latent diffusion model to obtain the noise-processed input vector.

[0068] It can be understood that in some embodiments, a structure-guided perception module is constructed and the input vector is input into the structure-guided perception module to extract multiple levels of features at different resolutions and combine the multiple levels of features with the corresponding target intermediate layer.

[0069] In step S500 of some embodiments, the denoising network UNet samples and generates a denoised output vector z r The pre-trained vector quantization generative adversarial network VQ-GAN is decompressed into pixel space to obtain a restored image with stray light removed.

[0070] Please refer to Figure 3 In some embodiments, step S500 includes:

[0071] S510: Perform feature fusion processing on the output vector after denoising and the input vector to obtain a fused feature vector.

[0072] S520: decompress the fused feature vector to obtain a restored image with stray light removed.

[0073] In step S510 of some embodiments, an adaptive feature fusion module is constructed, and the output vector after denoising is fused with the input vector by the adaptive feature fusion module. When training the newly constructed adaptive feature fusion module, the output vector z after denoising in the history can be fused. r As training data, train the adaptive feature fusion module.

[0074] It is understood that in some embodiments, the adaptive feature fusion module is an autoencoder composed of several layers of convolution (Conv) and residual dense blocks (Residual-in-Residual Dense Block, RRDB), which receives the input vector z spliced ​​in the channel dimension i and the denoised output vector z r As input, the specific expression is as follows:

[0075]

[0076] in It represents concatenation in dimension, which is usually composed of 2 RRDB layers and 2 convolutional layers.

[0077] The adaptive feature fusion module is used to fuse z i With z r , and the fused feature vector The image is passed to VQ-GAN for decompression to obtain a restored image with stray light removed.

[0078] It can be understood that this embodiment performs feature fusion processing on the output vector and the input vector to compensate for the randomness in the sampling process of the potential diffusion model and the information loss caused by the latent space compression, thereby maintaining the consistency of the input and output images in the area not contaminated by stray light.

[0079] Please refer to Figure 4 In some embodiments, step S510 includes:

[0080] S511, obtaining a brightness gradient mask.

[0081] S512: Adjust the preset attention value based on the brightness gradient mask.

[0082] S513: Based on the adjusted preset attention value, convolution processing is performed on the output vector after denoising and the input vector to obtain a fused feature vector.

[0083] In step S512 of some embodiments, the brightness gradient mask is used as a coefficient to adjust the weight matrix of the self-attention layer:

[0084]

[0085]

[0086] Among them, m represents the number of self-attention layers, Query, key, and value for self-attention calculation. is the input vector of the self-attention layer, is the mapping matrix, Represents the attention value of the mth layer, and Softmax represents the flexible maximum function.

[0087] See also Figure 5 , based on the brightness gradient mask, the preset attention value is adjusted to adjust the self-attention layer. Figure 5 As shown, for the input vector z i and the denoised output vector z r , when performing convolution processing, change the attention mechanism, that is, focus on the areas in the image that are not contaminated by stray light,

[0088] In some embodiments, step S511 includes:

[0089] The input image is converted into a brightness pixel matrix; the brightness pixel matrix is ​​pooled and activated to obtain an extended brightness pixel matrix as a brightness gradient mask.

[0090] In some embodiments, the input image is converted from the RGB space to the Y*CbCr luminance space, where the range of Y is usually [0,1], and a threshold s is set. In experiments, we usually take s=0.9 and generate the value of the i-th element of the mask LM (Luminance Mask) with a luminance pixel matrix size of 512x512 according to the following rules:

[0091]

[0092] Then, LM is average pooled and activated with SiLU function to obtain a mask of size 64x64. At this time, the mask is not a binary mask (not only 0 and 1) because it has been activated. Then, it is flattened to size 1x4096 and copied into 4096 rows to obtain a mask LM' of size 4096x4096. Mask LM' is the brightness gradient mask.

[0093] It can be understood that this embodiment uses a brightness gradient mask to guide the self-attention layer therein, focusing more on maintaining the structural consistency of the area not contaminated by stray light, thereby more effectively focusing on the area not contaminated by stray light.

[0094] In some embodiments, the image stray light removal method further includes:

[0095] Obtain a real image; compare the restored image with the real image and obtain a reconstruction loss value; based on the reconstruction loss value, update the pre-trained latent diffusion model.

[0096] In some embodiments, the real image refers to an image obtained by photographing an actual scene with a light source. In an embodiment in which multiple levels of feature injection are performed through a structure-guided perception module, after obtaining the restored image, the restored image is compared with the real image and the reconstruction loss value L1 is calculated, that is, by comparing the restored image with the real image, the difference between the pixels of the restored image and the pixels of the real image is determined, and the gradient is returned to the structure-guided perception module to update the parameters of the structure-guided perception module. In an embodiment in which feature fusion processing is performed through a feature fusion module, after obtaining the restored image, the restored image is compared with the real image and the reconstruction loss value L2 is calculated, that is, by comparing the restored image with the real image, the difference between the pixels of the restored image and the pixels of the real image is determined, and the gradient is returned to the feature fusion module to update the parameters of the feature fusion module, thereby completing the training of the feature fusion module.

[0097] This embodiment can enhance the analysis capability of multiple-level features and the processing capability of feature fusion.

[0098] In some embodiments, the image stray light removal method further includes:

[0099] Acquire a light source image dataset and a natural image dataset; filter out a first image from the light source image dataset and a second image from the natural image dataset according to a preset screening rule; and generate an input image and a real image based on the first image and the second image according to a preset fitting rule.

[0100] It is understandable that the light source image dataset and the natural image dataset can generate training data in the dataset through a data synthesis method. The light source image dataset includes light source images, stray light images caused by scattering, and stray light images caused by reflection, and the natural image dataset includes actual scene images.

[0101] The preset screening rule in some embodiments is: randomly select a first image from the light source image data set, and the first image may be a light source image L, a scattered stray light image S, or a reflected stray light image R. Randomly select a second image from the natural image data set, and the second image is a natural image B. It can be understood that in other embodiments, the first image selected from the light source image data set is the light source image L, and the natural image B with stray light is selected from the natural image data set.

[0102] When synthesizing training data, one of the preset fitting rules is: combine it with the light source image L, the scattered stray light image S and the reflected stray light image R to generate an input image; combine the natural image B with the light source image L to generate a real image. Another preset fitting rule can also be: generate an input image based on the light source image L and the scattered stray light image S, and use the natural image B as the real image.

[0103] This embodiment generates input images and real images according to the light source image dataset and the natural image dataset, which is conducive to obtaining a large amount of data for training the latent diffusion model, thereby facilitating improving the image processing capability of the latent diffusion model.

[0104] In some embodiments, the image stray light removal method further includes:

[0105] Data enhancement processing is performed on the input image and the real image respectively, wherein the data enhancement processing methods include image cropping, image flipping and image rotation.

[0106] It can be understood that when the data enhancement processing method is image cropping, the input image and the real image are cropped into images of size 512x512 respectively.

[0107] See also Figure 6 , Figure 6 The figure is a data flow diagram of a method for removing stray light from an image in an embodiment.

[0108] First, the input image is obtained and a mask is generated; the input image is compressed into an input vector, and then the input vector is analyzed at multiple levels to obtain multiple levels of features, and each level of features is combined with the middle layer of the denoising network to guide the denoising network and adjust the parameters of the latent diffusion model; the denoised input vector is input into the adjusted latent diffusion model to complete the denoising process; then, the denoised output vector is fused with the input vector. During the fusion process, the self-attention layer is adjusted according to the mask to focus on the area without stray light in the image; finally, the fused vector is decompressed to obtain the restored image.

[0109] Second, reference Figure 7 This embodiment provides an image stray light removal device, the device comprising:

[0110] The input module 701 is used to obtain an input image polluted by stray light and compress it to obtain an input vector.

[0111] The extraction module 702 is used to perform feature extraction on the input vector based on the resolution of the input image to obtain multiple levels of features.

[0112] The guiding module 703 is used to combine each level feature with the target intermediate layer in the denoising network of the pre-trained latent diffusion model to adjust the parameters of the pre-trained latent diffusion model.

[0113] The denoising module 704 is used to obtain the input vector after the denoising process, and denoise the input vector after the denoising process through the denoising network of the pre-trained latent diffusion model after parameter adjustment to generate a denoised output vector.

[0114] The output module 705 is used to decompress the output vector after the denoising process to obtain a restored image with stray light removed.

[0115] It will be appreciated by those skilled in the art that all or some of the steps and devices in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. It is well known to those skilled in the art that communication media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery media.

[0116] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0117] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing any of the above methods for removing stray light from an image when executing the computer program.

[0118] refer to Figure 8 , Figure 8 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0119] The processor 801 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0120] The memory 802 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 802 can store operating devices and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 802, and the processor 801 calls and executes the image stray light removal method of the embodiment of the present application;

[0121] Input / output interface 803, used to implement information input and output;

[0122] The communication interface 804 is used to realize the communication interaction between the device and other devices. The communication can be realized through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0123] A bus 805 that transmits information between various components of the device (e.g., the processor 801, the memory 802, the input / output interface 803, and the communication interface 804);

[0124] The processor 801 , the memory 802 , the input / output interface 803 and the communication interface 804 are connected to each other in communication within the device via a bus 805 .

[0125] It can be understood that the contents of the above method embodiments are all applicable to the electronic device embodiment, the functions specifically implemented by the electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the image stray light removal method as described in any one of the above specific embodiments.

[0127] An embodiment of the present application also discloses a computer program product, including a computer program or computer instructions, wherein the computer program or computer instructions are stored in a computer-readable storage medium, a processor of a computer device reads the computer program or computer instructions from the computer-readable storage medium, and the processor executes the computer program or computer instructions, so that the computer device executes the image stray light removal method as described in any of the previous embodiments.

[0128] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0129] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. It should be understood that in the present application, "at least one (item)" refers to one or more, and "a plurality" refers to two or more.

[0130] In the several embodiments provided in the present application, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0133] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0134] Although the description of the present application has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims by reference to the attached claims, taking into account the prior art, so as to effectively cover the intended scope of the present application. In addition, the above description of the present application is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present application that have not yet been foreseen may still represent equivalent changes to the present application.

Claims

1. A method for removing stray light from an image, characterized in that: The method comprises: An input image contaminated by stray light is obtained and compressed to obtain an input vector; Based on the resolution of the input image, feature extraction is performed on the input vector to obtain multiple levels of features; Combining each of the hierarchical features with a target intermediate layer in a denoising network of a pre-trained latent diffusion model to adjust parameters of the pre-trained latent diffusion model; Acquire a noise-processed input vector, and perform denoising on the noise-processed input vector through a denoising network of a pre-trained latent diffusion model with adjusted parameters to generate a denoised output vector; Performing feature fusion processing on the output vector after the denoising process and the input vector to obtain a fused feature vector; Decompressing the fused feature vector to obtain a restored image with stray light removed; The step of performing feature fusion processing on the output vector after the denoising process and the input vector to obtain a fused feature vector includes: Get the brightness gradient mask; Based on the brightness gradient mask, the preset attention value is adjusted, wherein the brightness gradient mask is used as a coefficient to adjust the weight matrix of the self-attention layer: Among them, m represents the number of self-attention layers, Query, key, and value for self-attention calculation. is the input vector of the self-attention layer, is the mapping matrix, represents the attention value of the mth layer, Softmax represents the flexible maximum function, and mask That is the brightness gradient mask; Based on the adjusted preset attention value, the denoising output vector is convolved with the input vector to obtain the fused feature vector.

2. The method for removing stray light from an image according to claim 1, characterized in that: The step of obtaining a brightness gradient mask comprises: Converting the input image into a matrix of brightness pixels; The brightness pixel matrix is ​​subjected to pooling processing and function activation processing to obtain an extended brightness pixel matrix as the brightness gradient mask.

3. The method for removing stray light from an image according to claim 1, characterized in that: The method further comprises: Get real images; Comparing the restored image with the real image and obtaining a reconstruction loss value; The pre-trained latent diffusion model is updated based on the reconstruction loss value.

4. The method for removing stray light from an image according to claim 3, characterized in that: The method further comprises: Obtain a light source image dataset and a natural image dataset; According to a preset screening rule, screening out a first image from the light source image data set, and screening out a second image from the natural image data set; According to a preset fitting rule, the input image and the real image are generated based on the first image and the second image.

5. The method for removing stray light from an image according to claim 4, characterized in that: The method further comprises: Data enhancement processing is performed on the input image and the real image respectively, wherein the data enhancement processing includes image cropping, image flipping and image rotation.

6. An image stray light removal device, characterized in that: The device comprises: An input module, used for acquiring an input image polluted by stray light and compressing it to obtain an input vector; An extraction module, configured to perform feature extraction on the input vector based on the resolution of the input image to obtain multiple levels of features; A guiding module, for combining each of the hierarchical features with a target intermediate layer in a denoising network of a pre-trained latent diffusion model to adjust parameters of the pre-trained latent diffusion model; A denoising module, used for obtaining a denoised input vector, and denoising the denoised input vector through a denoising network of a pre-trained latent diffusion model with adjusted parameters to generate a denoised output vector; An output module, used for performing feature fusion processing on the output vector after the denoising process and the input vector to obtain a fused feature vector; Decompressing the fused feature vector to obtain a restored image with stray light removed; The step of performing feature fusion processing on the output vector after the denoising process and the input vector to obtain a fused feature vector includes: Get the brightness gradient mask; Based on the brightness gradient mask, the preset attention value is adjusted, wherein the brightness gradient mask is used as a coefficient to adjust the weight matrix of the self-attention layer: Among them, m represents the number of self-attention layers, Query, key, and value for self-attention calculation. is the input vector of the self-attention layer, is the mapping matrix, represents the attention value of the mth layer, Softmax represents the flexible maximum function, and mask That is the brightness gradient mask; Based on the adjusted preset attention value, the denoising output vector is convolved with the input vector to obtain the fused feature vector.

7. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the image stray light removal method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image stray light removal method according to any one of claims 1 to 5 is implemented.

Images