A Hint-Based Real Image Dehazing Method at Inference Time

Abstract

The present invention discloses a real image defogging method during cue-based inference, comprising the steps of obtaining a real foggy image and sampling a clear image from a dataset; resizing the clear image to the same size as the real foggy image and segmenting the image into blocks; migrating the haze concentration of the segmented image blocks based on the color shift of the real foggy image to obtain visual cues; constructing a feature adaptation module to adjust the statistical values ​​of the real foggy features based on the visual cues; inserting the feature adaptation module into each basic module of the encoder of a pre-trained model to output the adjusted features; and inputting the adjusted features into the decoder of the pre-trained model to obtain the defogging result of the real image. The method can be plug-and-play used with image defogging models pre-trained on synthetic datasets to help generate better defogging results during the inference phase.

Description

Technical Field

[0001] The present invention relates to the field of image defogging, and in particular to a real image defogging method during hint-based reasoning. Background Art

[0002] Haze is a common atmospheric phenomenon. Due to the scattering and absorption of light by haze particles dispersed in the air, images captured during haze conditions often suffer from low contrast, poor visibility, and color distortion, negatively impacting downstream high-level vision tasks such as object detection, image classification, and semantic segmentation. Image dehazing aims to restore a clear image from hazy images. As a non-positive definite problem, image dehazing has been a hot topic in image processing. Early methods addressed this issue by introducing various prior information. However, when real-world hazy images do not satisfy these prior information, dehazing artifacts may occur. With the development of deep learning, many studies have utilized convolutional neural networks and large-scale synthetic datasets to train image dehazing models. While these methods have achieved some progress, the domain shift between real hazy images and synthetic datasets results in poor generalization performance of convolutional neural networks for real hazy images.

[0003] Traditional image defogging methods often rely on prior information manually summarized from foggy and fog-free images. Although this information can achieve a certain defogging effect to some extent, it is only effective in scenes that meet its basic assumptions. Deep learning-based image defogging methods learn more powerful defogging priors from a large number of foggy and fog-free images. However, due to the difficulty in obtaining paired foggy and fog-free image data in the real world, a synthetic method is often used to generate foggy images from fog-free images for model training. However, due to the domain offset problem between synthetic data and real data, the performance of defogging models trained using synthetic data will degrade in real scenes. In order to improve the generalization performance of these models, the present invention proposes a real image defogging method based on hint reasoning that does not require training. Summary of the Invention

[0004] The present invention proposes a real image defogging method during hint-based inference, which aims to overcome the technical defects of the existing technology. By adjusting the statistical indicators of foggy image features according to the generated hints during inference, the domain offset between the synthetic dataset and the real foggy pictures is reduced, thereby improving the defogging performance of the pre-trained image defogging model.

[0005] The object of the present invention is achieved through the following technical solution: a real image defogging method based on hint reasoning, the method comprising the following steps:

[0006] S1. Obtain a real foggy image and sample a clear image in the dataset;

[0007] S2, resize the clear image to the same size as the real foggy image and divide the image into blocks;

[0008] S3, performing haze concentration migration on the segmented image blocks according to the color deviation of the real foggy image to obtain visual cues;

[0009] S4. Adding feature adaptation processing after each basic module of the pre-trained model encoder; the feature adaptation processing adjusts the statistical value of the real fog feature according to the visual clue;

[0010] S5, input the real foggy image and visual cues into the pre-trained model encoder and output the adjusted features;

[0011] S6. Input the adjusted features into the decoder of the pre-trained model to obtain the defogging result of the real image.

[0012] Furthermore, the color shift of the real foggy image is obtained by calculating the dispersion metric of the hue space of the real foggy image, specifically:

[0013]

[0014] Where MOS is the scatter metric in the hue space, A pixel in the hue space. A smaller MOS indicates a more severe color shift in the foggy image; a larger MOS indicates a less severe color shift in the foggy image.

[0015] Furthermore, the haze concentration migration is specifically as follows: when MOS ≥ τ, select Migrate the haze concentration; otherwise, select Carry out the migration of haze concentration;

[0016] Where p is the generated visual cue, y s is a clear image sampled from the synthetic dataset, x r is a real foggy image, μ c and σ c is the mean and variance of a certain color channel; and μ c and σ c The average value of .

[0017] Furthermore, obtaining the visual prompt specifically includes: integrating the image blocks after the haze migration back into the complete image, and resizing it to the original size to generate the prompt.

[0018] Furthermore, the specific formula of the feature adaptation module includes:

[0019]

[0020] in, is the real foggy image feature, is the image feature after statistical value adjustment, and represents the target value of the mean and variance of each channel calculated based on the corresponding foggy image features and prompt features;

[0021] When μ c (f p )and In the same quadrant, Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged; when σ c (f p )and When the difference is within a certain range, Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged.

[0022] According to another aspect of the specification, there is provided a system for real image defogging during cue-based reasoning, comprising a cue generation module, a feature adaptation module, and a pre-trained defogging module;

[0023] The prompt generation module is used to generate visual prompts to guide the feature adaptation module to adjust the feature statistics;

[0024] The pre-trained defogging module includes an image defogging model pre-trained on a synthetic dataset, wherein each basic module of the encoder is followed by a feature adaptation module, and the decoder outputs a defogging image according to the adjusted features.

[0025] According to another aspect of the specification, a device for defogging real images during prompt-based reasoning is also provided, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the method for defogging real images during prompt-based reasoning is implemented.

[0026] According to another aspect of the specification, a computer-readable storage medium is provided, on which a program is stored. When the program is executed by a processor, the method for defogging real images during prompt-based reasoning is implemented.

[0027] Beneficial effects of the present invention:

[0028] The present invention uses a cue generation module to synthetically generate visual cues that closely resemble the actual input haze distribution. A feature adaptation module is incorporated into the pre-trained dehazing network, using the feature statistics of the visual cues to adjust the statistics of the actual input features, thereby improving the model's generalization performance in real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of a real image defogging method during hint-based reasoning according to an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a prompt generation module according to an embodiment of the present invention;

[0031] Figure 3 2 is a schematic diagram of a real image defogging device during prompt-based reasoning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, the present invention provides a real image defogging method based on hint reasoning: the method mainly includes two parts, specifically:

[0034] Preprocessing: This part aims to generate a visual cue based on the real foggy image input during the inference phase, which can guide the adjustment of the internal statistics of the feature adaptation module. The generated visual cue must meet the following two criteria: (1) It should be similar to the synthetic domain so that the encoder part of a pre-trained dehazing model can correctly extract features. (2) It must have a haze distribution similar to that of the input foggy image so that the subsequent adjustment of the feature statistics will not be too large, avoiding the collapse of the entire network. Therefore, the cue generation module randomly samples a clear image from the clear images in the synthetic dataset, and then transfers a haze distribution with a haze concentration similar to that of the real foggy image to the clear image, thereby meeting these two criteria.

[0035] In the feature conversion part, we first select an image dehazing model pre-trained on a synthetic dataset (in this example, AECR-Net is selected as the pre-trained model). All parameters of the pre-trained image dehazing model are frozen, indicating that no subsequent parameter updates will be performed. Then, a feature adaptation module is specially configured after each basic module of the encoder of the pre-trained dehazing model. At the same time, the foggy image input and the generated prompt p are used as the input of the encoder specially configured with the feature adaptation module. Each feature encoder adjusts the feature statistical indicators of the foggy image according to the statistical indicators of the prompt feature. Finally, the adjusted features are input into the original decoder to produce better dehazing results.

[0036] like Figure 1 and Figure 2 As shown, according to the two parts of this method, the specific implementation steps are as follows:

[0037] S1. Obtain a real foggy image and sample a clear image in the dataset;

[0038] S2, adjust the size of the clear image to the same size as the real foggy image, and divide the image into square blocks. The side length of the image block is Where W is the width of the real foggy image.

[0039] S3. Migrating the haze concentration of the segmented image blocks based on the color shift of the real hazy image to obtain visual cues; including:

[0040] Calculate the distribution metric of the hue space of the real foggy image. The calculation formula is as follows:

[0041]

[0042] Where MOS is the scatter metric in the hue space, x H is the hue space of the image, A pixel in the hue space. A smaller MOS indicates a more severe color shift in the foggy image; a larger MOS indicates the opposite.

[0043] The haze concentration is transferred for each image block. Adaptive instance normalization is used to transfer the haze distribution by aligning the channel mean and variance of the clear image block with the channel mean and variance of the hazy image block. The formula is as follows:

[0044]

[0045] Where p is the generated visual cue, y s is a clear image sampled from the synthetic dataset, x r is a real foggy image, μ c and σ c is the mean and variance of a color channel.

[0046] At the same time, considering that real foggy images have color casts, the visual cues generated using the above formula have similar color distributions to real foggy images, making it difficult to correct the color cast of real foggy images. Therefore, this paper proposes adaptive instance normalization of color balance to generate visual cues for white balance, and the formula is as follows:

[0047]

[0048] in and μ c and σ c The average value of .

[0049] When MOS ≥ τ, choose Formula 1: to migrate the haze concentration; otherwise, choose formula 2: The value of τ is 0.005, which can be optimized from a large number of images with and without color cast.

[0050] Finally, the image patches are integrated back into the full image and resized to the original size to generate the prompt.

[0051] S4. Construct a feature adaptation module to adjust the statistical value of the real fog feature according to the visual clue. The specific formula of the feature adaptation module is as follows:

[0052]

[0053] Among them, f p To visually cue image features, is the real foggy image feature, is the image feature after statistical value adjustment, and The target values ​​of the mean and variance of each channel are calculated based on the corresponding foggy image features and prompt features. c (f p )and In the same quadrant, Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged. When σ c (f p )and The difference in the mean of When the standard deviation of Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged. The corresponding calculation formula is as follows:

[0054]

[0055] In the above formula, α is a hyperparameter used to control σ c (f p )and In the embodiment of the present invention, α=2.

[0056] S5. Insert the feature adaptation module into each basic module of the AECR-Net encoder and output the adjusted features;

[0057] S6. Input the adjusted features into the decoder of AECR-Net to obtain the defogging result of the real image.

[0058] According to the aforementioned method embodiment, the present invention also provides a real image defogging system during prompt-based reasoning, comprising a prompt generation module, a feature adaptation module, and a pre-trained defogging module;

[0059] The prompt generation module is used to generate visual prompts to guide the feature adaptation module to adjust the feature statistics;

[0060] The pre-trained defogging module includes an image defogging model pre-trained on a synthetic dataset, wherein each basic module of the encoder is followed by a feature adaptation module, and the decoder outputs a defogging image according to the adjusted features.

[0061] Corresponding to the aforementioned embodiment of a method for defogging real images during prompt-based reasoning, the present invention also provides an embodiment of a device for defogging real images during prompt-based reasoning.

[0062] See also Figure 3 An embodiment of the present invention provides a real image defogging device during prompt-based reasoning, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a real image defogging method during prompt-based reasoning in the above embodiment.

[0063] The embodiment of the real image defogging device provided by the present invention during prompt-based reasoning can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 3 As shown in the figure, a hardware structure diagram of a real image defogging device based on prompt reasoning provided by the present invention is located in any device with data processing capabilities, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0064] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0065] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0066] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for defogging real images during prompt-based reasoning in the above embodiment is implemented.

[0067] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0068] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for defogging real images during prompt-based reasoning.

[0069] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the claims.

[0070] It should be understood that the above general description and the detailed description that follows are exemplary and explanatory only and do not limit the present application. The present application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope of the present application. The scope of the present application is limited only by the appended claims.

Claims

1. A real image defogging method based on hints during reasoning, characterized in that: The method comprises the following steps: S1. Obtain a real foggy image and sample a clear image in the dataset; S2, resize the clear image to the same size as the real foggy image and divide the image into blocks; S3, performing haze concentration migration on the segmented image blocks according to the color deviation of the real foggy image to obtain visual cues; The color shift of the real foggy image is obtained by calculating the dispersion metric of the real foggy image hue space, specifically: Where MOS is the scatter metric in the hue space, x H is the hue space of the image, is a pixel in the hue space; the smaller the MOS, the more serious the color deviation of the foggy image; the larger the MOS, the smaller the color deviation of the foggy image. c is the mean value of a certain color channel; The haze concentration migration is specifically as follows: when MOS ≥ τ, select Migrate the haze concentration; otherwise, select Carry out the migration of haze concentration; Where p is the generated visual cue, y s is a clear image sampled from the synthetic dataset, x r is a real foggy image, σ c is the variance of a certain color channel; and μ c and σ c The average value of τ is obtained by optimizing the pictures with and without color bias. S4. Add feature adaptation processing after each basic module of the pre-trained model encoder; The feature adaptation process adjusts the statistical values ​​of the real foggy features according to the visual cues; S5, input the real foggy image and visual cues into the pre-trained model encoder and output the adjusted features; S6. Input the output adjusted features into the decoder of the pre-trained model to obtain the dehazing result of the real image.

2. The method for real image defogging during hint-based reasoning according to claim 1, characterized in that: Obtaining visual cues specifically includes: integrating the image blocks after the haze migration back into the complete image, and resizing it to the original size to generate cues.

3. The method for real image defogging during hint-based reasoning according to claim 1, characterized in that: The specific formula for the feature adaptation process includes: Among them, f p To visually cue image features, is the real foggy image feature, is the image feature after statistical value adjustment, and represents the target value of the mean and variance of each channel calculated based on the corresponding foggy image features and prompt features; When μ c (f p )and In the same quadrant, Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged; when σ c (f p )and The difference in the mean of When the standard deviation of Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged, where α is a hyperparameter used to control σ c (f p )and The size of the gap between them.

4. A system for implementing the method according to any one of claims 1 to 3, characterized in that: It includes a prompt generation module, a feature adaptation module, and a pre-trained dehazing module; The cue generation module is used to generate visual cues to guide the feature adaptation module to adjust feature statistics. The steps include obtaining a real foggy image and sampling a clear image from a data set; resizing the clear image to the same size as the real foggy image and segmenting the image blocks; and performing haze concentration migration on the segmented image blocks based on the color shift of the real foggy image to obtain visual cues. The pre-trained defogging module includes an image defogging model pre-trained on a synthetic dataset, including an encoder and a decoder, wherein the encoder generates features according to the image and adjusts them according to the feature adaptation module, and the decoder outputs a defogging image according to the adjusted features; The feature adaptation module is installed after each basic module of the encoder in the pre-trained defogging module, and is used to adjust the statistical value of the real foggy feature according to the visual clues.

5. A real image defogging device for prompt-based reasoning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, the method for defogging real images during hint-based reasoning according to any one of claims 1 to 3 is implemented.

6. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a real image defogging method during hint-based reasoning according to any one of claims 1 to 3 is implemented.

Images