Prompt-based real image defogging method during reasoning

Abstract

The invention discloses a prompt-based real image defogging method during reasoning, and the method comprises the steps: obtaining a real foggy image, and sampling a clear image in a data set; the size of the clear image is adjusted to be the same as that of the real foggy image, and image block segmentation is carried out; haze concentration migration is carried out on the segmented image blocks according to the color cast of the real foggy image, and a visual prompt is obtained; constructing a feature adaptation module, and adjusting a statistical value of a real foggy feature according to a visual prompt; inserting the feature adaptation module into each basic module of a pre-training model encoder, and outputting adjusted features; and inputting the adjusted features into a decoder of a pre-training model to obtain a defogging result of the real image. The image defogging model can be used for pre-training on a synthetic data set in a plug-and-play manner, so that a better defogging result is generated in an inference stage.

Description

Technical Field

[0001] The present invention relates to the field of image defogging, and in particular to a real image defogging method during reasoning based on prompts. Background Art

[0002] Haze is a traditional atmospheric phenomenon. Due to the scattering and absorption of light by haze particles scattered in the air, images taken in haze weather are usually affected by low contrast, poor visibility and color distortion, which has a negative impact on downstream high-level visual tasks such as object detection, image classification and semantic segmentation. Image dehazing aims to restore a clear image from a hazy image. As a non-positive definite problem, image dehazing has always been a hot research topic in the field of image processing. Early methods solved this problem by introducing different prior information. However, when the hazy images in the real world do not meet these prior information, the dehazing results will produce artifacts. With the development of deep learning, many studies have used convolutional neural networks and large-scale synthetic datasets to train image dehazing models. Although these methods have made some progress, the domain offset problem in real hazy images and synthetic datasets will lead to poor generalization performance of convolutional neural networks for real hazy images.

[0003] Traditional image defogging methods often rely on prior information artificially summarized from foggy and fog-free images. Although this information can achieve a certain defogging effect to some extent, it is only effective in scenes that meet its basic assumptions. Image defogging methods based on deep learning learn more powerful defogging priors from a large number of foggy and fog-free images. However, since it is difficult to obtain paired foggy-fog-free image data in the real world, a synthetic method is often used to generate foggy images from fog-free images for model training. However, due to the domain offset problem between synthetic data and real data, the performance of the defogging model trained with synthetic data will degrade in real scenes. In order to improve the generalization performance of these models, the present invention proposes a real image defogging method based on prompt reasoning that does not require training. Summary of the invention

[0004] The present invention proposes a real image defogging method during hint-based reasoning, which aims to overcome the technical defects of the prior art and reduce the domain offset between the synthetic dataset and the real foggy picture by adjusting the statistical indicators of the foggy image features according to the generated hints during reasoning, thereby improving the defogging performance of the pre-trained image defogging model.

[0005] The object of the present invention is achieved through the following technical solution: a real image defogging method based on prompting, the method comprising the following steps:

[0006] S1. Obtain real foggy images and sample clear images in the dataset;

[0007] S2, adjusting the size of the clear image to the same size as the real foggy image, and dividing the image into blocks;

[0008] S3, performing haze concentration migration on the segmented image blocks according to the color deviation of the real hazy image to obtain visual cues;

[0009] S4, adding feature adaptation processing after each basic module of the pre-trained model encoder; the feature adaptation processing adjusts the statistical value of the real foggy feature according to the visual prompt;

[0010] S5, input the real foggy image and visual cues into the pre-trained model encoder and output the adjusted features;

[0011] S6. Input the adjusted features into the decoder of the pre-trained model to obtain the defogging result of the real image.

[0012] Furthermore, the color shift of the real foggy image is obtained by calculating the distribution metric of the hue space of the real foggy image, specifically:

[0013]

[0014] Where MOS is the scatter metric in the hue space, is a pixel in the hue space. The smaller the MOS, the more serious the color deviation of the foggy image; the larger the MOS, the smaller the color deviation of the foggy image.

[0015] Furthermore, the haze concentration migration is specifically as follows: when MOS ≥ τ, select Migrate the haze concentration; otherwise, choose Carry out the migration of haze concentration;

[0016] Where p is the generated visual cue, y s is a clear image sampled from the synthetic dataset, x r is a real foggy image, μ c With σ c is the mean and variance of a color channel; and They are μ c and σ c The average value of .

[0017] Furthermore, obtaining the visual prompt specifically includes: integrating the image blocks after the haze migration back into the complete image, and resizing it to the original size to generate the prompt.

[0018] Furthermore, the specific formula of the feature adaptation module includes:

[0019]

[0020] in, is the real foggy image feature, is the image feature after statistical value adjustment, and represents the target value of the mean and variance of each channel calculated according to the features and prompt features of the corresponding foggy image;

[0021] When μ c (f p )and In the same quadrant, Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged; when σ c (f p )and When the difference is within a certain range, Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged.

[0022] According to another aspect of the specification, there is provided a system for real image defogging during cue-based reasoning, comprising a cue generation module, a feature adaptation module, and a pre-trained defogging module;

[0023] The prompt generation module is used to generate visual prompts to guide the feature adaptation module to adjust the feature statistical data;

[0024] The pre-trained defogging module includes an image defogging model pre-trained on a synthetic data set, wherein a feature adaptation module is installed after each basic module of the encoder, and the decoder outputs a defogging image according to the adjusted features.

[0025] According to another aspect of the specification, there is also provided a real image defogging device during prompt-based reasoning, comprising a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, the real image defogging method during prompt-based reasoning is implemented.

[0026] According to another aspect of the specification, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the real image defogging method based on prompt-based reasoning is implemented.

[0027] Beneficial effects of the present invention:

[0028] The present invention uses a prompt generation module to generate visual prompts that are similar to the real input haze distribution in a synthetic way. The present invention inserts a feature adaptation module into the pre-trained dehazing network, and uses the feature statistical indicators of visual prompts to adjust the statistical indicators of real input features, thereby improving the generalization performance of the model in real scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a schematic diagram of a real image defogging method during hint-based reasoning according to an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a prompt generation module according to an embodiment of the present invention;

[0031] Figure 3 4 is a schematic diagram of a real image defogging device during prompt-based reasoning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0033] like Figure 1 As shown, the present invention provides a real image defogging method based on prompting during reasoning: the method mainly includes two parts, specifically:

[0034] Preprocessing part: The purpose of this part is to generate a visual cue based on the real foggy image input in the inference phase, which can guide the adjustment of the internal statistics of the feature adaptation module. The generated visual cue must meet the following two criteria: (1) It should be similar to the synthetic domain so that the encoder part of a pre-trained dehazing model can correctly extract features. (2) It must have a haze distribution similar to the input foggy image so that the subsequent adjustment of the feature statistics will not be too large to avoid the collapse of the entire network. Therefore, the cue generation module randomly samples a clear image from the clear images in the synthetic dataset, and then transfers the haze distribution similar to the haze concentration of the real foggy image to the clear image, thereby meeting these two criteria.

[0035] In the feature conversion part, first select an image defogging model pre-trained on a synthetic dataset (in this example, AECR-Net is selected as the pre-trained model). All parameters of the pre-trained image defogging model are frozen, indicating that no subsequent parameter updates will be performed. Then, a feature adaptation module is specially equipped behind each basic module of the encoder of the pre-trained defogging model. At the same time, the foggy image input and the generated prompt p are used as the input of the encoder specially equipped with the feature adaptation module. Each feature encoder adjusts the feature statistical indicators of the foggy image according to the statistical indicators of the prompt feature. Finally, the adjusted features are input into the original decoder to produce better defogging results.

[0036] like Figure 1 and Figure 2 As shown, according to the two parts of this method, the specific implementation steps are as follows:

[0037] S1. Obtain real foggy images and sample clear images in the dataset;

[0038] S2, adjust the size of the clear image to the same size as the real foggy image, and divide the image into square blocks. The side length of the image block is Where W is the width of the real foggy image.

[0039] S3, performing haze concentration migration on the segmented image blocks according to the color deviation of the real hazy image to obtain visual cues; including:

[0040] Calculate the distribution metric of the hue space of the real foggy image. The calculation formula is as follows:

[0041]

[0042] Where MOS is the scatter metric in the hue space, x H is the hue space of the image, is a pixel in the hue space. The smaller the MOS, the more serious the color deviation of the foggy image; the larger the MOS, the worse the color deviation.

[0043] The haze concentration is migrated for each image block. Adaptive instance normalization is used to transfer the haze distribution by aligning the channel mean and variance of the clear image block with the channel mean and variance of the hazy image block. The formula is as follows:

[0044]

[0045] Where p is the generated visual cue, y s is a clear image sampled from the synthetic dataset, x r is a real foggy image, μ c With σ c is the mean and variance of a color channel.

[0046] At the same time, considering the color cast of real foggy images, the visual cues generated using the above formula have similar color distribution to the real foggy images, and it is difficult to correct the color cast of the real foggy images. Therefore, the present invention proposes adaptive instance normalization of color balance to generate visual cues of white balance, and the formula is as follows:

[0047]

[0048] in and They are μ c and σ c The average value of .

[0049] When MOS ≥ τ, select formula 1: To migrate the haze concentration; otherwise, choose formula 2: The value of τ is 0.005, which can be optimized from a large number of images with and without color cast.

[0050] Finally, the image patches are integrated back into the full image and resized to the original size to generate the prompt.

[0051] S4. Construct a feature adaptation module to adjust the statistical value of the real foggy feature according to the visual prompt. The specific formula of the feature adaptation module is as follows:

[0052]

[0053] Among them, f p To visually cue image features, is the real foggy image feature, is the image feature after statistical value adjustment, and The target values ​​of the mean and variance of each channel are calculated based on the features and prompt features of the corresponding foggy image. c (f p )and In the same quadrant, Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged. When σ c (f p )and The difference in the mean of When the standard deviation of Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged. The corresponding calculation formula is as follows:

[0054]

[0055] In the above formula, α is a hyperparameter used to control σ c (f p )and In the example of the present invention, α=2.

[0056] S5, insert the feature adaptation module into each basic module of the AECR-Net encoder and output the adjusted features;

[0057] S6. Input the adjusted features into the decoder of AECR-Net to obtain the defogging result of the real image.

[0058] According to the aforementioned method embodiment, the present invention also provides a prompt-based reasoning real image defogging system, including a prompt generation module, a feature adaptation module and a pre-trained defogging module;

[0059] The prompt generation module is used to generate visual prompts to guide the feature adaptation module to adjust the feature statistical data;

[0060] The pre-trained defogging module includes an image defogging model pre-trained on a synthetic data set, wherein a feature adaptation module is installed after each basic module of the encoder, and the decoder outputs a defogging image according to the adjusted features.

[0061] Corresponding to the aforementioned embodiment of a real image defogging method during prompt-based reasoning, the present invention also provides an embodiment of a real image defogging device during prompt-based reasoning.

[0062] See also Figure 3 An embodiment of the present invention provides a real image defogging device during prompt-based reasoning, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a real image defogging method during prompt-based reasoning in the above embodiment.

[0063] An embodiment of a real image defogging device based on prompt-based reasoning provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 3 As shown, it is a hardware structure diagram of a real image defogging device based on prompt reasoning provided by the present invention, in which any device with data processing capability is located, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0064] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0065] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0066] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a real image defogging method based on prompt-based reasoning in the above embodiment is implemented.

[0067] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0068] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the real image defogging method based on prompt-based reasoning.

[0069] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0070] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.

Claims

1. A real image defogging method based on hints during reasoning, characterized in that: The method comprises the following steps: S1. Obtain real foggy images and sample clear images in the dataset; S2, adjusting the size of the clear image to the same size as the real foggy image, and dividing the image into blocks; S3, performing haze concentration migration on the segmented image blocks according to the color deviation of the real hazy image to obtain visual cues; S4, add feature adaptation processing after each basic module of the pre-trained model encoder; The feature adaptation process adjusts the statistical value of the real foggy feature according to the visual cue; S5, input the real foggy image and visual cues into the pre-trained model encoder and output the adjusted features; S6. Input the adjusted features into the decoder of the pre-trained model to obtain the defogging result of the real image.

2. The method for real image defogging during hint-based reasoning according to claim 1, characterized in that: The color shift of the real foggy image is obtained by calculating the distribution metric of the hue space of the real foggy image, specifically: Where MOS is the scatter metric in the hue space, x H is the hue space of the image, is a pixel in the hue space; the smaller the MOS, the more serious the color deviation of the foggy image; the larger the MOS, the smaller the color deviation of the foggy image.

3. The method for real image defogging during hint-based reasoning according to claim 2, characterized in that: The haze concentration migration is specifically as follows: when MOS ≥ τ, select ) to migrate the haze concentration; otherwise, select Carry out the migration of haze concentration; Where p is the generated visual cue, y s is a clear image sampled from the synthetic dataset, x r is a real foggy image, μ c With σ c is the mean and variance of a color channel; and They are μ c and σ c The value of τ is obtained by optimizing the color-biased and color-free images.

4. The method for real image defogging during hint-based reasoning according to claim 1, characterized in that: Obtaining visual cues specifically includes: integrating the image blocks after the haze migration back into the complete image, and resizing it to the original size to generate cues.

5. The method for real image defogging during hint-based reasoning according to claim 1, characterized in that: The specific formula of the feature adaptation processing includes: Among them, f p To visually cue image features, is the real foggy image feature, is the image feature after statistical value adjustment, and represents the target value of the mean and variance of each channel calculated according to the features and prompt features of the corresponding foggy image; When μ c (f p )and In the same quadrant, Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged; when σ c (f p )and The difference in the mean of When the standard deviation of Otherwise, in order to avoid excessive fluctuations in statistical values, Keep the original value unchanged, where α is a hyperparameter used to control σ c (f p )and The size of the gap between.

6. A system for implementing the method according to any one of claims 1 to 5, characterized in that: It includes a prompt generation module, a feature adaptation module, and a pre-trained dehazing module; The prompt generation module is used to generate visual prompts to guide the feature adaptation module to adjust the feature statistical data; including obtaining a real foggy image and sampling a clear image in a data set; adjusting the size of the clear image to be the same as the size of the real foggy image, and dividing the image blocks; performing haze concentration migration on the divided image blocks according to the color deviation of the real foggy image to obtain a visual prompt; The pre-trained defogging module includes an image defogging model pre-trained on a synthetic data set, including an encoder and a decoder, wherein the encoder generates features according to the image and adjusts them according to the feature adaptation module, and the decoder outputs a defogging image according to the adjusted features; The feature adaptation module is installed after each basic module of the encoder in the pre-trained defogging module, and is used to adjust the statistical value of the real foggy feature according to the visual clues.

7. A real image defogging device for prompt-based reasoning, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, a real image defogging method during hint-based reasoning as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a real image defogging method based on hints during reasoning as described in any one of claims 1 to 5 is implemented.

Images