Image defogging method and system based on depth information fog and transformer network

Abstract

The application discloses a kind of based on depth-of-field information fog and the image defogging method and system of Transform network, the method includes: using atmospheric scattering model and the method of depth estimation constructs training set, constructs feature concatenation module and coding-branch decoding module, the feature concatenation module is used to extract the high-dimensional feature of input image, the coding-branch decoding module is reconstructed to the high-dimensional feature extracted, and the generated haze-free image is obtained;Constructing defogging network model and corresponding loss function, using training set to train the defogging network model, the defogging network model includes feature concatenation module and coding-branch decoding module;Image defogging is completed using the defogging network model trained.The application is shown by a large number of experiments on mainstream data set that the proposed method realizes the most advanced defogging result.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image synthesis, and particularly relates to an image defogging method and system based on depth-of-field information and a Transform network. BACKGROUND

[0002] Image defogging is to eliminate the shielding and weakening of image details caused by the fog layer in the image while maintaining the clarity of the original image. This task plays an important role in the data preprocessing of many applications, such as automatic driving, city security, and other recognition tasks. In view of this, many researchers have proposed many methods for this task. In order to generate a target defogging image, in the defogging method based on a physical atmospheric scattering model, the estimated transmittance map and the atmospheric light map corresponding to the foggy image are obtained by transforming the original foggy image, and finally the defogged image is obtained through the model.

[0003] However, the use of a physical model or a simple combination of a physical model and a deep learning method ignores the non-uniformity of the fog in the image when restoring the fog-free image through the model, and often constructs a single and uniform transmittance map and atmospheric light map. Moreover, in the synthetic fog data set used for training, a fixed set of fog-containing data sets is often formed in advance. However, in a natural environment, the physical parameters corresponding to the fog in the same scene often change. On the other hand, actual fog formation involves many factors, and some factors are not fully considered in the atmospheric scattering model. The model trained by the data set constructed in this way has poor effect on real-world image defogging.

[0004] In addition, directly using a convolutional neural network to simply train to obtain the parameters in the physical model can cause the objects in the defogged image to form white edges, which greatly affects the quality of the image. In order to better cope with this challenge, the present application proposes an image defogging method based on the combination of a convolutional neural network and a physical model. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the present application provides an image defogging method and system based on depth-of-field information and a Transform network. In order to fully utilize the depth information of the image and generate synthetic foggy images in different time and space states, the present application uses two different fogging methods based on a physical model and random noise to generate a foggy data set with greater randomness in time and space, enhance the diversity of the generated foggy data set in time and space, and enhance the defogging ability of the defogging network obtained by the final training in the real-world foggy image.

[0006] In view of the problem that feature extraction is insufficient at a low scale feature level due to a traditional convolution operation, the application provides an encoding-branch decoding module based on a transformer, which can retain more image detail features to synthesize a higher-quality haze-free image.

[0007] In one aspect, the application provides an image haze removal method based on haze formation information and a transformer network, comprising the following steps:

[0008] Step 1: haze is added to an original haze-free image containing depth information according to an atmospheric scattering model, and the original haze-free image containing depth information and the image after haze addition are cropped to obtain a pair of haze and haze-free image segments, thereby forming a first pair of haze image training sets α ;

[0009] Step 2: the depth information image corresponding to a single original haze-free image not containing depth information is estimated using a depth estimation method, then haze is added to the original haze-free image not containing depth information, and the original haze-free image not containing depth information and the image after haze addition are cropped to obtain a pair of haze and haze-free image segments, thereby forming a second pair of haze image training sets β ,

[0010] Step 3: random haze is added to a single original haze-free image not containing depth information using a method of adding salt and pepper noise, then the original haze-free image not containing depth information and the image after random haze addition are cropped to obtain a pair of haze and haze-free image segments, thereby forming a third pair of haze image training sets δ ;

[0011] Step 4: a feature concatenation module and an encoding-branch decoding module are constructed, the feature concatenation module is used to extract high-dimensional features of an input image, and the encoding-branch decoding module is used to reconstruct the extracted high-dimensional features to obtain a generated haze-free image;

[0012] Step 5: a haze removal network model and a corresponding loss function are constructed, the haze removal network model is trained using the training sets in steps 1-3, and the haze removal network model comprises the feature concatenation module and the encoding-branch decoding module;

[0013] Step 6: the trained haze removal network model is used to complete image haze removal.

[0014] Further, the specific implementation of step S1 is as follows:

[0015] The public image dataset containing depth information is introduced, the public image dataset is added with a fog layer according to an atmospheric scattering model and corresponding depth information map, with time-varying random fog parameters, a random part of image segments is cropped from each complete original image in the public image dataset in a uniform window size, image segments of the same size are cropped from the corresponding area of the fog-added image, a pair of training samples is constructed through the cropped image segments in the original image and the fog-added image segments, and a first paired foggy image training set is constructed through multiple training samples α .

[0016] Further, the atmospheric scattering model is:

[0017]

[0018] wherein x represents the position of a pixel point, represents the atmospheric light value at infinity, that is, the foggy image obtained by the detection system, and represents the haze-free image, is a transmission function, representing the degree of light attenuation, wherein is the depth information at the point, and represents the scattering coefficient of the medium in which the light propagates.

[0019] Further, the specific implementation mode of constructing the feature concatenation module in step 4 is as follows:

[0020] Let the first paired foggy image dataset described in step 1 be (Adata k (x,y), Bdata k (x,y)), the second paired foggy image dataset described in step 2 be (Cdata k (x,y), Ddata k (x,y)), and the third paired foggy image dataset described in step 3 be (Edata k (x,y), Fdata k (x,y)), k∈[1,L], x∈[1,M], y∈[1,N]; merge the haze-free image data parts Adata k (x,y), Cdata k (x,y), and Edata k (x,y) in steps 1, 2, and 3 to obtain Gdata n (x,y), and merge the foggy image data parts Bdata k (x,y), Ddata k (x,y), and Fdata k (x,y) to obtain Hdata n(x,y);

[0021] Where L is the number of foggy image segments in the paired foggy image training set, and Adata k (x, y) represents the pixel in the x-th row and y-th column of the k-th hazy-free image segment in the paired hazy image training set, Bdata k (x,y) represents the pixel in the x-th row and y-th column of the k-th foggy image segment in the paired foggy image training set, M is the number of rows of the window used for cropping, and N is the number of columns of the window used for cropping; the subscript n indicates that the sum of the number of data samples corresponding to Adatak, Cdatak, and Edatak is n;

[0022] The specific processing procedure of the feature concatenation module is as follows:

[0023] 1) Use a convolution kernel with a stride of 1 and a kernel size of 1. A convolutional layer of 1 is suitable for images with foggy sources. Feature extraction is performed; where H is Hdata. n One of the images in (x,y);

[0024] 2) A ReLU layer is used to process the output of step 1) to prevent the network from overfitting;

[0025] 3) Use a convolution kernel with a stride of 1 and a kernel size of 3. The convolutional layer in step 3 extracts features from the output of step 2);

[0026] 4) Use a ReLU layer to process the output of step 3) to prevent the network from overfitting;

[0027] 5) After concatenating the outputs of 2) and 4), use a convolution with a stride of 1 and a kernel size of 5. A convolutional layer of 5 is used to extract features;

[0028] 6) Use a ReLU layer to process the output of step 5) to prevent the network from overfitting;

[0029] 7) After concatenating the outputs of 4) and 6), use a convolution with a stride of 1 and a kernel size of 7. The convolutional layer of layer 7 is used to extract features;

[0030] 8) Use a ReLU layer to process the output of step 7) to prevent the network from overfitting;

[0031] 9) After concatenating the outputs of 2), 4), 6), and 8), use a convolution with a stride of 1 and a kernel size of 3. 3 convolutional layers are used to extract features;

[0032] 10) The output of 9) is concatenated with the foggy image H corresponding to the input in 1) to obtain the extracted high-dimensional features.

[0033] Further, the specific implementation mode of the encoding-branch decoding module in step 4 is as follows:

[0034] 41) Taking the output of the feature concatenation module as input, using the Encoder module composed of Transformer to encode the image feature information, and outputting encoded information containing four different sizes;

[0035] 42) Taking the output of 41) as input, using the Decoder module 1 composed of Transformer to decode the image feature information, and outputting the decoding information of the fifth size;

[0036] 43) Taking the output of 42) as input, using the inverse convolution layer with a step of 2 to perform 6 times of upsampling; after each upsampling, add the encoded information with the same size in S1) respectively as the input of the next upsampling, and finally obtain the intermediate image .

[0037] 44) Taking the output of 41) as input, using the Decoder module 2 composed of Transformer to decode the image feature information;

[0038] Among them, the Encoder module, the Decoder module 1 and the Decoder module 2 all contain one or more layers of stacked Transformer Block and parallel Intra Patch Transformer Block;

[0039] 45) Taking the output of 44) as input, using the inverse convolution layer with a step of 2 to perform 6 times of upsampling; after each upsampling, add the encoded information with the same size in 41) respectively as the input of the next upsampling, and finally obtain the intermediate image .

[0040] 46) Through generate the defogging image J(x).

[0041] Further, the loss function of the defogging network model in step 5 is the weighted sum of the similarity loss and the natural image quality evaluation loss, that is,

[0042] Among them, represent the weight of L SSIM , the similarity loss, the natural image quality evaluation loss.

[0043] Further, the definition is as follows:​

[0044]

[0045] wherein, is the average value of all pixels of the two images, , is the variance of the image pixel values.

[0046] Further, is defined as follows:

[0047]

[0048] wherein represents the no-reference image to be detected, is a pre-selected natural image, represents the feature mean value of the Gaussian distribution established on the natural image, represents the covariance matrix thereof; represents the feature mean value of the Gaussian distribution established on the image to be detected, represents the covariance matrix thereof.

[0049] Another aspect of the present application provides an image defogging system based on depth information and a fogging and Transformer network, comprising the following modules:

[0050] A first data set construction module is configured to perform fogging processing on an original no-fog image containing depth information according to an atmospheric scattering model, and to crop the original no-fog image containing depth information and the fogging-processed image to obtain a pair of foggy and no-fog image segments, thereby forming a first paired foggy image training set α ;

[0051] A second data set construction module is configured to estimate the depth information image corresponding to a single original no-fog image not containing depth information using a depth estimation method, and then perform fogging processing on the original no-fog image not containing depth information, and crop the original no-fog image not containing depth information and the fogging-processed image to obtain a pair of foggy and no-fog image segments, thereby forming a second paired foggy image training set β ,

[0052] A third data set construction module is configured to perform random fogging on a single original no-fog image not containing depth information using a method of adding salt and pepper noise, and then crop the original no-fog image not containing depth information and the random fogging-processed image to obtain a pair of foggy and no-fog image segments, thereby forming a third paired foggy image training set δ ;

[0053] A model branch construction module is configured to construct a feature concatenation module and an encoding-branch decoding module, the feature concatenation module is configured to extract high-dimensional features of an input image, and the encoding-branch decoding module is configured to reconstruct the extracted high-dimensional features to obtain a generated haze-free image.

[0054] A haze removal network model construction module is configured to construct a haze removal network model and a corresponding loss function, and train the haze removal network model by using a constructed training set, the haze removal network model comprises the feature concatenation module and the encoding-branch decoding module.

[0055] An image haze removal module is configured to complete image haze removal by using the trained haze removal network model.

[0056] Compared with the prior art, the method has the advantages and beneficial effects that: the haze adding mode containing depth information is combined with the randomly generated haze adding mode, the robustness for different foggy scenes is enhanced; the module containing the Transformer is introduced, the extraction ability of long-range association in the image and the adaptability to non-uniform foggy scenes are enhanced; the loss function containing the no-reference indicator is introduced, and the visual effect of the generated haze-free image is more natural. The image haze removal network framework proposed in the method can be deployed on a local computing device and can be used in image haze removal preprocessing operations in various image processing fields, and has obvious advantages in target detection tasks, image classification tasks and other analysis applications. A large number of experiments on mainstream data sets show that the proposed method achieves the state-of-the-art (SOTA) result. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 A haze removal data generation diagram.

[0058] Figure 2 A haze removal network structure diagram.

[0059] Figure 3 A result image generated by the method on the RESIDE:V0 data set.

[0060] Figure 4 A qualitative comparison of the method with other three algorithms in the task.

[0061] Figure 5 Network structure diagrams of an encoder module, a decoder module 1 and a decoder module 2. DETAILED DESCRIPTION

[0062] To solve the problems in the prior art, the method provided in the application is a haze removal network method closer to a real environment, which is based on image depth information and random haze adding to form haze image data required for training.

[0063] The application provides an image defogging method based on depth-of-field information and a fogging and Transform network, and comprises the following steps.

[0064] Step 1: introduce a public image dataset containing depth information, add a fog layer to the public image dataset according to an atmospheric scattering model and corresponding depth information images with time-varying fogging parameters, and then cut a random part of an image segment from each complete original image in the public image dataset with a uniform window size, and cut an image segment of the same size from the corresponding area of the fog-added image. A pair of training samples is constructed through the cut image segment in the original image and the fog-added image segment, and a first paired foggy image training set is constructed through a plurality of training samples ;

[0065] Step 1: the paired foggy image dataset is (Adata k (x,y), Bdata k (x,y))

[0066] k∈[1,L],x∈[1,M],y∈[1,N]

[0067] Wherein, L is the number of foggy image segments in the paired foggy image training set, Adata k (x,y) is the pixel of the xth row and yth column of the kth fog-free image segment in the paired foggy image training set, Bdata k (x,y) is the pixel of the xth row and yth column of the kth foggy image segment in the paired foggy image training set, M is the number of rows of the window used for cutting, and N is the number of columns of the window used for cutting.

[0068] Step 2: introduce a fog-free image dataset not containing depth information, estimate the corresponding depth information image of each original image in the fog-free image dataset using a depth estimation method, and obtain a pair of cut image segments in the original image and the corresponding fog-added image segment training sample in the same fogging and cutting manner as step 1, and finally construct a second paired foggy image training set through a plurality of training samples ;

[0069] Step 2: the paired foggy image dataset is (Cdata k (x,y), Ddata k (x,y))

[0070] k∈[1,L],x∈[1,M],y∈[1,N]

[0071] Wherein, L is the number of foggy image segments in the paired foggy image training set, Cdata k(x,y) is the pixel at the xth row and yth column of the kth haze-free image segment in the haze-free image dataset, and Ddata k (x,y) is the pixel at the xth row and yth column of the kth haze-free image segment in the haze-free image dataset, and Ddata

[0072] The atmospheric scattering model used in steps 1 and 2 is:

[0073]

[0074] where x represents the position of a pixel point, represents the atmospheric light value at infinity, is the haze image obtained by the detection system, and represents a haze-free image, is a transmission function, representing the degree of light attenuation, where is the depth information at the point, and represents the scattering coefficient of the medium in which the light propagates.

[0075] Step 3: Introduce a haze-free image dataset that does not contain depth information, add random salt and pepper noise to each original image in the haze-free image dataset as a haze layer to form a haze, and then obtain a pair of cropped image segments and corresponding haze-added image segments in the training samples by the same cropping method as step 1, and construct a third paired haze image training set by multiple training samples ;

[0076] The paired haze image dataset in step 3 is: (Edata k (x,y), Fdata k (x,y))

[0077] k∈[1,L],x∈[1,M],y∈[1,N]

[0078] where L is the number of haze image segments in the paired haze image training set, Edata k (x,y) is the pixel at the xth row and yth column of the kth haze-free image segment in the haze-free image dataset, and Ddata k (x,y) is the pixel at the xth row and yth column of the kth haze-free image segment in the haze-free image dataset, and Ddata

[0079] Step 4: Train the Feature Concatenation Block (FCB) module and the Encoder-Parallel Decoder Block (EPDB);

[0080] Merge the haze-free image data parts Adata k (x,y), Cdata k (x,y), Edata k (x,y) to obtain Gdata n (x,y), the haze image data part Bdata k (x,y), Ddata k (x,y), Fdata k (x,y) to obtain Hdata n (x,y).

[0081] Wherein, subscript n represents the sum of the corresponding data sample numbers of Adata k、 Cdata k、 Edata k , x∈[1,M], y∈[1,N], M is the row number of the window used for cropping, and N is the column number of the window used for cropping;

[0082] The process of training the FCB module is as follows:

[0083] 1) A convolutional layer with a step size of 1 and a kernel size of 1 1 is used to extract features from the haze source image ; wherein H is one of the images in Hdata n (x,y).

[0084] 2) A ReLU layer is used to process the output of step 1) to prevent network overfitting;

[0085] 3) A convolutional layer with a step size of 1 and a kernel size of 3 3 is used to extract features from the output of 2);

[0086] 4) A ReLU layer is used to process the output of step 3) to prevent network overfitting;

[0087] 5) After concatenating the outputs of 2) and 4), a convolutional layer with a step size of 1 and a kernel size of 5 5 is used to extract features from it;

[0088] 6) A ReLU layer is used to process the output of step 5) to prevent network overfitting.

[0089] 7) After concatenating the outputs of 4) and 6), a convolutional layer with kernel size 7 and stride 1 is used 7) A convolutional layer is used to extract features from the output of 6).

[0090] 8) A ReLU layer is used to process the output of 7) to prevent overfitting of the network.

[0091] 9) After concatenating the outputs of 2), 4), 6) and 8), a convolutional layer with kernel size 3 and stride 1 is used 3) A convolutional layer is used to extract features from the output of 2).

[0092] 10) The output of 9) is concatenated with the hazy image H in 1) to obtain the extracted high-dimensional features.

[0093] The steps for training the EPDB module are as follows:

[0094] 1) The output of the FCB module is used as input to the Encoder module composed of Transformer, which encodes the image feature information and outputs four different sizes of encoded information.

[0095] 2) The output of 1) is used as input to the Decoder module 1 composed of Transformer, which decodes the image feature information and outputs the fifth size of decoded information.

[0096] 3) The output of 2) is used as input to the inverse convolutional layer with a stride of 2 for 6 times of upsampling; after each upsampling, the same size of encoded information in S1) is added respectively as the input of the next upsampling, and finally the intermediate image is obtained .

[0097] 4) The output of 1) is used as input to the Decoder module 2 composed of Transformer, which decodes the image feature information.

[0098] The Encoder module, the Decoder module 1, and the Decoder module 2 each include one or more stacked Transformer Blocks and parallel Intra Patch Transformer Blocks. The specific structure of the three modules can be found in Valanarasu J M J, Yasarla R, Patel V M. Transweather: Transformer-based restoration of images degraded by adverse weather conditions [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:2353-2363.

[0099] 5) Take the output of 4) as input, use the inverse convolution layer with a step of 2 to perform 6 times of upsampling; after each upsampling, add the encoded information with the same size in 1) respectively to obtain the input of the next upsampling, and finally obtain the intermediate image ;

[0100] 6) Generate the defogging image J(x) by ;

[0101] Step 5: Construct a defogging network model, input into the defogging network model in turn, and obtain a result defogging image J(x) by predicting through the defogging network model; construct a defogging network model loss function according to the predicted result defogging image J(x) and one of the images in Gdata n (x,y) in the foregoing step 4, i.e. the haze-free image G as a label, and further optimize and train to obtain a trained defogging network model;

[0102] The defogging network model in step 5 includes a feature concatenation module (FCB) and an encoder-parallel decoder block (EPDB).

[0103] The defogging network model loss function in step 5 is composed of an SSIM loss function and an NIQE loss function.

[0104]

[0105] wherein, represents the similarity loss of the defogging network, and the loss function is defined as follows:

[0106]

[0107] wherein, is the average value of all pixels of a haze-free image and the corresponding haze image in the training set, , is the variance of the pixel values of the two images;

[0108] denotes the natural image quality evaluation of the haze removal network, and the loss function is defined as follows:

[0109]

[0110] wherein denotes the no-reference image to be detected, is a natural image selected in advance, denotes the feature mean value of the Gaussian distribution established on the natural image, denotes the covariance matrix thereof; denotes the feature mean value of the Gaussian distribution established on the image to be detected, denotes the covariance matrix thereof; denotes the weight of L SSIM ;

[0111] Step 6: finally, the trained haze removal network is used to complete image haze removal.

[0112] Another aspect of the present application provides an image haze removal system based on depth-of-field information and a haze and Transformer network, comprising the following modules:

[0113] A first data set construction module is configured to perform haze processing on an original haze-free image containing depth information according to an atmospheric scattering model, and to crop the original haze-free image containing depth information and the haze-processed image to obtain a pair of haze and haze-free image segments, thereby forming a first pair of haze image training set α ;

[0114] A second data set construction module is configured to estimate the depth information image corresponding to a single original haze-free image not containing depth information using a depth estimation method, then perform haze processing on the original haze-free image not containing depth information, and crop the original haze-free image not containing depth information and the haze-processed image to obtain a pair of haze and haze-free image segments, thereby forming a second pair of haze image training set β ,

[0115] The third data set construction module is configured to add pepper and salt noise to a single original non-fog image without depth information to generate a random fog image, and then crop the original non-fog image without depth information and the random fog image to obtain a pair of fog and non-fog image segments, thereby forming a third pair of fog image training sets δ ;

[0116] The model branch construction module is configured to construct a feature concatenation module and an encoding-branch decoding module, the feature concatenation module is configured to extract high-dimensional features of an input image, and the encoding-branch decoding module is configured to reconstruct the extracted high-dimensional features to obtain a generated non-fog image.

[0117] The dehazing network model construction module is configured to construct a dehazing network model and a corresponding loss function, and train the dehazing network model by using the constructed training set, the dehazing network model comprises the feature concatenation module and the encoding-branch decoding module.

[0118] The image dehazing module is configured to complete image dehazing by using the trained dehazing network model.

[0119] The specific implementation modes of the modules and the corresponding steps are not described herein.

[0120] Application test: the application can be applied to a known foggy scene image, and a corresponding dehazed image of the foggy scene image can be obtained by using the network model described herein; the application can also be used to generate scene non-fog images in the field of public security and create artistic works in the field of cultural entertainment. The application is tested on the public data set RESIDE:V0 data set, and a realistic dehazed image can be obtained.

[0121] It should be understood that the above specific embodiments of the application are only used for illustrative or explanatory purposes, and do not constitute a limitation on the application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the application shall be included in the protection scope of the application. In addition, the appended claims of the application are intended to cover all variations and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

Claims

1. An image defogging method based on depth information fogging and a Transformer network, characterized in that, Comprising the following steps: Step 1, fogging processing is performed on the original fog-free image containing depth information according to an atmospheric scattering model, and the original fog-free image containing depth information and the fogging-processed image are cropped to obtain a pair of foggy and fog-free image segments, thereby forming a first paired foggy image training set α ; Step 2, estimate the depth information image corresponding to the original haze-free image pair without depth information using the method of depth estimation, then haze processing is performed on the original haze-free image without depth information, and the original haze-free image without depth information and the image after haze processing are cropped to obtain a pair of foggy and haze-free image segments, thereby forming a second paired foggy image training set β , Step 3, randomly fogging the original fog-free image without depth information using the method of adding pepper and salt noise, and then cropping the original fog-free image without depth information and the image after random fogging to obtain paired foggy and fog-free image segments, forming a third paired foggy image training set δ ; Step 4, constructing a feature concatenation module for extracting high-dimensional features of the input image and an encoding-branch decoding module for reconstructing the extracted high-dimensional features to obtain a generated haze-free image; Step 5, constructing a haze removal network model and a corresponding loss function, training the haze removal network model using the training set in steps 1-3, wherein the haze removal network model comprises the feature concatenation module and the encoding-branch decoding module; The loss function of the dehazing network model in step 5 is a weighted sum of the similarity loss and the natural image quality evaluation loss, i.e. wherein, represents L SSIM the weight of, represents the similarity loss, represents the natural image quality evaluation loss; The definitions are as follows: wherein, is the average value of all pixels of a fog-free image and the corresponding foggy image in the training set, , is the variance of the pixel values of the two images; The definitions are as follows: wherein represent the natural image to be detected, is a natural image selected beforehand, represent the feature mean of the Gaussian distribution established on the natural image, represent the covariance matrix thereof; represent the feature mean of the Gaussian distribution established on the natural image, represent the covariance matrix thereof; Step 6, completing image haze removal using the trained haze removal network model.

2. The image defogging method based on depth information fogging and the Transform network according to claim 1, wherein: The specific implementation of step S1 is as follows: The public image dataset containing depth information is introduced, the public image dataset is added with a fog layer according to an atmospheric scattering model and corresponding depth information map, with time-varying random fog parameters, a random part of image segments is cropped from each complete original image in the public image dataset with a uniform window size, image segments of the same size are cropped from the corresponding area of the fog-added image, a pair of training samples is constructed through the cropped image segments in the original image and the fog-added image segments, and a first paired foggy image training set is constructed through multiple training samples α .

3. The image defogging method based on depth information fogging and the Transformer network according to claim 2, wherein: The atmospheric scattering model is: where x denotes the position of the pixel, denotes the atmospheric light value at infinity, i.e. the foggy image obtained by the detection system, while denotes the fog-free image, is the transmission function, which denotes the degree of attenuation of the light, where is the depth information at that point, while denotes the scattering coefficient of the medium in which the light propagates.

4. The image defogging method based on depth information fogging and the Transformer network according to claim 1, wherein: The specific implementation of constructing the feature concatenation module in step 4 is as follows: The first paired foggy image data set described in step 1 is: (Adata k (x,y), Bdata k (x,y)) The second paired foggy image data set described in step 2 is: (Cdata k (x,y), Ddata k (x,y)) The third paired foggy image data set described in step 3 is: (Edata k (x,y), Fdata k (x,y)) k∈[1,L], x∈[1,M], y∈[1,N] The fog-free image data parts Adata k (x,y), Cdata k (x,y), Edata k (x,y) in steps 1, 2, and 3 are merged to obtain Gdata n (x,y) The foggy image data parts Bdata k (x,y), Ddata k (x,y), Fdata k (x,y) are merged to obtain Hdata n (x,y) wherein, L is the number of foggy image segments in the paired foggy image training set, Adata k (x, y) is the pixel at the xth row and yth column of the kth fog-free image segment in the paired foggy image training set, Bdata k (x, y) is the pixel at the xth row and yth column of the kth foggy image segment in the paired foggy image training set, M is the number of rows of the window used for cropping, N is the number of columns of the window used for cropping; subscript n is the sum of the data sample numbers corresponding to Adatak, Cdatak, and Edatak The specific processing process of the feature concatenation module is as follows: 1) using a stride of 1 and a kernel size of 1 1 of the convolutional layer of the fog source image performing feature extraction; wherein H is one of the images in Hdata n (x, y). 2) The output of step 1) is processed using a ReLU layer to prevent network overfitting; 3) using a stride of 1 and a kernel size of 3 the output of the convolutional layer of 3) performs feature extraction on the output of 2); 4) The output of step 3) is processed using a ReLU layer to prevent network overfitting; 5) after concatenating the outputs of 2) and 4), a convolutional layer with step size 1 and kernel size 5 5 the convolutional layer of which performs feature extraction; 6) The output of step 5) is processed using a ReLU layer to prevent network overfitting; 7) after concatenating the outputs of 4) and 6), a convolutional layer with step size 1 and kernel size 7 7, which extracts features from it; 8) The output of step 7) is processed using a ReLU layer to prevent network overfitting; 9) after concatenating the outputs of 2), 4), 6) and 8), a convolutional layer with a step size of 1 and a kernel size of 3 is applied 3 is applied to extract features from it; 10) The output of 9) is concatenated with the haze image H corresponding to the input in 1) to obtain the extracted high-dimensional features.

5. The image defogging method based on depth information fogging and the Transformer network according to claim 4, wherein: The specific implementation of the encoding-branch decoding module in step 4 is as follows: 41) Using the output of the feature concatenation module as input, using an Encoder module composed of Transformer to encode the image feature information, outputting encoding information containing 4 different sizes; 42) Using the output of 41) as input, using a Decoder module 1 composed of Transformer to decode the image feature information, outputting decoding information of the 5th size; 43) Using the output of 42) as input, using an inverse convolution layer with a step of 2 for 6 times of upsampling; After each up-sampling, add the encoded information of the same size in S1) respectively, as the input of the next up-sampling, and finally obtain the intermediate image ; 44) Using the output of 41) as input, using a Decoder module 2 composed of Transformer to decode the image feature information; Wherein, the Encoder module, the Decoder module 1 and the Decoder module 2 all contain one or more layers of stacked Transformer Block and parallel Intra Patch Transformer Block; 45) take the output of 44) as input, and perform 6 times upsampling with inverse convolution layer with step size 2; after each upsampling, add the encoded information with the same size in 41) respectively as the input of the next upsampling, and finally get the intermediate image ; 46) by A dehazed image J(x) is generated.

6. An image defogging system based on depth information and a Transformer network, used to implement an image defogging method based on depth information and a Transformer network according to any one of claims 1-5, characterized in that, Comprising the following modules: The first data set construction module is configured to perform fogging processing on the original fog-free image containing depth information according to an atmospheric scattering model, and to crop the original fog-free image containing depth information and the fogging-processed image, so as to obtain a pair of foggy and fog-free image segments, thereby forming a first pair of foggy image training set α ; The second data set construction module is configured to estimate the depth information image corresponding to the original non-fog image without depth information using a depth estimation method, then perform fogging processing on the original non-fog image without depth information, and crop the original non-fog image without depth information and the image after the fogging processing to obtain a pair of foggy and non-foggy image segments, thereby forming the second pair of foggy image training set β , The third data set construction module is configured to add pepper and salt noise to a single original non-fog image without depth information to generate a random fog image, and then crop the original non-fog image without depth information and the random fog image to obtain a pair of fog and non-fog image segments, thereby forming a third pair of fog image training sets δ ; The model branch construction module is used to construct a feature concatenation module for extracting high-dimensional features of the input image and an encoding-branch decoding module for reconstructing the extracted high-dimensional features to obtain a generated haze-free image; The haze removal network model construction module is used to construct a haze removal network model and a corresponding loss function, and train the haze removal network model using the constructed training set, wherein the haze removal network model comprises the feature concatenation module and the encoding-branch decoding module; The image haze removal module is used to complete image haze removal using the trained haze removal network model.

Images