Image defogging method and device based on Transformer structure integrating near infrared

Abstract

The present invention discloses an image defogging method and device of a Transformer structure fused with near-infrared. The method can utilize additional near-infrared features as supplementary information and adopts a Transformer deep neural network structure, which has stronger feature extraction capability and can generate defogging images with high resolution, high fidelity and prominent texture details. Near-infrared images and visible light images are taken for the same scene to form a data set. The near-infrared image and the visible light image are input into a pre-trained Transformer structure image defogging model; the model encodes and represents the visible light and near-infrared images, and obtains feature vectors of the visible light and near-infrared images respectively, and the two are fused to obtain an interactive feature vector; the feature vectors of the visible light and near-infrared images are decoded respectively to obtain visible light and near-infrared image sequences, and the two are merged with the interactive feature vector after processing, and the merged result is channel reorganized and convolved to output the defogging image result.

Description

Technical Field

[0001] The present invention belongs to the technical field of digital image processing, and specifically relates to an image defogging method and device with a Transformer structure fused with near-infrared, which is suitable for early preprocessing of computer vision applications and can be widely used in the fields of traffic target detection, road monitoring, etc. Background Art

[0002] In foggy weather, visibility is low, and the images and videos taken often have degraded quality. In order to improve the quality and clarity of images, it is necessary to perform defogging on the images.

[0003] Single image dehazing aims to generate a fog-free image from a blurred image. It is a classic image processing problem and has been an important research topic in the field of vision in the past decade. Dehazing algorithms can be divided into two categories: traditional dehazing methods and deep learning-based methods. Traditional methods are mainly divided into two categories: image enhancement and image restoration. Image enhancement methods highlight image details and enhance contrast. They have a wide range of applications and can effectively improve the contrast of foggy images. The dehazing algorithm based on image restoration studies the physical process of foggy image degradation. It observes and summarizes a large number of foggy and fog-free images, inverts the degradation process, and obtains an estimate of the clear image.

[0004] Traditional algorithms have good effects on fog image restoration in certain scenarios, but their scope of application is narrow. They are not fine enough in restoring image details, and the calculations are complex, which cannot meet the real-time processing requirements needed in many places today. There are two methods based on deep learning. One is to use physical models or manual features combined with neural networks, and the other is an end-to-end method that does not include physical models. These technologies require mapping of foggy images and their corresponding real clear images to train the model. Among them, the traditional convolutional neural network algorithm still has poor generalization ability and cannot correct noise and distortion.

[0005] With the vigorous development of attention mechanism, the self-attention mechanism Transformer structure has achieved good results in visual fields such as image processing and object detection. How to use the self-attention mechanism Transformer structure to dehaze images to obtain dehazed images with high resolution, high fidelity and rich texture details is a problem that has not been solved by existing technologies. Summary of the invention

[0006] In view of this, the present invention provides an image defogging method and device with a Transformer structure that integrates near-infrared, which can utilize additional near-infrared features as supplementary information. At the same time, it adopts a Transformer deep neural network structure, which has stronger feature extraction capabilities than traditional convolutional neural networks and generates defogging images with high resolution, high fidelity and prominent texture details.

[0007] The technical solution for implementing the present invention is as follows: an image defogging method integrating a near-infrared Transformer structure, characterized in that it comprises the following steps:

[0008] S1: Near-infrared images and visible light images are taken for the same scene to form a data set.

[0009] S2: Both the near-infrared image and the visible light image are input into the pre-trained Transformer structure image dehazing model; the Transformer structure image dehazing model first encodes and represents the visible light image and the near-infrared image to obtain the feature vector of the visible light image and the feature vector of the near-infrared image respectively, and then fuses the two to obtain the interactive feature vector; then the feature vector of the visible light image and the feature vector of the near-infrared image are decoded respectively to obtain the visible light image sequence and the near-infrared image sequence, which are merged with the interactive feature vector after processing, and then the merged result is channel reorganized and convolved to output the dehazed image result.

[0010] Furthermore, the Transformer structured image dehazing model includes a visible light image feature encoding module, a near infrared image feature encoding module, a feature interaction module, a visible light feature decoding module, a near infrared feature decoding module and a feature fusion module.

[0011] The processes executed by each module are as follows:

[0012] The visible light image feature encoding module represents the visible light image and obtains three groups of visible light image feature vectors according to the RGB channels. The visible light image feature encoding module first includes a downsampling convolution layer and a global average pooling layer, followed by three Transformer encoding units connected in series.

[0013] The near-infrared image feature encoding module characterizes the near-infrared image and obtains a set of feature vectors of the near-infrared image. The encoding module first includes a downsampling convolution layer and a global average pooling layer, and then two Transformer encoding units are serially connected.

[0014] In the feature interaction module, the feature vector of the visible light image and the feature vector of the near-infrared image are added together through feature maps and then passed through the deconvolution layer and the ReLU activation layer to obtain the interactive feature vector.

[0015] The visible light feature decoding module converts three sets of visible light image feature vectors into a visible light image sequence. The visible light feature decoding module contains three Transformer encoding units and an upsampling convolutional layer.

[0016] The near-infrared feature decoding module converts a set of near-infrared image feature vectors into a near-infrared image sequence. The near-infrared feature decoding module contains two Transformer encoding units and an upsampling convolutional layer.

[0017] In the feature fusion module, the visible light image sequence is globally averaged pooled and then a 3*3 convolution is performed, and the near-infrared image sequence is convolved twice with a 1*1 convolution. The results of the two-step operation are merged with the interactive feature vector by adding the feature map, and then the merged result is reorganized through channels. Finally, a 5*5 depth-separable convolution and a size-adjusted convolution are performed to obtain a clear color image after dehazing.

[0018] Furthermore, after outputting the dehazed image result, the dehazed image result is calibrated using an image discriminator; the image discriminator uses two block structures including a convolution layer-normalization layer-ReLU activation layer-pooling layer, and then verifies the deep information of the image by mixing channel attention with spatial attention, and finally determines the true or false probability of the image through the Softmax function.

[0019] Furthermore, the pre-trained Transformer structure image dehazing model is pre-trained in the following way:

[0020] Three types of losses are used to constrain the training of the Transformer structure image dehazing model. The first loss is Charbonnier loss.

[0021] The second loss is perceptual loss, which uses the feature map output by the pre-trained VGG16 network at the 14th layer to calculate the loss;

[0022] The third loss is the true or false probability loss obtained by the image discriminator.

[0023] Another embodiment of the present invention further provides an image defogging device with a Transformer structure fused with near-infrared, the device comprising a data acquisition module and a Transformer structure image defogging model module.

[0024] The data acquisition module is used to capture and acquire visible light images and near infrared images corresponding to the same scene.

[0025] Transformer structure image dehazing model module, with visible light image and near-infrared image, Transformer structure image dehazing model first encodes and represents the visible light image and the near-infrared image, respectively, and obtains the feature vector of the visible light image and the feature vector of the near-infrared image, and then fuses the two to obtain the interactive feature vector; then decode the feature vector of the visible light image and the feature vector of the near-infrared image respectively, and obtain the visible light image sequence and the near-infrared image sequence. After processing, the two are merged with the interactive feature vector, and then the merged result is channel reorganized and convolved to output the dehazed image result.

[0026] Preferably, the Transformer structure image dehazing model includes a visible light image feature encoding module, a near infrared image feature encoding module, a feature interaction module, a visible light feature decoding module, a near infrared feature decoding module, and a feature fusion module.

[0027] The visible light image feature encoding module represents the visible light image and obtains three sets of visible light image feature vectors according to the RGB channels. The encoding module first includes a downsampling convolution layer and a global average pooling layer, followed by three Transformer encoding units connected in series.

[0028] The near-infrared image feature encoding module characterizes the near-infrared image and obtains a set of feature vectors of the near-infrared image. The encoding module first includes a downsampling convolution layer and a global average pooling layer, and then two Transformer encoding units are serially connected.

[0029] In the feature interaction module, the feature vector of the visible light image and the feature vector of the near-infrared image are added together through feature maps and then passed through the deconvolution layer and the ReLU activation layer to obtain the interactive feature vector.

[0030] The visible light feature decoding module converts three sets of visible light image feature vectors into a visible light image sequence. The decoding module contains three Transformer encoding units and an upsampling convolutional layer.

[0031] The near-infrared feature decoding module converts a set of near-infrared image feature vectors into a near-infrared image sequence. The decoding module contains two Transformer encoding units and an upsampling convolutional layer.

[0032] In the feature fusion module, the visible light image sequence is globally averaged pooled and then a 3*3 convolution is performed, and the near-infrared image sequence is convolved twice with a 1*1 convolution. The results of the two-step operation are merged with the interactive feature vector by adding the feature map, and then the merged result is reorganized through channels. Finally, a 5*5 depth-separable convolution and a size-adjusted convolution are performed to obtain a clear color image after dehazing.

[0033] Beneficial effects:

[0034] 1. The image defogging method of the Transformer structure fused with near-infrared provided by the present invention adopts the Transformer structure image defogging model for defogging. The Transformer structure solves the problem that the local receptive field of traditional convolution is small and the details are lost in the deep layer. The use of near-infrared images can supplement the texture information of visible light images. Compared with the traditional method of single image defogging, it solves the problem of near-infrared and visible light image fusion and visible light image defogging at the same time.

[0035] 2. Compared with the traditional loss, the training strategy proposed by the present invention when training the Transformer structure image dehazing model can ensure that the gradient does not disappear and the generated image is more realistic. The proposed method has received better evaluation on the data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the process of an image defogging method using a Transformer structure fused with near infrared provided by an embodiment of the present invention;

[0037] Figure 2 It is a block diagram of the Transformer structure image defogging model provided by an embodiment of the present invention;

[0038] Figure 3 A block diagram of the image identifier provided by an embodiment of the present invention;

[0039] Figure 4 It is a module diagram of the coding unit in the Transformer structure image defogging model provided by an embodiment of the present invention;

[0040] Figure 5 The present invention provides a flowchart of using an electronic device. DETAILED DESCRIPTION

[0041] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0042] Embodiment 1:

[0043] This embodiment provides an image dehazing method using a Transformer structure that integrates near-infrared. The present invention uses near-infrared images to supplement the texture information of visible light images. Visible light images have the characteristics of rich colors and strong authenticity. Although near-infrared images have low resolution, they have strong penetration and can be processed by machine vision faster. Near-infrared imaging technology has strong anti-interference capabilities and can provide rich target information and background information. Therefore, the goal of fusing near-infrared images with visible light images is not only to retain the rich visible light image detail information, but also to increase the contrast between the target and the background to facilitate human vision and image processing. This method is as follows Figure 1 As shown, it is specifically implemented by the following steps:

[0044] S1. Take near-infrared images and visible light images of the same scene to form a data set.

[0045] The infrared image and the visible light image are input into the constructed Transformer structure image dehazing model accordingly.

[0046] S2. Obtain a clear color image result after defogging based on paired near-infrared image data and the Transformer structure image defogging model. In the embodiment of the present invention, the Transformer structure image defogging model includes a visible light image feature encoding module, a near-infrared image feature encoding module, a feature interaction module, a visible light feature decoding module, a near-infrared feature decoding module, and a feature fusion module; its components are as follows: Figure 2 shown.

[0047] Step 2 may include the following steps S21-S26;

[0048] S21. The visible light image is characterized by a visible light image feature encoding module, and three sets of visible light image feature vectors are obtained according to three different channels. The encoding module first includes a downsampling convolution layer and a global average pooling layer, followed by three Transformer encoding units connected in series; the Transformer encoding unit structure is as follows Figure 4 shown.

[0049] Among them, the visible light image is characterized, and three groups of feature vectors of visible light images are obtained according to three different channels, including:

[0050] S211.1 uses the Python image processing library to separate the three color channels of the visible light image and obtain the R, G, and B matrices;

[0051] S211.2 sends the R, G, and B matrices to the Transformer encoding unit respectively. Taking a set of matrices as an example, the matrix is ​​first expanded by reflection to obtain the matrix X, and then passes through two linear connection layers to calculate the query vector Q, key vector K, and value vector V of the matrix respectively. According to the division of fixed parameters, Q and K can be divided into multiple groups of Q i and K i , Q i and K i The transpose of is multiplied and normalized and then added to V i Multiply, output Z i The formula is as follows:

[0052]

[0053] Among them, d k It means Q i ,K i The vector dimension of , softmax is a normalized exponential function.

[0054] Furthermore, the output Z i They are spliced ​​together to form a matrix Z, which is the output of the multi-head attention, and then spliced ​​with the matrix X' after the convolution layer, and then passed through a residual structure composed of a linear connection layer and a multi-layer perceptron, and finally output Y, which is the result of the Transformer encoding unit for different channel matrices.

[0055] The final output vector Y obtained by the three R, G, and B matrices R , Y G , Y B These are three image feature vectors that incorporate the self-attention mechanism, representing the information content worth transmitting on different channels and are sent to the subsequent decoding module.

[0056] S22, characterize the near infrared image through the near infrared image feature encoding module to obtain a set of feature vectors of the near infrared image. The encoding module first includes a downsampling convolution layer and a global average pooling layer, and then connects two Transformer encoding units in series. The encoding unit is the same as S311.2, and the output Y is obtained. NIR , is the result after the near-infrared channel feature matrix passes through the Transformer encoding unit.

[0057] S23, through the feature interaction module, in order to deeply fuse the features of the two images at the feature level, using their complementarity, the encoding output vector Y of the visible light image R , Y G , Y B and the encoded output vector Y of the near infrared image NIRThe interactive feature vector is obtained by adding the feature maps and then passing through the deconvolution layer and the ReLU activation layer. The specific expression is as follows:

[0058] o=ReLU(Deconv(Y R +Y G +Y B +Y NIR ))

[0059] Deconv represents the deconvolution operation, and ReLU represents the numerical activation. The resulting vector o represents the fusion of the visible light and near-infrared feature vectors in the encoding stage, which will be sent to the subsequent feature fusion module to supplement the feature relationships obtained in each stage.

[0060] S24, through the visible light feature decoding module, the three groups of visible light images are encoded and output as vectors Y R , Y G , Y B Converted into a visible light image sequence M RGB The decoding module contains three Transformer encoding units and an upsampling convolutional layer. The Transformer unit in the decoder of the present invention is designed the same as the Transformer unit in the encoder, and the two use the same number of attention heads and encoder settings. The output of the decoder is consistent with the image dimension of the original input. During the training process, the Charbonnier function is used as the loss function, and the network weights are updated by calculating the loss between the decoder output and the encoder input. After the model converges, the output of the visible light feature decoder will be stored as M RGB .

[0061] S25, through the near infrared feature decoding module, a set of near infrared image feature vectors is converted into a near infrared image sequence M NIR , the decoding module contains two Transformer encoding units and an upsampling convolution layer. The encoding unit is the same as that in S34. The output of near-infrared feature decoding will be stored as M NIR ;

[0062] S26, through the feature fusion module, the visible light image sequence M RGB Perform global average pooling and then 3*3 convolution to obtain the near-infrared image sequence M NIR Perform two 1*1 convolutions, merge the results of the two-step operation with the interactive feature vector o by adding the feature map, and then reorganize the channels of the merged result. Finally, perform 5*5 depth-separable convolution and size-adjusted convolution to obtain a clear color image after dehazing. The expression is:

[0063] Middle1=Conv3*3 GAP(M RGB )+Conv 1*1 (Conv 1*1 (M NIR ))+o

[0064] Middle2=ChannelShuffle(Middle1)

[0065] O = ResizeConv(depthConv 5*5 (Middle2))

[0066] Among them, GAP represents global average pooling, ChannelShuffle represents the channel reorganization strategy, which can fully capture the information of the vector, ResizeConv represents the size adjustment convolution, eliminates the checkerboard effect that appears in the generated image, and finally outputs the image O.

[0067] S3, the generated clear color image result O after dehazing needs to be calibrated by the image discriminator, which uses two block structures including convolution layer-normalization layer-ReLU activation layer-pooling layer, and then verifies the deep information of the image by mixing channel attention with spatial attention, and finally judges the true and false probability of the image by the softmax function. The image discriminator is composed of Figure 3 shown.

[0068] In the embodiment of the present invention, the training process of the Transformer structure image defogging model is also included using three loss constraints. The first loss is Charbonnier loss, and the specific formula is: Where y represents the clear image estimated by the network, y' represents the real clear image, and ε represents a constant to ensure that L c The case where it is not zero and the training terminates unexpectedly;

[0069] The second type of loss is perceptual loss, which uses the feature map output by the pre-trained VGG16 network at the 14th layer to calculate the loss. The specific formula is: L per =||φ(y)-φ(y')||, where φ(z) represents the feature map of the 14th layer of the VGG16 network;

[0070] The third loss is the true and false probability loss obtained by the image discriminator. The specific formula is: Where N represents the number of images, D refers to the image discriminator, and G refers to the Transformer structure image dehazing model.

[0071] In the embodiment of the present invention, the training samples are 280 groups of reference RGB images, foggy RGB images and near-infrared images collected. The three types of images are all shot with A7C, the reference RGB images and foggy RGB images are shot with B+W486 (RGB) filters, and the near-infrared images are shot with 093 (near-infrared) filters. All three images are shot at fixed points, that is, the foggy RGB images and near-infrared images are shot at a fixed location on a foggy day, and the reference RGB images are shot at a fixed location on a sunny day. Later, the three images are aligned by similarity transformation. For each group of samples, foggy RGB images and near-infrared images are used as input, and the reference RGB images are used as labels. Due to the small amount of data, data enhancement techniques are used to expand the data set, such as affine transformation, horizontal flipping, etc. The data set is expanded to 400 groups.

[0072] In the training experiment, 320 groups were divided into training sets in the data set, and the remaining 80 groups were test sets. The PyTorch 1.10 framework was used. Adam was selected as the optimizer, the learning rate was set to 0.0001, the batch size was set to 4, and the number of iterations was 400. The foggy RGB image and the near-infrared image were input into the Transformer structure, and the output defogging RGB image was obtained after calculation. The defogging RGB image was compared with the corresponding reference RGB image of the same group, and the Charbonnier loss, perceptual loss, and true and false probability loss were calculated. The loss function was derived for each parameter to obtain the gradient, and the gradient descent method was used to update the network parameters in the Transformer structure. The foggy image and the near-infrared image were input again and iterated until the loss function converged and the number of iterations was zero to end the training.

[0073] Compared with traditional losses, the training strategy proposed in the present invention can ensure that the gradient does not disappear, and the generated images are more realistic. The proposed method has received better evaluation on the data set.

[0074] Embodiment 2:

[0075] The present embodiment proposes an image defogging device with a Transformer structure integrating near infrared, and the device includes a data acquisition module and a Transformer structure image defogging model module.

[0076] A data acquisition module, used for capturing and acquiring visible light images and near infrared images corresponding to the same scene;

[0077] The Transformer structure image defogging model module uses the visible light image and the near-infrared image. The Transformer structure image defogging model first encodes and represents the visible light image and the near-infrared image to obtain the feature vector of the visible light image and the feature vector of the near-infrared image respectively, and then the two are fused to obtain the interactive feature vector; then the feature vector of the visible light image and the feature vector of the near-infrared image are decoded respectively to obtain a visible light image sequence and a near-infrared image sequence, and the two are merged with the interactive feature vector after processing, and then the merged result is channel reorganized and convolved to output the defogging image result.

[0078] In the embodiment of the present invention, a data processing module and a data partitioning module may be added between the above two modules.

[0079] The data processing module is used to augment the near-infrared image dataset and expand the dataset quantity; it is used to enhance the near-infrared dataset and adjust the color scale of the area with low contrast;

[0080] The data partitioning module is used to perform partitioning operations based on the public data set and the near-infrared data set to obtain a training set, which can be used to train the Transformer structure image dehazing model.

[0081] In the embodiment of the present invention, the Transformer structure image defogging model includes a visible light image feature encoding module, a near infrared image feature encoding module, a feature interaction module, a visible light feature decoding module, a near infrared feature decoding module, and a feature fusion module; the Transformer structure image defogging model is composed of the following components: Figure 2 shown.

[0082] Furthermore, based on the paired near-infrared image data and the Transformer structure image defogging model, a clear color image result after defogging is obtained, including:

[0083] The visible light image is characterized by the visible light image feature encoding module, and three sets of visible light image feature vectors are obtained according to the RGB channels. The encoding module first includes a downsampling convolution layer and a global average pooling layer, followed by three Transformer encoding units connected in series.

[0084] The near-infrared image is characterized by a near-infrared image feature encoding module to obtain a set of feature vectors of the near-infrared image. The encoding module first includes a downsampling convolution layer and a global average pooling layer, followed by two Transformer encoding units connected in series.

[0085] Through the feature interaction module, the feature vector of the visible light image and the feature vector of the near infrared image are added by the feature map and then passed through the deconvolution layer and the ReLU activation layer to obtain the interactive feature vector;

[0086] The three sets of visible light image feature vectors are converted into a visible light image sequence through a visible light feature decoding module, which contains three Transformer encoding units and an upsampling convolutional layer;

[0087] A set of near-infrared image feature vectors is converted into a near-infrared image sequence through a near-infrared feature decoding module, which contains two Transformer encoding units and an upsampling convolutional layer;

[0088] Through the feature fusion module, the visible light image sequence is globally averaged pooled and then subjected to 3*3 convolution, and the near-infrared image sequence is subjected to two 1*1 convolutions. The results of the two-step operation are merged with the interactive feature vector by adding the feature map, and then the merged result is channel reorganized. Finally, 5*5 depth-separable convolution and size-adjusted convolution are performed to obtain a clear color image result after dehazing.

[0089] Embodiment 3:

[0090] Figure 5 It is a flowchart of using an electronic device provided by an embodiment of the present invention, and the electronic device 500 may have large differences due to different configurations or performances. In the electronic device 500, 501 is a near-infrared glass slide, and the optional band range is 700nm-1500nm, which is used to collect near-infrared pictures. When in use, it is installed or disassembled to obtain visible light and near-infrared paired images while ensuring that the other devices do not move. The electronic device 500 includes one or more photographic lenses 502 and one or more CMOS elements 503, wherein the photographic lens 502 can use different focal lengths such as telephoto, short focal and wide angle to collect pictures in different scenes, and the CMOS element 503 needs to support no high-pass filtering. The electronic device also includes one or more processors (central processing units, CPU) 504 and one or more memories 505, wherein the memory 505 stores at least one instruction, and at least one instruction is loaded and executed by the processor 504 to implement the steps of the above-mentioned image defogging method of the Transformer structure fused with near-infrared.

[0091] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image dehazing method integrating a near-infrared Transformer structure, characterized in that: The steps include: S1: Take near-infrared images and visible light images of the same scene to form a data set; S2: Both the near-infrared image and the visible light image are input into the pre-trained Transformer structure image dehazing model; The Transformer structure image dehazing model first encodes and represents the visible light image and the near infrared image, respectively obtaining the feature vector of the visible light image and the feature vector of the near infrared image, and then fusing the two to obtain the interactive feature vector; Then, the feature vector of the visible light image and the feature vector of the near infrared image are decoded respectively to obtain a visible light image sequence and a near infrared image sequence, which are processed and merged with the interactive feature vector; The merged result is then subjected to channel reorganization and convolution processing to output the dehazed image result.

2. The image defogging method of the Transformer structure fused with near infrared as claimed in claim 1, characterized in that: The Transformer structure image dehazing model includes a visible light image feature encoding module, a near infrared image feature encoding module, a feature interaction module, a visible light feature decoding module, a near infrared feature decoding module and a feature fusion module; The processes executed by each module are as follows: The visible light image feature encoding module characterizes the visible light image and obtains feature vectors of three groups of visible light images according to RGB channels. The visible light image feature encoding module first includes a downsampling convolution layer and a global average pooling layer, and then three Transformer encoding units are serially connected; The near-infrared image feature encoding module characterizes the near-infrared image to obtain a set of feature vectors of the near-infrared image. The encoding module first includes a downsampling convolution layer and a global average pooling layer, and then two Transformer encoding units are serially connected; In the feature interaction module, the feature vector of the visible light image and the feature vector of the near infrared image are added by a feature map and then passed through a deconvolution layer and a ReLU activation layer to obtain an interactive feature vector; The visible light feature decoding module converts the three groups of visible light image feature vectors into a visible light image sequence, and the visible light feature decoding module includes three Transformer encoding units and an upsampling convolution layer; The near-infrared feature decoding module converts a set of near-infrared image feature vectors into a near-infrared image sequence, and the near-infrared feature decoding module includes two Transformer encoding units and an upsampling convolutional layer; The feature fusion module performs global average pooling and then 3*3 convolution on the visible light image sequence, and performs two 1*1 convolutions on the near-infrared image sequence. The results of the two-step operation are merged with the interactive feature vector by adding feature maps, and then the merged result is channel reorganized. Finally, 5*5 depth-separable convolution and size-adjusted convolution are performed to obtain a clear color image result after dehazing.

3. The image defogging method of the Transformer structure fused with near infrared as claimed in claim 1 or 2, characterized in that: After outputting the dehazed image result, the method also includes calibrating the dehazed image result using an image discriminator; the image discriminator uses two block structures including a convolution layer-normalization layer-ReLU activation layer-pooling layer, and then verifies the deep information of the image by mixing channel attention with spatial attention, and finally judges the true or false probability of the image through a Softmax function.

4. The image defogging method of the Transformer structure fused with near infrared as claimed in claim 2, characterized in that: The pre-trained Transformer structure image dehazing model is pre-trained in the following way: Three types of losses are used to constrain the training of the Transformer structure image dehazing model. The first loss is Charbonnier loss. The second loss is perceptual loss, which uses the feature map output by the pre-trained VGG16 network at the 14th layer to calculate the loss; The third loss is the true or false probability loss obtained by the image discriminator.

5. Image defogging device integrating near-infrared Transformer structure, characterized in that: The device includes a data acquisition module and a Transformer structure image defogging model module; A data acquisition module, used for capturing and acquiring visible light images and near infrared images corresponding to the same scene; The Transformer structure image dehazing model module uses the visible light image and the near-infrared image. The Transformer structure image dehazing model first encodes and represents the visible light image and the near-infrared image to obtain the feature vector of the visible light image and the feature vector of the near-infrared image respectively, and then fuses the two to obtain the interactive feature vector; then decodes the feature vector of the visible light image and the feature vector of the near-infrared image respectively to obtain a visible light image sequence and a near-infrared image sequence, and merges the two after processing with the interactive feature vector, and then performs channel reorganization and convolution processing on the merged result to output the dehazed image result.

6. The image defogging device with a Transformer structure integrating near infrared as claimed in claim 5, characterized in that: The Transformer structure image dehazing model includes a visible light image feature encoding module, a near infrared image feature encoding module, a feature interaction module, a visible light feature decoding module, a near infrared feature decoding module, and a feature fusion module; The visible light image feature encoding module characterizes the visible light image and obtains three sets of visible light image feature vectors according to RGB channels. The encoding module first includes a downsampling convolution layer and a global average pooling layer, and then three Transformer encoding units are serially connected; The near-infrared image feature encoding module characterizes the near-infrared image to obtain a set of feature vectors of the near-infrared image. The encoding module first includes a downsampling convolution layer and a global average pooling layer, and then two Transformer encoding units are serially connected; In the feature interaction module, the feature vector of the visible light image and the feature vector of the near infrared image are added by a feature map and then passed through a deconvolution layer and a ReLU activation layer to obtain an interactive feature vector; The visible light feature decoding module converts the three groups of visible light image feature vectors into a visible light image sequence, and the decoding module includes three Transformer encoding units and an upsampling convolution layer; The near-infrared feature decoding module converts a set of near-infrared image feature vectors into a near-infrared image sequence, and the decoding module includes two Transformer encoding units and an upsampling convolutional layer; The feature fusion module performs global average pooling and then 3*3 convolution on the visible light image sequence, and performs two 1*1 convolutions on the near-infrared image sequence. The results of the two-step operation are merged with the interactive feature vector by adding feature maps, and then the merged result is channel reorganized. Finally, 5*5 depth-separable convolution and size-adjusted convolution are performed to obtain a clear color image result after dehazing.

Images