RGB-FIR multispectral image defogging method based on generative adversarial network

Through the RGB-FIR multispectral image defogging method based on adversarial generation network, combined with the complementary characteristics of RGB and FIR images, the problems of prior knowledge dependence and high computational complexity in the existing technology are solved, and the problem of high computational complexity is realized in complex scenarios Efficient fog removal effect.

CN119090773APending Publication Date: 2024-12-06NANTONG UNIV
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411275755.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing image defogging technology has the problems of high prior knowledge dependence, high computational complexity, poor edge and details processing, and single data modality, resulting in poor adaptability in complex scenarios and dense fog conditions.

Method used

Using the RGB-FIR multispectral image defogging method based on adversarial generation network, a multi-level feature fusion module and a multi-scale image enhancement module are used to combine the complementary characteristics of RGB and FIR images to construct a multi-effect loss function and optimize the network parameters to achieve defogging. Fog effect.

Benefits of technology

It significantly enhances the image details and color recovery effect under dense fog conditions, provides richer feature information, and improves the adaptability and efficiency of image fog removal.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090773A_ABST
    Figure CN119090773A_ABST
Patent Text Reader

Abstract

The invention provides an RGB-FIR multispectral image defogging method based on a generative adversarial network, and relates to the technical field of computer vision model design, and the invention provides an RGB-FIR multispectral image defogging model construction method based on the generative adversarial network, and the method comprises the following steps: S1, information collection: S2, constructing an RGB-FIR multispectral defogging model based on multistage feature fusion; s3, constructing a fusion-enhancement multi-loss function by adopting a multi-effect learning strategy based on adversarial generation, training the model by utilizing an RGB-FIR image data set, and optimizing network parameters; and S4, model testing. The model fully considers the complementary characteristics of two optical images of RGB-FIR under the condition of fog with different concentrations due to different imaging principles of the two optical images of RGB-FIR. According to the model provided by the invention, information fusion is carried out on optical image features of RGB-FIR modes, advantage complementation of multi-band optical images is realized, details, texture and color information of restored images are remarkably enhanced, and a clear defogging effect is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision model construction, and in particular to an RGB-FIR multispectral image defogging method based on a generative adversarial network. Background Art

[0002] With the rapid development of artificial intelligence technology, computer vision technology based on deep learning algorithms has been widely used in road traffic, video surveillance, national security and environmental monitoring and other fields. Among them, image dehazing technology is an image processing method that aims to eliminate visual degradation caused by haze, atmospheric particles, etc. in images. In practical applications, due to haze and atmospheric scattering effects, the contrast and color saturation in the image will be significantly reduced, and the details will become blurred, thus affecting the usability and visual quality of the image. In particular, image degradation caused by haze may seriously affect the performance and reliability of the system. Therefore, in order to enhance the reliability of computer vision tasks such as target tracking, detection, and behavior recognition, improve image details, and restore image color saturation, multimodal dehazing technology based on deep learning has very important research significance and practical value.

[0003] Existing image defogging methods have certain limitations when dealing with complex scenes, weather conditions such as dense fog, or different imaging modalities. At present, image defogging methods can be roughly divided into traditional image defogging methods and deep learning-based image defogging methods. Traditional learning methods mainly include image enhancement-based and prior knowledge-based defogging methods. Image enhancement-based defogging algorithm: By removing and reducing the noise of the image, a fog-free and clear image is restored. For example: histogram equalization, Retinex theory, wavelet transform, etc. Defogging method based on prior knowledge: Usually based on the atmospheric degradation model, by estimating the scene depth map (i.e., fog concentration) and air light component, a clear image of the original scene is restored. For example: dark channel prior, image entropy optimization, multi-scale Retinex, color attenuation prior, etc. The defogging model based on deep learning refers to estimating transmittance and atmospheric light through convolutional neural networks (CNN), generative adversarial networks (GAN), diffusion models, etc., solving fog-free images based on the atmospheric scattering physical model or directly generating fog-free images through mapping relationships between images.

[0004] In summary, the existing image dehazing technology has the following problems: ① Traditional learning methods are highly dependent on prior knowledge, and do not handle edges and details well, resulting in color distortion; ② Some image dehazing technologies based on physical models have high computational complexity and are difficult to meet real-time requirements; ③ The data modality is single, the image feature information is limited, there is a lack of fusion of multimodal information, and the adaptability to complex scenes is poor. Summary of the invention

[0005] The purpose of the present invention is to solve the technical problems existing in the image defogging technology in the prior art and to design an RGB-FIR multimodal image defogging method based on a generative adversarial network.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for constructing an RGB-FIR multispectral image dehazing model based on a generative adversarial network includes the following steps:

[0008] S1: Information collection: Collect RGB and FIR fog images in real indoor and outdoor scenes, and obtain image pairs with the same field of view and image resolution through screening and registration, and make a dataset that can be used by the RGB-FIR multispectral dehazing network;

[0009] S2: According to the optical complementary characteristics of RGB-FIR images, an RGB-FIR multispectral dehazing model based on multi-level feature fusion is constructed; the RGB-FIR multispectral dehazing model based on multi-level feature fusion includes two sub-network modules: a multi-level feature fusion module and a multi-scale image enhancement module. The input of the "multi-level feature fusion module" is the image pair after S1 registration, and the output feature of the "multi-level feature fusion module" is the input of the "multi-scale image enhancement module". Finally, the multimodal fusion feature under foggy conditions is established through the "multi-scale image enhancement module". Figure X CF Y RGB The nonlinear mapping relationship outputs a clear image without fog;

[0010] S3: Adopting a multi-effect learning strategy based on adversarial generation, constructing a "fusion-enhancement" multiple loss function, using the RGB-FIR image dataset to train the RGB-FIR multispectral dehazing model with multi-level feature fusion, and optimizing the network parameters;

[0011] S4: Model testing.

[0012] Preferably, the specific steps of S1 are as follows:

[0013] The binocular camera based on the principle of thermal imaging is used for video acquisition. At the same time, an ultrasonic water mist machine is used to create artificial fog under this condition to collect RGB and FIR video data under various fog conditions. The collected RGB and FIR video data are cropped into image data through the OpenCV library, and the multispectral images are preliminarily screened to delete invalid image data.

[0014] Then, use the imresize function to resize the far infrared FIR and visible light I RGBThe two images are preliminarily registered. The Harris algorithm is used to extract the feature point information of the two optical images from the processed images. The feature descriptor SIFT and Euclidean distance are used to find the correspondence between the feature points of the two images, and the feature points are matched. The MSAC (M-estimator Sample Consensus) algorithm is used to obtain a more accurate transformation relationship for the feature matching results of the preliminary registration. Finally, after fine image registration, RGB-FIR image pairs with the same field of view and resolution are obtained to create an RGB-FIR image dataset.

[0015] Finally, the prepared RGB-FIR image dataset is converted into a TXT file used in model training; the content of the TXT file is as follows: the number of lines in the TXT file is equal to the number of groups of image pairs in the dataset, each line corresponds to an image group, and each content is divided into four sections, namely: group number, foggy RGB image address, foggy FIR image address, and clear fog-free RGB image (true value).

[0016] Preferably, the "multi-level feature fusion module" described in S2 is composed of two sub-network modules: a "multi-modal pixel-level fusion module" and a "multi-scale feature-level fusion module", wherein the "multi-level feature fusion module" is composed of I FIR and I RGB After the pixel-level and feature-level fusion are performed by the “multi-modal pixel-level fusion module” and the “multi-scale feature-level fusion module” respectively, the fused feature X is output. CF .

[0017] Preferably, the “multimodal pixel-level fusion module” inputs the original feature X FIR and X RGB , a three-branch forward pixel-level information fusion method is adopted: the upper branch feature stream is the RGB original pixel feature, the lower branch feature stream is the FIR original pixel feature, and the middle branch is the RGB-FIR pixel fusion feature stream.

[0018] Preferably, the multimodal pixel-level fusion module structure and feature fusion process in S2 are as follows:

[0019] First, use X FIR The weight value W generated by the cross-pooled multi-scale cross-modal attention model (MCAMC) FIR , for X RGB Adjust to generate optimized RGB feature X' RGB , and the original FIR feature Figure X FIR Perform the first addition to obtain the fused feature X' fusion , for X' fusionGenerate new weight value X' using MCAMC FIR Perform secondary adjustment to obtain feature X″ RGB , X″ RGB Again with characteristics Figure X FIR Perform the second addition to obtain the pixel fusion feature X″ fusion ;

[0020] The input of the cross-pooling-based multi-scale cross-modal attention model is the FIR original feature X FIR , X FIR Perform average pooling of two scales in the horizontal and vertical directions, as shown in formula (1-2), where the pooling scale factor k m ∈[3,7]:

[0021]

[0022] Four pooling features in the horizontal and vertical directions with consistent spatial resolution are obtained by upsampling And design to use RAM modules Perform channel compression, as shown in formula (3-6);

[0023]

[0024]

[0025] X hw =Concat(X mean , X max ) (5)

[0026] U hw =Con(X hw ) (6)

[0027] X mean Represents the average value of all channels of the pixel, X max represents the maximum value of the channel where the pixel is located, Conv is a convolution with a convolution kernel size of 1×1, and Concat represents splicing on the channel; secondly, the compressed features are spliced ​​through the channels to obtain the joint feature X of the four-channel multi-scale cross pooling c ;

[0028] Finally, the spatial feature encoder-decoder SFED is designed to perform nonlinear fusion mapping of feature points at the same spatial position on different channels, as shown in formula (7):

[0029]

[0030] In the formula, F EDIt is a nonlinear fusion mapping function, which includes a dimension reduction module based on full connection, an activation function ReLU and a dimension increase module based on full connection; F ex Reshape is used to realize the mapping of one-dimensional optimized feature vector to two-dimensional spatial feature map, F fc1 、F fc2 They are the dimensionality reduction process and dimensionality increase process based on full connection respectively; finally, the spatial attention importance weight W is generated by ReLU and Sigmoid activation functions respectively. f , as shown in formula (8):

[0031]

[0032] Where σ is the ReLU activation function, is the Sigmoid activation function.

[0033] Preferably, the "multi-scale feature-level fusion module" is composed of three-branch generators of two scales; each branch generator is composed of a front-end convolution, three residual blocks in the middle, and a back-end deconvolution;

[0034] The middle branch small-scale generator combines the underlying multimodal pixel-level fusion features X″ fusion After two times downsampling, the residual module is used to extract high-level features, and the high-level features are element-wise added to the convolutional features of the front end of the upper branch large-scale generator and the lower branch large-scale generator, and then input into the residual block of the corresponding large-scale generator;

[0035] Finally, the output features of the upper and lower large-scale generators are added to obtain the intermediate output result X of the multi-level feature fusion network CF , X CF Then with the input feature X RGB Perform channel splicing to obtain the output of the "multi-level feature fusion module".

[0036] Preferably, the feature extraction process of the middle branch small-scale generator is as shown in formula (9-11):

[0037]

[0038] G S =F rct (U DS ) (11)

[0039] In the above formula, Downsample is a two-fold downsampling operation, Conv 7 It is a convolution operation with a kernel of 7 and a stride of 1. LN represents layer normalization. RB contains three residual blocks connected in series. US represents three deconvolution operations with a kernel of 3 and a stride of 2, a layer normalization and a ReLU activation function operation.rct To include a pair of features U DS Taking the outermost pixel as the symmetry axis, perform a symmetrical mirror filling operation with a value of 3, a convolution with a kernel of 7, and a Tanh function operation; G S is the output feature of the small-scale generator;

[0040] The input of the upper and lower branch large-scale generators is X FIR and X RGB , the feature fusion process is as follows:

[0041]

[0042] G RGB =F rct (U RGB ) (16)

[0043] G FIR =F rct (U FIR ) (17)

[0044] In the above formula, Conv 3 It is represented as a convolution with a kernel of 3 and a stride of 2; is the convolution feature obtained after two convolution operations and a ReLU activation function; DC includes one deconvolution with a kernel of 3 and a step size of 2, one layer normalization and a ReLU activation function operation; U RGB , U FIR Features and The features extracted after DC operation, G RGB , G FIR They are the output features of the upper and lower branch large-scale generators respectively;

[0045] Finally, the output features of the upper and lower branch large-scale generators are added to X by element-wise addition. RGB Perform channel splicing to obtain feature-level fusion results As shown in formula (18-19):

[0046] X CF =G RGB +G FIR (18)

[0047]

[0048] Preferably, the “migration-enhancement” process of the features of the “multi-scale image enhancement module” is as follows:

[0049] Fusion Features As the input of the “multi-scale image enhancement module”, it is first enhanced by the Enhance Block and then combined with the intermediate output feature X of the multi-level feature fusion module CF Perform channel stitching, then perform a second enhancement, and finally output the dehazed image

[0050] The Enhance Block is an enhancement block based on a pyramid structure. The input feature of the enhancement block is After SE channel attention optimization, downsampling at four scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 is performed to construct a four-branch multi-scale pyramid structure. Then, the features after upsampling alignment and channel attention optimization are used for channel splicing, and the enhanced result is output through 3×3 convolution. The process is shown in formula (20-24):

[0051]

[0052] Where Conv 3 Indicates the convolution operation with a kernel of 3 and a stride of 1. in is the input of the pyramid structure, is the output of the pyramid structure, F AP (x,b) means performing average pooling of scale b on x; Upsampling represents the upsampling function, and tanh represents the activation function Tanh. and Represent the output features after the first and second enhancements respectively.

[0053] Preferably, the specific steps of S3 are as follows:

[0054] 3-1 According to the content of S1, through image registration, a data set of RGB-FIR image pairs with the same field of view and resolution is prepared (a total of 11,736 images, including 13 different indoor and outdoor scenes), and the RGB-FIR image pairs under different fog conditions are divided into training sets and test sets in a ratio of 7:3, and TXT files for training and testing are prepared for training;

[0055] 3-2 Configuration of hardware equipment and experimental environment; The server environment is as follows: Windows 11 system, CPU (Intel Core i7-11800H), GPU (NVIDIA GeForce RTX3060), RAM (32GB), CUDA 11.3; Pycharm is used as the programming platform, Python 3.9 and pytorch 1.10 are used to configure the virtual environment, initialize the model parameters, and set the number of training rounds to 200;

[0056] 3-3 Design of multi-effect loss function L based on learning strategy of adversarial generative network MF , the RGB-FIR multi-spectral dehazing model with multi-level feature fusion is used as the generator G, and the multi-scale discriminator D is constructed; then, in the process of 200 rounds of iterative training, the weight parameters of the RGB-FIR multi-spectral dehazing model with multi-level feature fusion are corrected and optimized; the network weight file is finally saved as the optimal parameter model;

[0057] Among them, the multi-effect loss function L MF By adversarial loss L A , feature matching loss L FM , Perceptual loss L VGG and fidelity loss L F Four loss functions are constructed; by L FM , L VGG and L F Different weighting factors are assigned to control the relative importance among the four losses, as shown in Equation (25):

[0058] L MF =L A +λ 1 L FM +λ 2 L VGG +λ 3 L F (25)

[0059] The adversarial loss function L A As a multi-scale discriminator D k (k∈[1,2]) Identify multi-level fusion features X at different resolutions CF With clear image Y RGB (true value), identify the multi-level fusion feature X CF And generate images The evaluation criteria for authenticity are as follows (26-27):

[0060]

[0061] In the formula, represents the adversarial loss of the multi-scale discriminator, The generated image output by the generator; Indicates the expectations of the sample;

[0062] The feature matching loss function L FM As a multi-scale discriminator D k (k∈[1,2]) Identify the generated image and the clear image Y output by the generator at different resolutions RGB The authenticity evaluation standard between (true value) is as follows:

[0063]

[0064] In the formula is the kth discriminator D k The resulting feature matching loss, T is the total number of layers used for feature extraction, N i is the number of elements in each layer, is the operator for extracting features at the i-th layer, represents the expectation of the sample;

[0065] The perceptual loss function L VGG Maintaining the consistency between the generated image and the true image from the perspective of perception and semantics makes the generated image more consistent with the human eye's perception of image quality. The definition is as follows:

[0066]

[0067] In the formula, It is represented as the final result of the multi-spectral dehazing network output, H i and W i is the height and width of the i-th feature map, C i Indicates the number of channels, C i H i W i represents the size of the feature map of the i-th layer, represents the feature representation of the i-th layer in the VGG network;

[0068] The fidelity loss function L F Directly use the Euclidean distance to measure the true value image Y RGB And generate images The difference between them is defined as follows:

[0069]

[0070] Preferably, it is obtained by constructing using the construction method described above.

[0071] The image defogging method of the present invention aims at the problems of the existing image defogging methods, such as high dependence on prior knowledge, high model calculation complexity, poor processing of edges and details, color distortion, single data modality, and poor adaptability to complex scenes, and establishes an RGB-FIR multispectral image defogging method based on a generative adversarial network. The model fully considers the complementary characteristics of the two optical images of RGB-FIR under fog conditions of different concentrations due to their different imaging principles. Therefore, the RGB image with rich color and detailed texture characteristics is combined with the FIR image with "fog-penetrating" characteristics, which can achieve modal complementarity in different fog scenes and provide richer and more effective feature information. Secondly, a "fusion-enhancement" multi-effect defogging model structure based on a generative adversarial network is constructed to achieve adaptive coordination of parameters between modules, significantly enhance the details, texture and color information of the restored image under dense fog conditions, and obtain a clear defogging effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a flow chart of the RGB-FIR multispectral image dehazing method based on the adversarial generative network proposed by the present invention;

[0073] Figure 2 This is a schematic diagram of the RGB-FIR multispectral image dehazing model structure based on the adversarial generative network proposed in the present invention;

[0074] Figure 3 This is a schematic diagram of the structure of the "multimodal pixel-level fusion module" proposed in the present invention;

[0075] Figure 4 This is a schematic diagram of the structure of the "cross-pooled multi-scale cross-modal attention model" proposed in the present invention;

[0076] Figure 5 This is a schematic diagram of the structure of the "spatial feature encoder-decoder" proposed in the present invention;

[0077] Figure 6 This is a schematic diagram of the structure of the "multi-scale feature-level fusion module" proposed in the present invention;

[0078] Figure 7 This is a schematic diagram of the structure of the "multi-scale image enhancement module" proposed in the present invention;

[0079] Figure 8 This is a schematic diagram of the structure of the enhancement block "Enhance Block" proposed in the present invention;

[0080] Fig. 9 This is an example diagram comparing the defogging effects of the multimodal defogging method of the present invention and other traditional image defogging methods under different foggy conditions;

[0081] Fig.10 This is an example diagram comparing the defogging effects of the multimodal defogging method of the present invention and other deep learning image defogging methods under different fog conditions. DETAILED DESCRIPTION

[0082] The present invention is further described in detail below in conjunction with specific embodiments.

[0083] See also Figure 1 , a method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network, comprising the following steps:

[0084] S1: Information collection:

[0085] In one embodiment, a thermal imaging binocular camera and an ultrasonic water fog machine are used to manually collect RGB and FIR fog images in real indoor and outdoor scenes, and image pairs with the same field of view and image resolution are obtained through screening and alignment to produce a data set that can be used by the RGB-FIR multispectral dehazing network.

[0086] The specific steps are as follows:

[0087] 1-1 First, a binocular camera based on the principle of thermal imaging is used for video acquisition. At the same time, an ultrasonic water mist machine is used to create artificial fog under this condition to collect RGB and FIR video data under various fog conditions. The collected RGB and FIR video data are cropped into image data through the OpenCV library, and the multispectral images are preliminarily screened to delete invalid image data.

[0088] 1-2 Then, use the imresize function to resize the far infrared I FIR and visible light I RGB The two images are preliminarily registered. The Harris algorithm is used to extract the feature point information of the two optical images from the processed images. The feature descriptor SIFT and Euclidean distance are used to find the correspondence between the feature points of the two images, and the feature points are matched. The MSAC (M-estimator Sample Consensus) algorithm is used to obtain a more accurate transformation relationship for the feature matching results of the preliminary registration. Finally, after fine image registration, RGB-FIR image pairs with the same field of view and resolution are obtained to create an RGB-FIR image dataset.

[0089] 1-3Finally, convert the prepared RGB-FIR image dataset into a TXT file used in model training. The content of the TXT file is as follows: The number of lines in the TXT file is equal to the number of image pairs in the dataset. Each line corresponds to an image group, and each content is divided into four sections, namely: group number, foggy RGB image address, foggy FIR image address, and clear fog-free RGB image (true value).

[0090] S2: Constructing an RGB-FIR multispectral dehazing model based on multi-level feature fusion:

[0091] In one implementation, a defogging model is constructed based on the optical complementary characteristics of the RGB-FIR image, specifically:

[0092] 2-1 First, the RGB-FIR multispectral dehazing model based on multi-level feature fusion consists of two sub-network modules: "multi-level feature fusion module" and "multi-scale image enhancement module", with the following structure: Figure 2 As shown in the figure, the input of the "multi-level feature fusion module" is the image pair after S1 registration, and the output features of the "multi-level feature fusion module" are the input of the "multi-scale image enhancement module". Finally, the multi-modal fusion features under foggy conditions are established through the "multi-scale image enhancement module". Figure X CF Y RGB The nonlinear mapping relationship outputs a clear image without fog;

[0093] The “multi-level feature fusion module” described in 2-1-1 is composed of two sub-network modules: “multi-modal pixel-level fusion module” and “multi-scale feature-level fusion module”. FIR and I RGB After the pixel-level and feature-level fusion are performed by the “multi-modal pixel-level fusion module” and the “multi-scale feature-level fusion module” respectively, the fused feature X is output. CF .

[0094] The structure and feature fusion process of the "multimodal pixel-level fusion module" described in (a) are as follows:

[0095] The “multimodal pixel-level fusion module” inputs the original feature X FIR and X RGB , adopts the upper, middle and lower three-branch forward pixel-level information fusion method: the upper branch feature flow is the RGB original pixel feature, the lower branch feature flow is the FIR original pixel feature, and the middle branch is the RGB-FIR pixel fusion feature flow. The structure is as follows Figure 3 shown.

[0096] First, use X FIRThe weight value W generated by the cross-pooled multi-scale cross-modal attention model (MCAMC) FIR , for X RGB Adjust to generate optimized RGB feature X' RGB , and the original FIR feature Figure X FIR Perform the first addition to obtain the fused feature X' fusion , for X' fusion Generate new weight value X' using MCAMC FIR Perform secondary adjustment to obtain feature X″ RGB , X″ RGB Again with characteristics Figure X FIR Perform the second addition to obtain the pixel fusion feature X″ fusion .

[0097] Among them, the structure of the multi-scale cross-modal attention model (MCAMC) based on cross pooling is as follows: Figure 4 As shown, the input of the MCAMC is the original FIR feature X FIR , X FIR Perform average pooling of two scales in the horizontal and vertical directions, as shown in formula (1-2), where the pooling scale factor k m ∈[3,7]:

[0098]

[0099] Four pooling features in the horizontal and vertical directions with consistent spatial resolution are obtained by upsampling And design to use RAM modules Perform channel compression as shown in formula (3-6).

[0100]

[0101] X hw =Concat(X mean , X max ) (5)

[0102] U hw =Con(X hw ) (6)

[0103] X mean Represents the average value of all channels of the pixel, X max Indicates the maximum value of the channel where the pixel is located, Conv is a convolution with a convolution kernel size of 1×1, and Concat represents splicing on the channel. Secondly, the compressed features are spliced ​​through the channels to obtain the joint feature X of the four-channel multi-scale cross pooling c .

[0104] Finally, a spatial feature encoder-decoder SFED is designed to perform nonlinear fusion mapping on feature points at the same spatial position on different channels. The spatial feature encoder-decoder (SFED) structure is as follows: Figure 5 As shown in formula (7):

[0105]

[0106] In the formula, F ED It is a nonlinear fusion mapping function, which includes a fully connected dimension reduction module, an activation function ReLU and a fully connected dimension increase module. ex Reshape is used to realize the mapping of one-dimensional optimized feature vector to two-dimensional spatial feature map, F fc1 、F fc2 They are the dimension reduction process and dimension increase process based on full connection respectively. Finally, the spatial attention importance weight W is generated by ReLU and Sigmoid activation functions respectively. f , as shown in formula (8):

[0107]

[0108] Where σ is the ReLU activation function, is the Sigmoid activation function.

[0109] The structure and feature fusion process of the "multi-scale feature level fusion module" described in (b) are as follows:

[0110] The structure of the “multi-scale feature level fusion module” is as follows Figure 6 As shown in the figure, the “multi-scale feature-level fusion module” consists of three-branch generators at two scales. Each branch generator consists of a front-end convolution, three residual blocks in the middle, and a back-end deconvolution. The middle branch small-scale generator converts the underlying multimodal pixel-level fusion feature X″ fusion After two times downsampling, the residual module is used to extract high-level features, and the high-level features are added element-wise with the convolution features of the front end of the upper branch large-scale generator and the lower branch large-scale generator, and then input into the residual block of the corresponding large-scale generator. Finally, the output features of the upper and lower large-scale generators are added to obtain the intermediate output result X of the multi-level feature fusion network. CF , X CF Then with the input feature X RGB Perform channel splicing to obtain the output of the "multi-level feature fusion module".

[0111] Among them, the feature extraction process of the middle branch small-scale generator is as shown in formula (9-11):

[0112]

[0113] G S =F rct (U DS ) (11)

[0114] In the above formula, Downsample is a two-fold downsampling operation, Conv 7 It is a convolution operation with a kernel of 7 and a stride of 1. LN represents layer normalization. RB contains three residual blocks connected in series. US represents three deconvolution operations with a kernel of 3 and a stride of 2, a layer normalization and a ReLU activation function operation. rct To include a pair of features U DS Taking the outermost pixel as the symmetry axis, perform a symmetrical mirror filling operation with a value of 3, a convolution with a kernel of 7, and a Tanh function operation. S is the output feature of the small-scale generator.

[0115] The input of the upper and lower branch large-scale generators is X FIR and X RGB , the feature fusion process is as follows:

[0116]

[0117] G RGB =F rct (U RGB ) (16)

[0118] G FIR =F rct (U FIR ) (17)

[0119] In the above formula, Conv 3 Represented as a convolution with a kernel of 3 and a stride of 2. The convolution feature is obtained after two convolution operations and a ReLU activation function. DC includes one deconvolution with a kernel of 3 and a stride of 2, one layer normalization, and a ReLU activation function operation. RGB , U FIR Features and The features extracted after DC operation, G RGB , G FIR They are the output features of the upper and lower branch large-scale generators respectively.

[0120] Finally, the output features of the upper and lower branch large-scale generators are added to X by element-wise addition. RGB Perform channel splicing to obtain feature-level fusion results As shown in formula (18-19):

[0121] X CF=G RGB +G FIR (18)

[0122]

[0123] The structure of the "multi-scale image enhancement module" described in 2-1-2 is as follows Figure 7 As shown, the “migration-enhancement” process of the features of the “multi-scale image enhancement module” is as follows:

[0124] Fusion Features As the input of the “multi-scale image enhancement module”, it is first enhanced by the Enhance Block and then combined with the intermediate output feature X of the multi-level feature fusion module CF Perform channel stitching, then perform a second enhancement, and finally output the dehazed image

[0125] The Enhance Block is an enhancement block based on a pyramid structure. The structure of the Enhance Block is as follows: Figure 8 As shown, the input features of the enhancement block After SE channel attention optimization, downsampling at four scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 is performed to construct a four-branch multi-scale pyramid structure. Then, the features after upsampling alignment and channel attention optimization are used for channel splicing, and the enhanced result is output through 3×3 convolution. The process is shown in formula (20-24):

[0126]

[0127]

[0128] Where Conv 3 Indicates the convolution operation with a kernel of 3 and a stride of 1. in is the input of the pyramid structure, is the output of the pyramid structure, F AP (X,B) means performing average pooling of scale b on x. Upsampling represents the upsampling function, and tanh represents the activation function Tanh. and Represent the output features after the first and second enhancements respectively.

[0129] S3: A multi-effect learning strategy based on adversarial generation is adopted to construct a "fusion-enhancement" multiple damage function. The RGB-FIR image dataset is used to train the RGB-FIR multispectral dehazing model with multi-level feature fusion and optimize the network parameters.

[0130] In one implementation, the specific steps of S3 are as follows:

[0131] 3-1 According to the content of S1, through image registration, a data set of RGB-FIR image pairs with the same field of view and resolution is prepared (a total of 11,736 images, including 13 different indoor and outdoor scenes), and the RGB-FIR image pairs under different fog conditions are divided into training sets and test sets in a ratio of 7:3, and TXT files for training and testing are prepared for training;

[0132] 3-2 Configuration of hardware equipment and experimental environment. The server environment is as follows: Windows 11 system, CPU (Intel Core i7-11800H), GPU (NVIDIA GeForce RTX3060), RAM (32GB), CUDA 11.3. Pycharm is used as the programming platform, Python 3.9 and pytorch 1.10 are used to configure the virtual environment, initialize the model parameters, and set the number of training rounds to 200.

[0133] 3-3 Design of multi-effect loss function L based on learning strategy of adversarial generative network MF , the RGB-FIR multi-spectral dehazing model with multi-level feature fusion is used as the generator G, and the multi-scale discriminator D is constructed; then, in the process of 200 rounds of iterative training, the weight parameters of the RGB-FIR multi-spectral dehazing model with multi-level feature fusion are corrected and optimized; the network weight file is finally saved as the optimal parameter model.

[0134] Among them, the multi-effect loss function L MF By adversarial loss L A , feature matching loss L FM , Perceptual loss L VGG and fidelity loss L F Four loss functions are constructed. FM , L VGG and L F Different weighting factors are assigned to control the relative importance among the four losses, as shown in Equation (25):

[0135] L MF =L A +λ 1 L FM +λ 2 L VGG +λ 3 L F (25)

[0136] The adversarial loss function L A As a multi-scale discriminator D k(k∈[1,2]) Identify multi-level fusion features X at different resolutions CF With clear image Y RGB (true value), identify the multi-level fusion feature X CF And generate images The evaluation criteria for authenticity are as follows (26-27):

[0137]

[0138] In the formula, represents the adversarial loss of the multi-scale discriminator, The generated image output by the generator. Represents the expectation of the sample.

[0139] The feature matching loss function L FM As a multi-scale discriminator D k (k∈[1,2]) Identify the generated image and the clear image Y output by the generator at different resolutions RGB The authenticity evaluation standard between (true value) is as follows:

[0140]

[0141] In the formula is the kth discriminator D k The resulting feature matching loss, T is the total number of layers used for feature extraction, N i is the number of elements in each layer, is the operator for extracting features at the i-th layer, represents the expectation of the sample.

[0142] The perceptual loss function L VGG Maintaining the consistency between the generated image and the true image from the perspective of perception and semantics makes the generated image more consistent with the human eye's perception of image quality. The definition is as follows:

[0143]

[0144] In the formula, It is represented as the final result of the multi-spectral dehazing network output, H i and W i is the height and width of the i-th feature map, C i Indicates the number of channels, C i H i W i represents the size of the feature map of the i-th layer, Represents the feature representation of the i-th layer in the VGG network.

[0145] The fidelity loss function L FDirectly use the Euclidean distance to measure the true value image Y RGB And generate images The difference between them is defined as follows:

[0146]

[0147] S4: Use the trained RGB-FIR multispectral dehazing model to perform dehazing operation on the test data set composed of RGB-FIR images to obtain a clear image after dehazing.

[0148] 4-1 According to the test data set and test TXT file created in S3, the test set of RGB-FIR images to be tested that have been precisely aligned is used as the input of the "multi-level feature fusion module". The model file model.py trained in S3 is used to perform a dehazing test and output a haze-free image.

[0149] 4-2 Evaluate the quality of the images output by the model. Use SSIM, PSNR, MSE and color saturation as evaluation indicators to evaluate the dehazing effect of the model, and use the same training set and test set to train different dehazing models to compare the dehazing effects.

[0150] Comparison of defogging effects of different models Fig. 9 / 10, the comparison structure of evaluation indicators is shown in Table 1 and Table 2.

[0151] Table 1 Comparison of indoor image dehazing evaluation results

[0152]

[0153]

[0154] Table 2 Comparison of outdoor image dehazing evaluation results

[0155]

[0156] In addition, the present application also provides an RGB-FIR multispectral image dehazing model based on a generative adversarial network, which is constructed using the above-mentioned construction method.

[0157] Combination Fig. 9 , Fig.10 From Table 1 and Table 2, it can be seen that compared with the prior art, the images obtained using the above defogging model are clearer.

[0158] In summary, the RGB-FIR multispectral image defogging model based on the generative adversarial network provided in this application takes into account the complementary characteristics of the two optical images of RGB-FIR under different fog concentrations due to their different imaging principles. Therefore, combining the RGB image with rich color and detailed texture characteristics with the FIR image with "fog-penetrating" characteristics can achieve modal complementarity in different fog scenes and provide richer and more effective feature information. Secondly, a "fusion-enhancement" multi-effect defogging model structure based on the generative adversarial network is constructed to achieve adaptive coordination of parameters between modules, significantly enhance the details, texture and color information of the restored image under dense fog conditions, and obtain a clear defogging effect.

Claims

1. A method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network. Features: The following steps are involved: S1: Information collection: Collect RGB and FIR fog images in real indoor and outdoor scenes, and obtain image pairs with the same field of view and image resolution through screening and registration, and make a dataset that can be used by the RGB-FIR multispectral dehazing network; S2: According to the optical complementary characteristics of RGB-FIR images, an RGB-FIR multispectral dehazing model based on multi-level feature fusion is constructed; the RGB-FIR multispectral dehazing model based on multi-level feature fusion includes two sub-network modules, a multi-level feature fusion module and a multi-scale image enhancement module. The input of the "multi-level feature fusion module" is the image pair after S1 registration, and the output feature of the "multi-level feature fusion module" is the input of the "multi-scale image enhancement module". Finally, the multimodal fusion feature map X under foggy conditions is established through the "multi-scale image enhancement module". CF Y RGB The nonlinear mapping relationship outputs a clear image without fog; S3: Adopting a multi-effect learning strategy based on adversarial generation, constructing a "fusion-enhancement" multiple loss function, using the RGB-FIR image dataset to train the RGB-FIR multispectral dehazing model with multi-level feature fusion, and optimizing the network parameters; S4: Model testing.

2. According to the method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network according to claim 1, Features: The specific steps of S1 are as follows: The binocular camera based on thermal imaging principle is used for video acquisition. At the same time, under this condition, an ultrasonic water mist machine is used for artificial fogging to collect RGB and FIR video data under various fog conditions. And through the OpenCV library, the collected RGB and FIR video data are cropped into image data, and the multispectral image is preliminarily screened to delete invalid image data; Then, use the imresize function to resize the far infrared FIR and visible light I RGB The two images are preliminarily registered. The Harris algorithm is used to extract the feature point information of the two optical images from the processed images. The feature descriptor SIFT and Euclidean distance are used to find the correspondence between the feature points of the two images, and the feature points are matched. The MSAC (M-estimator Sample Consensus) algorithm is used to obtain a more accurate transformation relationship for the feature matching results of the preliminary registration. Finally, after fine image registration, RGB-FIR image pairs with the same field of view and resolution are obtained to create an RGB-FIR image dataset. Finally, the prepared RGB-FIR image dataset is converted into a TXT file used in model training; the content of the TXT file is as follows: the number of lines in the TXT file is equal to the number of groups of image pairs in the dataset, each line corresponds to an image group, and each content is divided into four sections, namely: group number, foggy RGB image address, foggy FIR image address, and clear fog-free RGB image.

3. According to the method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network according to claim 1, Features: The "multi-level feature fusion module" described in S2 is composed of two sub-network modules: a "multi-modal pixel-level fusion module" and a "multi-scale feature-level fusion module", wherein the "multi-level feature fusion module" is composed of two sub-network modules: a "multi-modal pixel-level fusion module" and a "multi-scale feature-level fusion module". FIR and I RGB After the pixel-level and feature-level fusion are performed by the "multi-modal pixel-level fusion module" and the "multi-scale feature-level fusion module" respectively, the fused feature X is output. CF .

4. According to the method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network according to claim 3, Features: The "multimodal pixel-level fusion module" inputs the original feature X FIR and X RGB , a three-branch forward pixel-level information fusion method is adopted: the upper branch feature stream is the RGB original pixel feature, the lower branch feature stream is the FIR original pixel feature, and the middle branch is the RGB-FIR pixel fusion feature stream.

5. According to the method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network according to claim 4, Features: The multimodal pixel-level fusion module structure and feature fusion process in S2 are as follows: First, use X FIR The weight value W generated by the cross-pooled multi-scale cross-modal attention model FIR , for X RGB Adjust to generate optimized RGB feature X′ RGB , and the original FIR feature map X FIR Perform the first addition to obtain the fused feature X′ fusion , for X′ fusion Generate new weight value X′ using MCAMC FIR Perform secondary adjustment to obtain feature X″ RGB , X″ RGB Then with the feature map X FIR Perform the second addition to obtain the pixel fusion feature X″ fusion ; The input of the cross-pooling-based multi-scale cross-modal attention model is the FIR original feature X FIR , X FIR Perform average pooling of two scales in the horizontal and vertical directions, as shown in formula (1-2), where the pooling scale factor k m ∈[3, 7]: Four pooling features in the horizontal and vertical directions with consistent spatial resolution are obtained by upsampling And design to use RAM modules Perform channel compression, as shown in formula (3-6); X hw =Concat(X mean ,X max ) (5) U hw =Conv(X hw ) (6) X mean Represents the average value of all channels of the pixel, X max represents the maximum value of the channel where the pixel is located, Conv is a convolution with a convolution kernel size of 1×1, and Concat represents splicing on the channel; secondly, the compressed features are spliced ​​through the channels to obtain the joint feature X of the four-channel multi-scale cross pooling c ; Finally, the spatial feature encoder-decoder SFED is designed to perform nonlinear fusion mapping of feature points at the same spatial position on different channels, as shown in formula (7): In the formula, F ED It is a nonlinear fusion mapping function, which includes a dimension reduction module based on full connection, an activation function ReLU and a dimension increase module based on full connection; F ex Reshape is used to realize the mapping of one-dimensional optimized feature vector to two-dimensional spatial feature map, F fc1 、F fc2 They are the dimensionality reduction process and dimensionality increase process based on full connection respectively; finally, the spatial attention importance weight W is generated by ReLU and Sigmoid activation functions respectively. f , as shown in formula (8): Where σ is the ReLU activation function, is the Sigmoid activation function.

6. According to the method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network as described in claim 3, Features: The "multi-scale feature-level fusion module" consists of three-branch generators of two scales; each branch generator consists of a front-end convolution, three residual blocks in the middle, and a back-end deconvolution; The middle branch small-scale generator combines the underlying multimodal pixel-level fusion features X″ fusion After two times downsampling, the residual module is used to extract high-level features, and the high-level features are element-wise added to the convolutional features of the front end of the upper branch large-scale generator and the lower branch large-scale generator, and then input into the residual block of the corresponding large-scale generator; Finally, the output features of the upper and lower large-scale generators are added to obtain the intermediate output result X of the multi-level feature fusion network CF , X CF Then with the input feature X RGB Perform channel splicing to obtain the output of the "multi-level feature fusion module".

7. According to claim 6, a method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network, Features: The feature extraction process of the middle branch small-scale generator is as shown in formula (9-11): G S =F rct (U DS ) (11) In the above formula, Downsample is a two-fold downsampling operation, Conv 7 It is a convolution operation with a kernel of 7 and a stride of 1. LN represents layer normalization. RB contains three residual blocks connected in series. US represents three deconvolution operations with a kernel of 3 and a stride of 2, a layer normalization and a ReLU activation function operation. rct To include a pair of features U DS Taking the outermost pixel as the symmetry axis, perform a symmetrical mirror filling operation with a value of 3, a convolution with a kernel of 7, and a Tanh function operation; G S is the output feature of the small-scale generator; The input of the upper and lower branch large-scale generators is X FIR and X RGB , the feature fusion process is as follows: G RGB =F rct (U RGB ) (16) G FIR =F rct (U FIR ) (17) In the above formula, Conv 3 It is represented as a convolution with a kernel of 3 and a stride of 2; is the convolution feature obtained after two convolution operations and a ReLU activation function; DC includes one deconvolution with a kernel of 3 and a step size of 2, one layer normalization and a ReLU activation function operation; U RGB , U FIR Features and The features extracted after DC operation, G RGB , G FIR They are the output features of the upper and lower branch large-scale generators respectively; Finally, the output features of the upper and lower branch large-scale generators are added to X by element-wise addition. RGB Perform channel splicing to obtain feature-level fusion results As shown in formula (18-19): X CF =G RGB +G FIR (18) 8. According to the method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network as described in claim 1, Features: The "migration-enhancement" process of the features of the "multi-scale image enhancement module" is as follows: Fusion Features As the input of the "multi-scale image enhancement module", it is first enhanced by the Enhance Block and then combined with the intermediate output feature X of the multi-level feature fusion module CF Perform channel stitching, then perform a second enhancement, and finally output the dehazed image The Enhance Block is an enhancement block based on a pyramid structure. The input feature of the enhancement block is After SE channel attention optimization, downsampling at four scales of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 is performed to construct a four-branch multi-scale pyramid structure. Then, the features after upsampling alignment and channel attention optimization are used for channel splicing, and the enhanced result is output through 3×3 convolution. The process is shown in formula (20-24): Where Conv 3 Indicates the convolution operation with a kernel of 3 and a stride of 1. in is the input of the pyramid structure, is the output of the pyramid structure, F AP (x, b) means performing average pooling of scale b on x; Upsampling represents the upsampling function, and tanh represents the activation function Tanh. and Represent the output features after the first and second enhancements respectively.

9. According to the method for constructing a RGB-FIR multispectral image dehazing model based on a generative adversarial network according to claim 1, Features: The specific steps of S3 are as follows: 3-1 According to the content of S1, through image registration, a data set of RGB-FIR image pairs with the same field of view and resolution is prepared (a total of 11,736 images, including 13 different indoor and outdoor scenes), and the RGB-FIR image pairs under different fog conditions are divided into training sets and test sets in a ratio of 7:3, and TXT files for training and testing are prepared for training; 3-2 Configuration of hardware equipment and experimental environment; The server environment is as follows: Windows 11 system, CPU (Intel Corei7-11800H), GPU (NVIDIA GeForce RTX3060), RAM (32GB), CUDA11.3; Pycharm is used as the programming platform, Python3.9 and pytorch1.10 are used to configure the virtual environment, initialize the model parameters, and set the number of training rounds to 200; 3-3 Design of multi-effect loss function L based on learning strategy of adversarial generative network MF , the RGB-FIR multi-spectral dehazing model with multi-level feature fusion is used as the generator G, and the multi-scale discriminator D is constructed; then, in the process of 200 rounds of iterative training, the weight parameters of the RGB-FIR multi-spectral dehazing model with multi-level feature fusion are corrected and optimized; the network weight file is finally saved as the optimal parameter model; Among them, the multi-effect loss function L MF By adversarial loss L A , feature matching loss L FM , Perceptual loss L VGG and fidelity loss L F Four loss functions are constructed; by L FM , L VGG and L F Different weighting factors are assigned to control the relative importance among the four losses, as shown in Equation (25): L MF =L A +λ 1 L FM +λ 2 L VGG +λ 3 L F (25) The adversarial loss function L A As a multi-scale discriminator D k (k∈[1,2]) Identify multi-level fusion features X at different resolutions CF With clear image Y RGB (true value), identify the multi-level fusion feature X CF And generate images The evaluation criteria for authenticity are as follows (26-27): In the formula, represents the adversarial loss of the multi-scale discriminator, The generated image output by the generator; Indicates the expectations of the sample; The feature matching loss function L FM As a multi-scale discriminator D k (k∈[1,2]) Identify the generated image and the clear image Y output by the generator at different resolutions RGB The authenticity evaluation standard between (true value) is as follows: In the formula is the kth discriminator D k The resulting feature matching loss, T is the total number of layers used for feature extraction, N i is the number of elements in each layer, is the operator for extracting features at the i-th layer, represents the expectation of the sample; The perceptual loss function L VGG Maintaining the consistency between the generated image and the true image from the perspective of perception and semantics makes the generated image more consistent with the human eye's perception of image quality. The definition is as follows: In the formula, It is represented as the final result of the multi-spectral dehazing network output, H i and W i is the height and width of the i-th feature map, C i Indicates the number of channels, C i H i W i represents the size of the feature map of the i-th layer, represents the feature representation of the i-th layer in the VGG network; The fidelity loss function L F Directly use the Euclidean distance to measure the true value image Y RGB And generate images The difference between them is defined as follows:

10. A RGB-FIR multispectral image dehazing model based on generative adversarial network, Features: The method is constructed using any one of claims 1 to 9.

Citation Information

Cited By

  • Generative adversarial model construction method and device for SWE image generation

    CN119850973A

  • Deep learning defogging method based on feature grid affine transformation

    CN120163723A

  • Real-time monitoring system for ultrahigh pressure hose joint assembly

    CN121686346A

  • Cross-modal fusion power transmission line haze enhancement method and system based on depth perception

    CN122820453A