Infrared and visible light image fusion method for inspection robot

Abstract

The invention discloses an infrared and visible light image fusion method for an inspection robot. The infrared and visible light image fusion method specifically comprises the following steps: S1, pre-training an infrared image noise suppression network NANet and a visible light image light intensity sensing network LIPNet; s2, using an IF Block and an EF Block based on Transform to extract image features, and using the IF Block and the EF Block based on Transform to extract image features; s3, calculating an adaptive weight map by using an adaptive weight estimation module (AWEM), and guiding feature retention by using the weight map; and S4, realizing feature reconstruction by using a feature fusion module formed by three convolutional layers, and generating a fused image. According to the infrared and visible light image fusion method for the inspection robot disclosed by the invention, the performance of the fused image in an advanced visual task is effectively improved, and the influence of noise and light pollution on the fusion performance is effectively reduced by performing comparative analysis on a public data set and various mainstream fusion methods from subjective and objective angles; and the salient target and background texture information can still be fully reserved in a severe scene.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to an infrared and visible light image fusion method for an inspection robot. Background Art

[0002] Traditional infrared and visible light image fusion methods mainly include: multi-scale transformation-based methods, subspace-based methods, sparse representation-based methods, saliency representation-based methods and other methods. Although traditional methods have achieved good results, they have certain limitations. For example, the manually designed activity level measurement leads to insufficient extracted image features, and in order to adapt to different and complex scenes, the fusion rules are too complex and the amount of calculation is large.

[0003] With the continuous improvement of computer hardware performance and the rapid development of deep learning technology, researchers have begun to combine deep learning with image fusion technology. In 2017, Wu et al. first combined the deep Boltzmann machine (DBM) with the non-subsampled shearlet transform technology to achieve image fusion. This is the earliest study to apply deep learning technology to the field of image fusion, opening up a new path for image fusion research based on deep learning. Subsequently, a number of IVIF studies based on deep learning have rapidly developed in this field. Existing IVIF methods based on deep learning can be divided into the following five categories: fusion methods based on convolutional neural networks, fusion methods based on autoencoders (AE), fusion methods based on generative adversarial networks, fusion methods based on transformers, and other methods.

[0004] Although the existing methods have achieved good results, they have certain limitations in dealing with the problem of infrared image noise and visible light image light pollution reducing the quality of the fused image. Summary of the invention

[0005] The invention discloses an infrared and visible light image fusion method for an inspection robot, aiming to solve the technical problem that there are certain limitations in processing infrared image noise and visible light image light pollution to reduce the quality of fused images as mentioned in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for fusing infrared and visible light images for an inspection robot comprises the following steps:

[0008] S1, pre-trained infrared image noise suppression network NANet and visible light image light intensity perception network LIPNet;

[0009] S2, use Transformer-based IF Block and EF Block to extract image features;

[0010] S3, using an adaptive weight estimation module (AWEM) to calculate an adaptive weight map, and using the weight map to guide feature retention;

[0011] S4, using a feature fusion module consisting of three convolutional layers to achieve feature reconstruction and generate a fused image;

[0012] S5, using dual discriminators to drive the generation of fusion network optimization;

[0013] S6, use DeepLabV3+ semantic segmentation network to drive the generation of fusion network optimization;

[0014] The generative fusion network includes a noise suppression module (NATM), a light intensity perception module (LIPTM) and an adaptive weight estimation module (AWEM). The noise suppression module performs denoising on infrared images through a pre-trained noise suppression network and extracts infrared image features using a Transformer. The light intensity perception module processes light pollution in visible light images through a light intensity perception network and extracts visible light image features using a Transformer. The adaptive weight estimation module uses a dual-branch structured codec to respectively calculate the weight maps of source images of different modalities, splices the channel dimensions of the weight maps of different modalities, and then uses a densely connected convolutional layer to calculate the adaptive weight map for guiding the feature map extracted by the noise suppression module and the LIPTM module to retain features. The dual discriminator and semantic segmentation driven strategy drives the image fusion network to generate a fused image with better quality through the designed generative adversarial loss and semantic segmentation loss.

[0015] In a preferred solution, the core network of the noise suppression module is a noise suppression network, and the infrared image is denoised to obtain a noisy image as training data:

[0016] IR N =IR+N;

[0017] Among them, IR represents the original infrared image, N is Gaussian noise, IR N To add noise to the noisy image, IR N and IR are used as training data and their corresponding labels respectively, NANet is trained and the denoised image IR′ is obtained. The core part of the light intensity perception module is the light intensity perception network, and the light intensity perception network adopts a supervised training method. Pseudo-strong light pollution is added to the visible light image in the night scene as part of the training data:

[0018] VIS L =VIS+L

[0019] Among them, VIS represents the original visible light image, L represents the pseudo strong light pollution, and VIS L VIS represents a visible light image with pseudo strong light pollution added. L The dataset consists of two parts: visible light images polluted by pseudo strong light and original visible light images without pollution;

[0020] VIS L and VIS are used as training data and their corresponding labels, respectively, to train LIPNet and obtain the optimized visible light image VIS′. Different from the Gaussian noise of infrared images, light pollution is mainly reflected in pixel intensity, contrast and structure, so the structural similarity loss function L is used. SSIM To guide network training, the calculation formula of SSIM is as follows:

[0021]

[0022] in, and They represent the comparison of the original visible light image VIS and the optimized visible light image VIS′ in terms of brightness, contrast and structure, respectively, and their calculation formulas are as follows:

[0023]

[0024] Among them, μ VIS′ and μ VIS denote the mean of VIS′ and VIS, σ VIS′ and σ VIS denote the standard deviation of VIS′ and VIS, σ VIS′,VIS represents the covariance of VIS′ and VIS, C is a constant to prevent division by zero;

[0025] Let k 1 =0.01, k 2 =0.03, L is 255, α, β and γ are parameters to balance each component. In this paper, all three parameters are 1. The structural similarity loss function L SSIM The definition of L is: SSIM =1-SSIM(VIS′,VIS)

[0026] LIPNet uses a codec structure to weaken strong light pollution. The encoder can effectively extract the feature information of the image. The encoder is used to reconstruct and optimize the image from the features. In order to avoid information loss when processing uncontaminated images, a jump connection structure is added between the corresponding layers of the encoder and decoder.

[0027] In a preferred solution, IF Block and EF Block are used in both NATM and LIPTM to extract intra-modal features and extra-modal related features. Given an infrared image feature f with a size of H×W×C, ir and visible light image feature f vi , the shift window mechanism divides it into There are non-overlapping M×M windows. At this time, the size of the input feature f is Calculate the weight parameter matrix for each window separately:

[0028] {Q,K,V}={XW Q ,XW K ,XW V}

[0029] Among them, X is the local window feature after segmentation, W Q , W K and W V For a weight parameter matrix with dimension d, calculate its self-attention:

[0030]

[0031] Where p is the relative position code, and the self-attention is expanded into a Swin Transformer layer (STL) consisting of window multi-head self-attention (WMSA) and moving window multi-head self-attention (SWMSA) as shown in Figure 4 As shown in Figure 2, layer normalization (LN) is located between multi-head self-attention and multi-layer perceptron (MLP). Given different modal local feature windows X 1 and X 2 Calculate the inter-modal output features Y 1 and Y 2 :

[0032]

[0033] Y 1 =LN(WMSA(Q 1 ,K 2 ,V 2 ))+Q 1

[0034] Y 2 =LN(WMSA(Q 2 ,K 1 ,V 1 ))+Q 2

[0035] Each IF Block and EF Block consists of two STLs, and the second STL in the IF Block obtains {Q, K, V} for interaction with the EF Block.

[0036] In a preferred solution, the adaptive weight estimation module uses a dual-branch structure codec to calculate a weight map from the source image to guide the retention of fusion feature information, and uses a pseudo-twin encoder with the same network structure to extract the feature map of the source image; then, the pseudo-twin decoder is used to reconstruct the feature map to obtain a single-modal weight map; considering that the weight maps of different modalities are correlated, their dimensions are spliced ​​and input into the convolutional layer of the dense connection structure for integration, and the final weight map M is obtained after Sigmoid calculation. The calculation formula of the weight map is shown as follows:

[0037] M = AWEM (IR, VIS);

[0038] Where IR and VIS represent the original infrared image and visible light image, respectively. AWEM(·) represents the adaptive weight module, which uses the weight map M to guide the infrared features f extracted from NATM and LIPTM, respectively. ir and visible light characteristics f vi , the process is shown as follows:

[0039]

[0040] Then, the adaptively weighted feature map dimensions are concatenated to obtain a fusion feature f with a channel number of 2. fused , as follows:

[0041] f fused =concat(f i ' r ,f v ' i ).

[0042] In a preferred solution, the dual discriminator and the semantic segmentation network use generative adversarial loss and semantic loss to drive the generative fusion network to produce a fused image with higher quality and more suitable for computer advanced vision tasks. The dual discriminator is composed of an infrared discriminator and a visible light discriminator. Different discriminators have the same network structure but different network parameters. The generative adversarial loss is composed of a generator loss and a discriminator loss:

[0043] L GAN =L Generator +L Discriminator

[0044] Among them, the generator loss L Generator The content loss L content and adversarial loss Ladversarial The composition is as shown below:

[0045] L Generator =L content +L adversarial

[0046] Among them, the content loss L content and adversarial loss L adversarial The calculation formulas are:

[0047]

[0048] L adversarial = E[log(1-D ir (G(IR,VIS)))]+E[log(1-D vi (G(IR,VIS)))]

[0049] The input of the discriminator is the fused image and the source image of its corresponding modality. The infrared discriminator is used to guide the fused image to retain significant thermal target information, and the visible light discriminator is used to guide the fusion result to retain background texture detail information. The loss function of the discriminator is:

[0050]

[0051] DeepLabV3+ is used as the semantic segmentation network, and the cross entropy loss function is used as the semantic segmentation loss. The calculation formula is as follows:

[0052]

[0053] The infrared and visible light image fusion method for inspection robots provided by the present invention has the advantages of using a Transformer-based noise suppression module and a light intensity perception module to suppress redundant information in infrared images and visible light images respectively, and extract features of images of different modalities; secondly, the weight map is calculated by an adaptive weight estimation module of a dual-branch codec structure, which is used to guide the retention of fusion feature information, and the feature is reconstructed by a fusion module to generate a fused image; then, an infrared discriminator and a visible light discriminator are used to constrain the fused image to maintain the same information distribution as the source image; finally, the semantic segmentation loss is used to drive the high-level semantic information back to the fusion network, which effectively improves the performance of the fused image in high-level visual tasks. By comparing and analyzing with a variety of mainstream fusion methods from subjective and objective perspectives on public data sets, the experimental results show that this method effectively reduces the impact of noise and light pollution on fusion performance, and can still fully retain the effect of significant target and background texture information in harsh scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1This is an overall framework diagram of an infrared and visible light image fusion method for an inspection robot proposed by the present invention.

[0055] Figure 2 This is a noise suppression network structure diagram of the infrared and visible light image fusion method for an inspection robot proposed in the present invention.

[0056] Figure 3 This is a light intensity perception network structure diagram of the infrared and visible light image fusion method for an inspection robot proposed by the present invention.

[0057] Figure 4 This is a schematic diagram of the STL structure of an infrared and visible light image fusion method for an inspection robot proposed by the present invention.

[0058] Figure 5 This is a schematic diagram of the IFBlock and EF Block structures of an infrared and visible light image fusion method for an inspection robot proposed in the present invention.

[0059] Figure 6 This is a schematic diagram of the codec structure of the infrared and visible light image fusion method for an inspection robot proposed by the present invention.

[0060] Figure 7 This is a schematic diagram of the discriminator structure of the infrared and visible light image fusion method for an inspection robot proposed by the present invention.

[0061] Figure 8 This is the NANet denoising effect of the infrared and visible light image fusion method for inspection robots proposed in the present invention.

[0062] Fig. 9 This is the LIPNet optimization effect of the infrared and visible light image fusion method for inspection robots proposed in the present invention.

[0063] Fig.10 This is a fusion result diagram of the infrared and visible light image fusion method for an inspection robot proposed in the present invention. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0065] A method for fusion of infrared and visible light images for inspection robots. The overall framework is as follows: Figure 1As shown in the figure, it mainly includes three parts: generative fusion network, dual discriminator network and semantic segmentation driving network. Among them, the generative fusion network is mainly composed of three parts: noise suppression module (NATM) based on Transformer, light intensity perception module (LIPTM) based on Transformer, and adaptive weight estimation module (AWEM). Among them, NATM uses the pre-trained noise suppression network to reduce the noise of infrared images and uses Transformer to extract infrared image features; LIPTM uses the light intensity perception network to process the light pollution phenomenon that may exist in visible light images and uses Transformer to extract visible light image features; AWEM uses the codec with a dual-branch structure to calculate the weight maps of different modal source images respectively, and after splicing the channel dimensions of the weight maps of different modalities, the adaptive weight map is calculated by introducing a densely connected convolutional layer to guide the feature map extracted by NATM and LIPTM modules to retain features; the dual discriminator and semantic segmentation driving strategy drive the image fusion network to generate better quality fused images through the designed generative adversarial loss and semantic segmentation loss.

[0066] In a preferred embodiment, NATM is used to suppress noise and fully extract feature information of infrared images, and the core network of the module is the noise suppression network (Noise Abatement Net, NANet).

[0067] Due to the imaging characteristics of thermal imaging sensors, the background information of infrared images usually contains Gaussian noise. In order to solve this problem, a noise suppression network is designed to remove noise information in infrared images. In order to fully utilize the potential of neural networks in supervised training, first, the infrared image is denoised to obtain a noisy image as training data:

[0068] IR N =IR+N (1)

[0069] Among them, IR represents the original infrared image, N is Gaussian noise, IR N is the noisy image after adding noise.

[0070] IR N and IR are used as training data and their corresponding labels, respectively, to train NANet and obtain the denoised image IR′. Considering that the noise is mainly reflected in the detail information, a gradient-based loss function is designed:

[0071] Lgrad =f(▽IR′,▽IR) (2)

[0072] Where f(·) is the Minkowski distance with p = 2, i.e., L 2 norm, ▽ represents the Sobel operator used to calculate the gradient.

[0073]

[0074] In the above formula, H and W are the height and width of the image respectively, and (i,j) is the pixel position.

[0075] As an auxiliary network, the noise suppression network should have high computational efficiency, so the network structure must be simple and efficient. Based on this, this paper designs a convolutional neural network based on a residual connection structure as a noise suppression network. Figure 2 As shown;

[0076] The use of residual connection structure effectively avoids the possibility of infrared image texture detail information in the noise suppression process, and makes the gradient information more efficiently transmitted in the network, improving the generalization of the noise suppression network. In addition, not using downsampling operations can also effectively avoid information loss, further improving the retention of useful information in the original infrared image.

[0077] In a preferred embodiment, LIPTM is used to sense the location and intensity of light pollution in a visible light image and perform attenuation processing to obtain an optimized visible light image, and then the intra-modality fusion block and the inter-modality fusion block are used to fully extract the visible light image features. The core part of LIPTM is the light intensity perception network (LIPNet).

[0078] Considering that visible light images are more sensitive to strong light pollution such as car headlights in low-light scenes, a light intensity perception network is designed to reduce the impact of possible strong light pollution on the quality of fused images. Similar to the NATM concept, LIPNet also uses a supervised training method. First, pseudo strong light pollution is added to visible light images in night scenes as part of the training data:

[0079] VIS L =VIS+L (5)

[0080] Among them, VIS represents the original visible light image, L represents the pseudo strong light pollution, and VIS L Represents a visible light image with pseudo strong light pollution added.

[0081] Because visible light images are not always captured in low-light scenes or with strong light pollution, LIPNet should have high scene robustness, effectively reduce light pollution when strong light appears in visible light images, and make the processed visible light images have the same information distribution as the original visible light images when there is no pollution in the scene. Therefore, the other part of the training set should be composed of original pollution-free visible light images that have no intersection with the first part of the data. L The dataset consists of two parts: visible light images polluted by pseudo strong light and unpolluted original visible light images.

[0082] VIS L and VIS are used as training data and their corresponding labels, respectively, to train LIPNet and obtain the optimized visible light image VIS′. Different from the Gaussian noise of infrared images, light pollution is mainly reflected in pixel intensity, contrast and structure, so the structural similarity loss function L is used. SSIM To guide network training, the calculation formula of SSIM is as follows:

[0083]

[0084] in, and Respectively represent the comparison of the original visible light image VIS and the optimized visible light image VIS′ in terms of brightness, contrast and structure. The calculation formulas are as follows:

[0085]

[0086]

[0087] Among them, μ VIS′ and μ VIS denote the mean of VIS′ and VIS, σ VIS′ and σ VIS denote the standard deviation of VIS′ and VIS, σ VIS′,VIS represents the covariance of VIS′ and VIS, and C is a constant to prevent division by zero. Let k 1 =0.01, k 2 = 0.03, L is 255, α, β and γ are parameters to balance each component, and all three parameters are 1 in this paper. Structural similarity loss function L SSIM is defined as:

[0088] L SSIM =1-SSIM(VIS′,VIS) (10)

[0089] LIPNet uses an encoder-decoder structure to reduce strong light pollution. The encoder can effectively extract the feature information of the image, and the encoder is used to reconstruct and optimize the image from the features. In order to avoid information loss when processing uncontaminated images, a jump connection structure is added between the corresponding layers of the encoder and decoder. The network structure of LIPNet is as follows Figure 3 shown.

[0090] In a preferred embodiment, IF Block and EF Block are used in both NATM and LIPTM to extract intra-modal features and extra-modal related features. ir and visible light image feature f vi , the shift window mechanism divides it into There are non-overlapping M×M windows. At this time, the size of the input feature f is Calculate the weight parameter matrix for each window separately:

[0091] {Q,K,V}={XW Q ,XW K ,XW V} (11)

[0092] Among them, X is the local window feature after segmentation, W Q , W K and W V is a weight parameter matrix with dimension d.

[0093] Then calculate its self-attention:

[0094]

[0095] Among them, p is the relative position code.

[0096] The self-attention is expanded to a Swin Transformer layer (STL) consisting of window multi-head self-attention (WMSA) and moving window multi-head self-attention (SWMSA) as shown in Figure 4 As shown in Figure 2, layer normalization (LN) is located between multi-head self-attention and multi-layer perceptron (MLP).

[0097] In a preferred embodiment, when extracting features from different modalities, not only the correlation between the features themselves should be considered, but also the correlation between the information between the modalities. In order to further integrate the information between different modalities, an out-of-module fusion block based on the cross-attention mechanism is designed to realize feature extraction based on cross-domain information perception. Given different modal local feature windows X 1 and X 2Calculate the inter-modal output features Y 1 and Y 2 .

[0098]

[0099] Y 1 =LN(WMSA(Q 1 ,K 2 ,V 2 ))+Q 1 (15)

[0100] Y 2 =LN(WMSA(Q 2 ,K 1 ,V 1 ))+Q 2 (16)

[0101] Each IF Block and EF Block consists of two STLs. The second STL in the IF Block obtains {Q, K, V} for interaction with the EF Block, such as Figure 5 shown.

[0102] In a preferred embodiment, the adaptive weight estimation module uses a dual-branch structure codec to calculate the weight map from the source image to guide the retention of fusion feature information. The pseudo-twin encoder with the same network structure is used to extract the feature map of the source image; then, the pseudo-twin decoder is used to reconstruct the feature map to obtain a single-modal weight map; considering that the weight maps of different modalities are correlated, their dimensions are spliced ​​and input into the convolution layer of the dense connection structure for integration, and the final weight map M is obtained through Sigmoid calculation. The structure of the codec is as follows: Figure 6 As shown;

[0103] The calculation formula of the weight graph is as follows:

[0104] M=AWEM(IR,VIS) (17)

[0105] Where IR and VIS represent the original infrared image and visible light image, respectively, and AWEM(·) represents the adaptive weight module.

[0106] The weight map M is used to guide the infrared features f extracted from NATM and LIPTM respectively. ir and visible light characteristics f vi , the process is shown as follows:

[0107]

[0108] Then, the adaptively weighted feature map dimensions are concatenated to obtain a fusion feature f with a channel number of 2.fused As follows:

[0109] f fused =concat(f i ' r ,f v ' i ) (20)

[0110] In a preferred embodiment, the dual discriminator and the semantic segmentation network use generative adversarial loss and semantic loss to drive the generative fusion network to produce a fused image with higher quality and more suitable for computer advanced vision tasks. The dual discriminator is composed of an infrared discriminator and a visible light discriminator. Different discriminators have the same network structure but different network parameters. The structure of the discriminator is as follows: Figure 7 shown.

[0111] In a preferred embodiment, the generative adversarial loss consists of a generator loss and a discriminator loss:

[0112] L GAN =L Generator +L Discriminator (twenty one)

[0113] Among them, the generator loss L Generator The content loss L content and adversarial loss L adversarial The composition is shown in the following formula.

[0114] L Generator =L content +L adversarial (twenty two)

[0115] Among them, the content loss L content and adversarial loss L adversarial The calculation formulas are:

[0116]

[0117] L adversarial = E[log(1-D ir (G(IR,VIS)))]+E[log(1-D vi (G(IR,VIS))(24)

[0118] The input of the discriminator is the fused image and the source image of its corresponding modality. The infrared discriminator is used to guide the fused image to retain significant thermal target information, and the visible light discriminator is used to guide the fusion

[0119] The result retains the background texture details. The loss function of the discriminator is:

[0120]

[0121] DeepLabV3+ is used as the semantic segmentation network, and the cross entropy loss function is used as the semantic segmentation loss. The calculation formula is as follows:

[0122]

[0123] Embodiment 1:

[0124] The present invention proposes a method for fusion of infrared and visible light images based on noise suppression and light intensity perception. The principle of the present invention is: first, pre-train the infrared image noise suppression network and the visible light image light intensity perception network, and use the data enhancement method of random noise addition to produce a training data set, and use gradient loss and structural similarity loss to guide the update and optimization of network parameters. Secondly, the features of different modal images are extracted respectively through the intra-modal and inter-modal feature extraction modules based on Transformer, thereby improving the feature extraction capability of the backbone network. Thirdly, an adaptive weight estimation module with a dual-branch codec structure is introduced to calculate the weight map, which improves the model's retention of complementary information and allows the model to adaptively retain features according to the activity of the source image information. Finally, the generation of a fusion model driven by a dual discriminator and a semantic segmentation network can improve the quality of the fused image and its performance in advanced visual tasks. The overall scheme diagram is shown in the attached figure. Figure 1 shown.

[0125] The specific steps include:

[0126] S1, pre-trained infrared image noise suppression network NANet and visible light image light intensity perception network LIPNet;

[0127] S2, use Transformer-based IF Block and EF Block to extract image features;

[0128] S3, using an adaptive weight estimation module (AWEM) to calculate an adaptive weight map, and using the weight map to guide feature retention;

[0129] S4, using a feature fusion module consisting of three convolutional layers to achieve feature reconstruction and generate a fused image;

[0130] S5, using dual discriminators to drive the generation of fusion network optimization;

[0131] S6. Use DeepLabV3+ semantic segmentation network to drive the generation of fusion network optimization.

[0132] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for fusion of infrared and visible light images for inspection robots, comprising a generative fusion network, a dual discriminator network and a semantic segmentation drive network, characterized in that: The specific steps include: S1, pre-trained infrared image noise suppression network NANet and visible light image light intensity perception network LIPNet; S2, use Transformer-based IF Block and EF Block to extract image features; S3, using an adaptive weight estimation module (AWEM) to calculate an adaptive weight map, and using the weight map to guide feature retention; S4, using a feature fusion module consisting of three convolutional layers to achieve feature reconstruction and generate a fused image; S5, using dual discriminators to drive the generation of fusion network optimization; S6, use DeepLabV3+ semantic segmentation network to drive the generation of fusion network optimization; The generated fusion network includes a noise suppression module (NATM), a light intensity perception module (LIPTM) and an adaptive weight estimation module (AWEM). The noise suppression module performs noise reduction processing on infrared images through a pre-trained noise suppression network and uses a Transformer to extract infrared image features; the light intensity perception module processes light pollution in visible light images through a light intensity perception network and uses a Transformer to extract visible light image features; the adaptive weight estimation module uses a codec with a dual-branch structure to respectively calculate the weight maps of source images of different modalities, splices the channel dimensions of the weight maps of different modalities, and uses a densely connected convolutional layer to calculate the adaptive weight map for guiding the feature map extracted by the noise suppression module and the LIPTM module to retain features; The dual discriminator and semantic segmentation driven strategy drives the image fusion network to generate better quality fused images through the designed generative adversarial loss and semantic segmentation loss.

2. The infrared and visible light image fusion method for an inspection robot according to claim 1, characterized in that: The core network of the noise suppression module is the noise suppression network, which adds noise to the infrared image to obtain a noisy image as training data: GO N =IR+N; Among them, IR represents the original infrared image, N is Gaussian noise, IR N To add noise to the noisy image, IR N and IR are used as training data and their corresponding labels respectively, NANet is trained and the denoised image IR′ is obtained.

3. The infrared and visible light image fusion method for an inspection robot according to claim 2, characterized in that: The core part of the light intensity perception module is the light intensity perception network, and the light intensity perception network adopts a supervised training method, adding pseudo strong light pollution to the visible light images in night scenes as part of the training data: SCREW L =VIS+L Among them, VIS represents the original visible light image, L represents the pseudo strong light pollution, and VIS L VIS represents a visible light image with pseudo strong light pollution added. L The dataset consists of two parts: visible light images polluted by pseudo strong light and original visible light images without pollution; VIS L and VIS are used as training data and their corresponding labels, respectively, to train LIPNet and obtain the optimized visible light image VIS′. Different from the Gaussian noise of infrared images, light pollution is mainly reflected in pixel intensity, contrast and structure, so the structural similarity loss function L is used. SSIM To guide network training, the calculation formula of SSIM is as follows: in, and They represent the comparison of the original visible light image VIS and the optimized visible light image VIS′ in terms of brightness, contrast and structure, respectively, and their calculation formulas are as follows: Among them, μ VIS′ and μ VIS denote the mean of VIS′ and VIS, σ VIS′ and σ VIS denote the standard deviation of VIS′ and VIS, σ VIS′,VIS represents the covariance of VIS′ and VIS, C is a constant to prevent division by zero; Let k1 = 0.01, k2 = 0.03, L = 255, α, β and γ are the parameters to balance each component. In this paper, all three parameters are 1. The structural similarity loss function L SSIM is defined as: L SSIM =1-SSIM(VIS′,VIS) LIPNet uses a codec structure to weaken strong light pollution. The encoder can effectively extract the feature information of the image, and the encoder is used to reconstruct and optimize the image from the features.

4. The infrared and visible light image fusion method for an inspection robot according to claim 3, characterized in that: Both NATM and LIPTM use IFBlock and EF Block to extract intra-modal features and extra-modal related features. Given an infrared image feature f with a size of H×W×C ir and visible light image feature f vi , the shift window mechanism divides it into There are non-overlapping M×M windows. At this time, the size of the input feature f is Calculate the weight parameter matrix for each window separately: {Q,K,V}={XW Q ,XW K ,XW V } Among them, X is the local window feature after segmentation, W Q , W K and W V is a weight parameter matrix with dimension d.

5. The infrared and visible light image fusion method for an inspection robot according to claim 4, characterized in that: Calculate its self-attention: Among them, p is the relative position encoding, the self-attention is expanded to a Swin Transformer layer (STL) consisting of window multi-head self-attention (WMSA) and moving window multi-head self-attention (SWMSA), and the layer normalization is located between the multi-head self-attention and the multi-layer perceptron.

6. The infrared and visible light image fusion method for an inspection robot according to claim 5, characterized in that: Given different modal local feature windows X1 and X2, calculate their inter-modal output features Y1 and Y2 respectively: Y1=LN(WMSA(Q1,K2,V2))+Q1 Y2=LN(WMSA(Q2,K1,V1))+Q2 Each IF Block and EF Block consists of two STLs, and the second STL in the IF Block obtains {Q, K, V} for interaction with the EF Block.

7. The infrared and visible light image fusion method for an inspection robot according to claim 6, characterized in that: The adaptive weight estimation module uses a dual-branch codec to calculate the weight map from the source image to guide the retention of fusion feature information, and uses a pseudo-twin encoder with the same network structure to extract the feature map of the source image; then, the pseudo-twin decoder is used to reconstruct the feature map to obtain a single-modal weight map; considering that the weight maps of different modalities are correlated, their dimensions are spliced ​​and input into the convolutional layer of the densely connected structure for integration, and the final weight map M is obtained after Sigmoid calculation.

8. The infrared and visible light image fusion method for an inspection robot according to claim 7, characterized in that: The calculation formula of the weight map is shown as follows: M = AWEM (IR, VIS); Where IR and VIS represent the original infrared image and visible light image, respectively. AWEM(·) represents the adaptive weight module, which uses the weight map M to guide the infrared features f extracted from NATM and LIPTM, respectively. ir and visible light characteristics f vi , the process is shown as follows: Then, the adaptively weighted feature map dimensions are concatenated to obtain a fusion feature f with a channel number of 2. fused , as follows: f fused =concat(f i ′ r ,f v ′ i )。 9. The infrared and visible light image fusion method for an inspection robot according to claim 8, characterized in that: The dual discriminator and semantic segmentation network use generative adversarial loss and semantic loss respectively to drive the generative fusion network to produce a fused image with higher quality and more suitable for computer advanced vision tasks. The dual discriminator consists of an infrared discriminator and a visible light discriminator. Different discriminators have the same network structure but different network parameters.

10. The infrared and visible light image fusion method for an inspection robot according to claim 9, characterized in that: Generative adversarial loss consists of generator loss and discriminator loss: L GAN =L Generator +L Discriminator Among them, the generator loss L Generator The content loss L content and adversarial loss L adversarial The composition is as shown below: L Generator =L content +L adversarial Among them, the content loss L content and adversarial loss L adversarial The calculation formulas are: L adversarial =E[log(1-D ir (G(IR,VIS)))]+E[log(1-D vi (G(IR,VIS)))] The input of the discriminator is the fused image and the source image of its corresponding modality. The infrared discriminator is used to guide the fused image to retain significant thermal target information, and the visible light discriminator is used to guide the fusion result to retain background texture detail information. The loss function of the discriminator is: DeepLabV3+ is used as the semantic segmentation network, and the cross entropy loss function is used as the semantic segmentation loss. The calculation formula is as follows:

Images