Infrared and visible light image fusion method based on illumination adaptation and attention guidance

Through the light adaptation and attention-guided method, the characteristics of infrared and visible light images are extracted, which solves the problems of unbalanced brightness and blurred edges of the fused image, improves the clarity and visual effects of the image, and is suitable for a variety of scenes.

CN118918019APending Publication Date: 2024-11-08EAST CHINA UNIV OF SCI & TECH

Patent Information

Application Number
CN202410966693.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-18
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The existing infrared and visible light image fusion methods have problems such as missing unique features of infrared visible light images, imbalance of brightness of the fusion image and blurred edge contours of the fused image. The traditional method requires a lot of manual rule setting, and the image quality is average, so it cannot adapt to multiple scenarios. use.

Method used

Using a method based on light adaptation and attention guidance, by acquiring visible light and infrared images, segmentation processing and feature extraction, the light environment adaptation network and feature reconstruction network are used to generate the final fusion image, and through the attention mechanism and residual network Extract high-frequency features, complement and fusion of feature information, and use loss function constraints to generate clear fusion images.

Benefits of technology

It improves the clarity and visual effects of infrared visible light fusion images, solves the problems of brightness imbalance and blurred edge contours, enhances feature extraction capabilities, and generates clearer and more informative fusion images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918019A_ABST
    Figure CN118918019A_ABST
Patent Text Reader

Abstract

The invention discloses an infrared and visible light image fusion method based on illumination adaptation and attention guidance. The method comprises the following steps: S1, obtaining a visible light image of a target image and a corresponding infrared image; s2, performing segmentation processing on the visible light image and the infrared image, and dividing the visible light image by setting a brightness threshold; s3, sending the divided visible light image into a light environment adaptation network, and training to obtain a light environment adaptation weight; s4, sending the segmented visible light image and the infrared image into a feature extraction network in a one-to-one correspondence manner, extracting high-frequency common features and unique features of the visible light image and the infrared image, and performing feature information complementation on the visible light image and the infrared image; and S5, sending the final features of the visible light image and the infrared image into a feature reconstruction network to generate a final fusion image. The defects in the prior art can be overcome, and the definition and the visual effect of the infrared visible light fusion image are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to an infrared and visible light image fusion method based on illumination adaptation and attention guidance. Background Art

[0002] The purpose of image fusion is to generate a more information-rich fused image by combining the important contents of the source images, thereby making up for the defect that a single-modality image cannot fully represent all the information in a complex scene. The sub-branch of infrared and visible light image fusion aims to achieve the complementarity and enhancement of multi-modal image information.

[0003] Due to their different physical properties and imaging mechanisms, infrared images and visible light images retain significant differences in the content of the same scene. Visible light images can provide visual detail information such as target morphology and can vividly depict objects under normal lighting conditions. However, visible light images are easily affected by factors such as extreme weather, resulting in the loss of key information. In conditions such as low light intensity or smoke obstruction, the reliability of the information provided by the image is low. In contrast, infrared images can effectively resist interference from extreme conditions, capture the thermal radiation of the target, and have advantages in anti-interference and night imaging.

[0004] The traditional method of infrared and visible light image fusion is effective, but it requires a lot of manual rule setting, and the obtained image quality is average, which cannot be used in a variety of scenarios. The current deep learning image fusion algorithm effectively makes up for the defects of the traditional method, has a strong feature extraction capability, and is suitable for images in a variety of scenarios. However, the current deep learning method still has problems such as missing unique features of infrared and visible light images, unbalanced brightness of the fused image, and blurred edge contours of the fused image. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide an infrared and visible light image fusion method based on illumination adaptation and attention guidance, which can solve the shortcomings of the prior art and improve the clarity and visual effect of the infrared and visible light fusion image.

[0006] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0007] A method for fusion of infrared and visible light images based on illumination adaptation and attention guidance comprises the following steps:

[0008] S1, obtaining a visible light image and a corresponding infrared image of a target image;

[0009] S2, performing segmentation processing on the visible light image and the infrared image to obtain segmented visible light image and infrared image, and dividing the visible light image by setting a brightness threshold;

[0010] S3, sending the divided visible light image to the light environment adaptation network, and training to obtain the light environment adaptation weight;

[0011] S4, the segmented visible light image and infrared image are sent to the feature extraction network one by one, and the high-frequency common features and unique features of the visible light image and the infrared image are deeply extracted by using the residual network, and the feature information of the visible light image and the infrared image is complemented by the attention guidance mechanism;

[0012] S5. Send the final features of the visible light image and the infrared image to the feature reconstruction network, use the light environment adaptation weights obtained in step S3 to guide the feature fusion process of the visible light image and the infrared image, assign fusion weights to them, and use the loss function constraints to generate the final fused image.

[0013] Preferably, the target image dataset in step S1 consists of an M3FD dataset containing visible light RGB color images, a RoadScene dataset and an MSRS dataset; part of the M3FD dataset is used as a training set for the network model, and the RoadScene dataset, the MSRS dataset and the rest of the M3FD dataset are used to test the performance of the fused image generated by the model.

[0014] Preferably, in step S2, dividing the visible light image comprises the following steps:

[0015] Step S21, obtaining the height and width of the visible light image and the infrared image;

[0016] Step S22, segment the infrared image and the visible light image in terms of size, and the size of each image after segmentation is a pixels × a pixels, and the step length of each segmentation is b pixels, where a and b are both constant parameters;

[0017] Step S23: Calculate the average brightness of the segmented visible light image.

[0018]

[0019] Where L represents the average brightness, W represents the width of the image, H represents the height of the image, R(x,y), G(x,y), and B(x,y) represent the RGB components of the image at position (x,y) respectively; the brightness threshold is set to γ, where γ is a constant parameter. Images above the brightness threshold are classified as bright images, and images below the brightness threshold are classified as dark images.

[0020] Step S24, dividing the bright image in step S23 again, and classifying it into a daytime bright image and a nighttime bright image according to whether the image environment scene is in daytime or nighttime;

[0021] Step S25, the dark image in step S23 is divided again, and is classified into a day dark image and a night dark image according to whether the image environment scene is in daytime or night.

[0022] Preferably, in step S3, the light environment adaptation network consists of three convolution blocks with attention mechanisms, an average pooling layer and a fully connected layer; each convolution block contains a two-dimensional convolution, a batch normalization process, two attention guidance modules and a maximum pooling layer.

[0023] Preferably, the calculation method of the light environment adaptation weight is:

[0024] When fusing images, the light environment adaptation weights of visible light images and infrared images are:

[0025]

[0026]

[0027] Among them, W vis With W ir They represent the light environment adaptability weights guiding the fusion weights of visible light images and infrared images; α and β are constant parameters; P D With P N Respectively represent the number of images in daytime scenes and the number of images in night scenes, and the calculation method is:

[0028]

[0029]

[0030] in, and They represent the number of images classified as bright during the day and dark during the day in the dataset after being classified by the light environment adaptation network. and They represent the number of images classified as bright night and dark night in the dataset after being classified by the light environment adaptation network.

[0031] Preferably, in step S4, the feature extraction network includes:

[0032] A feature extraction module is used to extract the features of infrared images and visible light images;

[0033] The cross-modal differential attention mechanism complementary module is used to exchange feature information of infrared images and visible light images, enhance common features and retain their respective unique features.

[0034] As a preferred method, the feature extraction module consists of two branches, namely the infrared image feature extraction branch and the visible light image feature extraction branch; the two branches have the same structure, consisting of a 1×1 convolution layer and 5 layers of residual blocks containing a full-scale information aggregation mechanism; the cross-modal difference attention mechanism complementary module is located between the infrared image feature extraction branch and the visible light image feature extraction branch. First, the high-frequency features of the two images are extracted through the defined attention mechanism. The definition algorithm is as follows:

[0035] A=C2(R(C1(L(x))))S,

[0036] Where A represents the feature extraction module function, x represents the input feature map, C1 represents the convolution layer that reduces the number of feature map channels by half, C2 represents the convolution layer that reduces the number of feature map channels to 1, R represents the ReLU activation function, S represents the Sigmoid activation function, and L is the Laplacian operator;

[0037] After extracting the features, the attention weights calculated by the Sigmoid function are used to complement the feature information of the two images, and then the complementary features are added to the original features to obtain the final output features. The specific algorithm is defined as follows:

[0038] VI out =VI+(IR⊙A(IR))

[0039] IR out =IR+(VI⊙A(VI)),

[0040] Among them, VI out With IR out represents the output feature map of the visible light image and the infrared image after feature complementation, VI and IR represent the originally extracted visible light feature map and infrared feature map, A represents the feature extraction module function, and ⊙ represents element-by-element multiplication.

[0041] Preferably, in step S5, the feature reconstruction network is composed of 5 residual blocks including a full-scale information aggregation mechanism; the specific algorithm of the full-scale information aggregation mechanism is as follows:

[0042] Four adaptive average pooling layers are used, corresponding to pooling window sizes of 1×1, 2×2, 3×3 and 6×6, to capture feature information of different scales. The output of each adaptive average pooling layer is further reduced to one-quarter of the original channel through a 1×1 convolution layer, and batch normalization and PReLu activation are applied. The feature maps of different scales are then restored to the original input size through bilinear upsampling operations, and then merged in the channel dimension to form a feature image with 4 times the original number of channels. Finally, a 1×1 convolution layer is passed to reduce the number of channels to one-quarter, and batch normalization and PReLu activation are used to obtain a feature map of the same size as the original feature map.

[0043] Preferably, the specific algorithm for defining the loss function in step S5 is:

[0044] L total =λ1·L IW +λ2·L Al +λ3·L Tl ,

[0045] Among them, λ1, λ2 and λ3 are constant coefficients of each term of the loss function, L IW is the comprehensive light environment intensity loss of infrared image and visible light image, L Al is the auxiliary intensity loss of the light environment, L Tl is the image texture loss.

[0046] As a preference, the specific algorithms for the comprehensive light environment intensity loss of infrared images and visible light images, the light environment auxiliary intensity loss, and the image texture loss are defined as:

[0047] L IW =w ir ·L IR +w vis ·L VIS

[0048]

[0049]

[0050] Among them, w vis With w ir is the light environment adaptation weight of visible light image and infrared image, where M and N represent the width and height of the image, and I f ,I i with I v represent the fused image, infrared image and visible light image respectively, θ(·) represents the smooth L1 loss function, represents the gradient operator, L IR With L VISare the intensity losses of infrared images and visible light images respectively, and the specific algorithm is defined as:

[0051]

[0052]

[0053] In the formula, θ(·) represents the smooth L1 loss function, which is expressed as follows:

[0054]

[0055] Where x represents the numerical difference between the predicted box and the true box.

[0056] The beneficial effects brought about by adopting the above technical solution are:

[0057] 1. By designing the visible light image illumination preprocessing method and the light environment adaptation network, the light brightness information in the visible light image is extracted, so that the fusion strategy of the fused image can be changed according to different lighting environments, thus compensating for the unbalanced and uneven image brightness after the fusion of infrared and visible light images.

[0058] 2. In the design of the residual block, a module with a full-scale information aggregation mechanism is added (different from the traditional residual network module). The pyramid pooling method is used to aggregate the global context information of different regions, and pools the different sub-regions of the feature map at each level to obtain context information from local to global, effectively capturing the multi-scale features of the scene. Therefore, in this residual network, the feature distortion problem caused by jump connections in the current residual network is effectively alleviated, and the overall feature extraction performance of the residual network is enhanced.

[0059] 3. By using a high-pass filter (here is the Laplacian operator) to extract the high-frequency features of the two images and then interact and supplement the information, while making up for the defect that the traditional attention mechanism is prone to over-smoothing and losing the high-frequency features of the image, it also improves the feature extraction ability of the network, making the network pay more attention to the salient information in the image and retaining more image texture details, thereby generating a clearer and more information-rich fusion image. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a flow chart of the present invention.

[0061] Figure 2 It is a structural schematic diagram of the light environment adaptation network of the present invention.

[0062] Figure 3 It is a schematic diagram of the structure of the fusion network of the present invention.

[0063] Figure 4It is a schematic diagram of the full-scale information aggregation mechanism module of the present invention. DETAILED DESCRIPTION

[0064] like Figure 1 As shown, a specific embodiment of the present invention includes the following steps:

[0065] Step S1, obtaining a visible light image and a corresponding infrared image of a target image.

[0066] The target image datasets are the M3FD dataset, RoadScene dataset, and MSRS dataset, which contain visible light RGB color images with obvious brightness differences. Most of the M3FD dataset is used as a training set for the network model, and the RoadScene dataset, MSRS dataset, and part of the M3FD dataset are used to test the performance of the fused image generated by the model.

[0067] The ratio of the M3FD dataset used for the light environment adaptation network and the fusion network is 2:8, and for both networks, the ratio of the training set to the test set is 9:1.

[0068] Step S2, segment the visible light image and the infrared image to expand the data volume of the training set and achieve the effect of data enhancement, so as to facilitate the subsequent network training to obtain better results. After obtaining the segmented visible light image and infrared image, calculate the average brightness value of the segmented visible light image, and set the brightness threshold to divide the visible light image according to the average brightness value.

[0069] The specific implementation steps of image segmentation are as follows:

[0070] Step S21, obtaining the height and width of the visible light image and the infrared image.

[0071] Step S22, the infrared image and the visible light image are segmented in size. After segmentation, the size of each image is a pixels × a pixels. The step length of each segmentation is b pixels. Both a and b are constant parameters, where a=64 and b=64.

[0072] Step S23: Calculate the average brightness of the segmented visible light image.

[0073]

[0074] Where L represents the average brightness, W represents the width of the image, H represents the height of the image, R(x,y), G(x,y), and B(x,y) represent the RGB components of the image at position (x,y), respectively; the brightness threshold is set to γ, where γ is a constant parameter. Images above the brightness threshold are classified as bright images, and images below the brightness threshold are classified as dark images; here γ = 0.3.

[0075] Step S24, dividing the bright image in step S23 again, and classifying it into a daytime bright image and a nighttime bright image according to whether the image environment scene is in daytime or nighttime;

[0076] Step S25, the dark image in step S23 is divided again, and is classified into a day dark image and a night dark image according to whether the image environment scene is in daytime or night.

[0077] In step S3, the four types of visible light images classified in S2 and labeled accordingly are sent to the light environment adaptation network for training to obtain the light environment adaptation weights.

[0078] like Figure 2 As shown in the figure, the light environment adaptation network includes the following structure: three convolution blocks with attention mechanisms, which are used to maximize the extraction of the brightness and darkness features of the image. Each convolution block contains a two-dimensional convolution, a batch normalization process, two attention guidance modules and a maximum pooling layer; an average pooling layer, which is used to reduce the dimension of the data by averaging the feature maps to reduce the risk of overfitting; and a fully connected layer, which is used to complete the final image classification task. Among the attention guidance modules, one is a general SE Net module, which improves the representation ability of the convolutional network by explicitly modeling the dependencies between the convolutional feature channels; the other is a general ECA-Net module, which does not require dimensionality increase or decrease for the features, and by capturing the dynamic relationship between channels, enables the network to more accurately focus on the features that are more important to the current task.

[0079] By putting the visible light training set of the fusion network into the trained light environment adaptation network, the light environment adaptation weight can be obtained. The specific calculation method is as follows:

[0080]

[0081]

[0082] Among them, W vis With W ir They represent the light environment adaptability weights guiding the fusion weights of visible light images and infrared images; α and β are constant parameters; P D With P N Respectively represent the number of images in daytime scenes and the number of images in night scenes, and the calculation method is:

[0083]

[0084]

[0085] in, and They represent the number of images classified as bright during the day and dark during the day in the dataset after being classified by the light environment adaptation network. and They represent the number of images classified as bright night and dark night in the dataset after being classified by the light environment adaptation network.

[0086] like Figure 3 Shown is a schematic diagram of the structure of the fusion network in the present invention, which consists of a feature extraction network and a feature reconstruction network.

[0087] Step S4, the two segmented images are sent to the feature extraction network in the fusion network one by one, and the common features and unique features of the two images are deeply extracted using the residual network, and the feature information of the two images is complemented by the attention guidance mechanism. The feature extraction network in the fusion network is composed of a feature extraction module and a cross-modal difference attention mechanism complementary module. The feature extraction module is used to extract the respective features of the infrared image and the visible light image; the cross-modal difference attention mechanism complementary module is used to communicate the feature information of the infrared image and the visible light image, enhance the common features and retain their respective unique features.

[0088] The specific structure of the feature extraction module is as follows: it has two branches, namely the infrared image feature extraction branch and the visible light image feature extraction branch; the two branches have the same structure, consisting of a 1×1 convolution layer (Conv) and a 4-layer residual block (Resblock); the 1*1 convolution layer is used for data dimensionality reduction to reduce the complexity of the network; the 4-layer residual block is used for centralized feature extraction, and the features that are easy to be missed are supplemented through the residual structure. Each residual block includes 3 LeakyReLU activation functions and 1 full-scale information aggregation mechanism module.

[0089] like Figure 4As shown in Figure 1, the specific process of the full-scale information aggregation mechanism module is as follows: four adaptive average pooling layers are used, corresponding to pooling window sizes of 1×1, 2×2, 3×3 and 6×6, to capture feature information of different scales. The output of each adaptive average pooling layer is further reduced to one-fourth of the original channel through a 1×1 convolution layer, and batch normalization and PReLu activation are applied. The feature maps of different scales are then restored to the original input size through bilinear upsampling operations, and then merged in the channel dimension to form a feature image with 4 times the number of original channels. Finally, a 1×1 convolution layer is passed to reduce the number of channels to one-fourth, and batch normalization and PReLu activation are used to obtain a feature map with the same size as the original feature map. The image passes through the residual network module with a full-scale information aggregation mechanism. After the pyramid pooling process, the residual network will capture global context information of different scales, effectively capture the multi-scale features of the scene, improve the feature extraction performance of the residual network, and effectively alleviate the feature distortion of the traditional residual network.

[0090] The specific structure and algorithm of the cross-modal difference attention mechanism complementary module are as follows: It is located between the infrared image feature extraction branch and the visible light image feature extraction branch. First, the high-frequency features of the two images are extracted through the defined attention mechanism. The specific algorithm defined is:

[0091] A=C2(R(C1(L(x))))S,

[0092] Where A represents the feature extraction module function, x represents the input feature map, C1 represents the convolution layer that reduces the number of feature map channels by half, C2 represents the convolution layer that reduces the number of feature map channels to 1, R represents the ReLU activation function, S represents the Sigmoid activation function, and L is the Laplacian operator;

[0093] After extracting the features, the attention weights calculated by the Sigmoid function are used to complement the feature information of the two images, and then the complementary features are added to the original features to obtain the final output features. The specific algorithm is defined as follows:

[0094] VI out =VI+(IR⊙A(IR))

[0095] IR out =IR+(VI⊙A(VI)),

[0096] Among them, VI out With IR out represents the output feature map of the visible light image and the infrared image after feature complementation, VI and IR represent the originally extracted visible light feature map and infrared feature map, A represents the feature extraction module function, and ⊙ represents element-by-element multiplication.

[0097] In step S5, the final features of the visible light image and the infrared image obtained after extraction are sent to the feature reconstruction network in the fusion network, and the light environment adaptability weight obtained in step S3 is used to guide the feature fusion process of the visible light image and the infrared image, and the fusion proportion is assigned to them. The loss function constraint is used to generate the final fused image. The specific structure of the feature reconstruction network in the fusion network in step S5 consists of 5 residual blocks containing a full-scale information aggregation mechanism, and its structure is consistent with the residual block structure described in S4.

[0098] The specific algorithm for loss function definition is:

[0099] L total =λ1·L IW +λ2·L Al +λ3·L Tl ,

[0100] Among them, λ1, λ2 and λ3 are constant coefficients of each term of the loss function, L IW is the comprehensive light environment intensity loss of infrared image and visible light image, L Al is the auxiliary intensity loss of the light environment, L Tl is the image texture loss.

[0101] The specific algorithms for the comprehensive light environment intensity loss, light environment auxiliary intensity loss, and image texture loss of infrared images and visible light images are defined as:

[0102] L IW =w ir ·L IR +w vis ·L VIS

[0103]

[0104]

[0105] Among them, w vis With w ir is the light environment adaptation weight of visible light image and infrared image, where M and N represent the width and height of the image, and I f ,I i with I v represent the fused image, infrared image and visible light image respectively, θ(·) represents the smooth L1 loss function, represents the gradient operator, L IR With L VIS are the intensity losses of infrared images and visible light images respectively, and the specific algorithm is defined as:

[0106]

[0107]

[0108] In the formula, θ(·) represents the smooth L1 loss function, which is expressed as follows:

[0109]

[0110] Where x represents the numerical difference between the predicted box and the true box.

[0111] Comparative experiment

[0112] The present invention is quantitatively tested and compared with the existing infrared and visible light image fusion methods. Specifically, the methods used for comparison are all newer deep learning methods in recent years: DIVFusion, GANMcC, PIAFusion, STDFusionNet, FusionGAN. The evaluation indicators used for comparison are 6 common image objective evaluation indicators, including visual fidelity VIF, mutual information MI, spatial frequency SF, difference correlation synthesis SCD, mean square error MSE, and edge information-based indicator Qabf. Ten images in M3FD were randomly selected, and the average values ​​of the above indicators were taken for comparison.

[0113] Table 1 Objective performance of different fusion methods on the M3FD dataset

[0114]

[0115] The bold data in the table are the best. It can be seen from the table that the fused image obtained by the method in this paper has excellent texture effect and good visual fidelity. It can also obtain more feature information on the edge of the image, with rich details and high contrast.

[0116] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for fusion of infrared and visible light images based on illumination adaptation and attention guidance, characterized in that The following steps are involved: S1, obtaining a visible light image and a corresponding infrared image of a target image; S2, performing segmentation processing on the visible light image and the infrared image to obtain segmented visible light image and infrared image, and dividing the visible light image by setting a brightness threshold; S3, sending the divided visible light image to the light environment adaptation network, and training to obtain the light environment adaptation weight; S4, the segmented visible light image and infrared image are sent to the feature extraction network one by one, and the high-frequency common features and unique features of the visible light image and the infrared image are deeply extracted by using the residual network, and the feature information of the visible light image and the infrared image is complemented by the attention guidance mechanism; S5. Send the final features of the visible light image and the infrared image to the feature reconstruction network, use the light environment adaptation weights obtained in step S3 to guide the feature fusion process of the visible light image and the infrared image, assign fusion weights to them, and use the loss function constraints to generate the final fused image.

2. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 1, characterized in that: In step S1, the target image dataset consists of the M3FD dataset, the RoadScene dataset, and the MSRS dataset containing visible light RGB color images; part of the M3FD dataset is used as a training set for the network model, and the RoadScene dataset, the MSRS dataset, and the rest of the M3FD dataset are used to test the performance of the fused image generated by the model.

3. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 1, characterized in that: In step S2, dividing the visible light image includes the following steps: Step S21, obtaining the height and width of the visible light image and the infrared image; Step S22, segment the infrared image and the visible light image in terms of size, and the size of each image after segmentation is a pixels × a pixels, and the step length of each segmentation is b pixels, where a and b are both constant parameters; Step S23: Calculate the average brightness of the segmented visible light image. Where L represents the average brightness, W represents the width of the image, H represents the height of the image, R(x,y), G(x,y), and B(x,y) represent the RGB components of the image at position (x,y) respectively; the brightness threshold is set to γ, where γ is a constant parameter. Images above the brightness threshold are classified as bright images, and images below the brightness threshold are classified as dark images. Step S24, dividing the bright image in step S23 again, and classifying it into a daytime bright image and a nighttime bright image according to whether the image environment scene is in daytime or nighttime; Step S25, the dark image in step S23 is divided again, and is classified into a day dark image and a night dark image according to whether the image environment scene is in daytime or night.

4. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 1, characterized in that: In step S3, the light environment adaptation network consists of three convolution blocks with attention mechanisms, an average pooling layer and a fully connected layer; each convolution block contains a two-dimensional convolution, a batch normalization process, two attention guidance modules and a maximum pooling layer.

5. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 4 is characterized in that: The calculation method of light environment adaptation weight is: When fusing images, the light environment adaptation weights of visible light images and infrared images are: Among them, W vis With W ir They represent the light environment adaptability weights guiding the fusion weights of visible light images and infrared images; α and β are constant parameters; P D With R N Respectively represent the number of images in daytime scenes and the number of images in night scenes, and the calculation method is: in, and They represent the number of images classified as bright during the day and dark during the day in the dataset after being classified by the light environment adaptation network. and They represent the number of images classified as bright night and dark night in the dataset after being classified by the light environment adaptation network.

6. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 1, characterized in that: In step S4, the feature extraction network includes: A feature extraction module is used to extract the features of infrared images and visible light images; The cross-modal differential attention mechanism complementary module is used to exchange feature information of infrared images and visible light images, enhance common features and retain their respective unique features.

7. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 6, characterized in that: The feature extraction module consists of two branches, namely, infrared image feature extraction branch and visible light image feature extraction branch; The two branches have the same structure, consisting of a 1×1 convolutional layer and 5 residual blocks containing a full-scale information aggregation mechanism; the cross-modal difference attention mechanism complementary module is located between the infrared image feature extraction branch and the visible light image feature extraction branch. First, the high-frequency features of the two images are extracted through the defined attention mechanism. The definition algorithm is as follows: A=C2(R(C1(L(x))))S, Where A represents the feature extraction module function, x represents the input feature map, C1 represents the convolution layer that reduces the number of feature map channels by half, C2 represents the convolution layer that reduces the number of feature map channels to 1, R represents the ReLU activation function, S represents the Sigmoid activation function, and L is the Laplacian operator; After extracting the features, the attention weights calculated by the Sigmoid function are used to complement the feature information of the two images, and then the complementary features are added to the original features to obtain the final output features. The specific algorithm is defined as follows: WE out =VI+(IR⊙A(IR)) AND out =IR+(VI⊙A(VI)), Among them, VI out With IR out represents the output feature map of the visible light image and the infrared image after feature complementation, VI and IR represent the originally extracted visible light feature map and infrared feature map, A represents the feature extraction module function, and ⊙ represents element-by-element multiplication.

8. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 1, characterized in that: In step S5, the feature reconstruction network consists of 5 residual blocks including the full-scale information aggregation mechanism; the specific algorithm of the full-scale information aggregation mechanism is as follows: Four adaptive average pooling layers are used, corresponding to pooling window sizes of 1×1, 2×2, 3×3 and 6×6, to capture feature information of different scales. The output of each adaptive average pooling layer is further reduced to one-quarter of the original channel through a 1×1 convolution layer, and batch normalization and PReLu activation are applied. The feature maps of different scales are then restored to the original input size through bilinear upsampling operations, and then merged in the channel dimension to form a feature image with 4 times the original number of channels. Finally, a 1×1 convolution layer is passed to reduce the number of channels to one-quarter, and batch normalization and PReLu activation are used to obtain a feature map of the same size as the original feature map.

9. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 8, characterized in that: The specific algorithm for defining the loss function in step S5 is: L total =λ1·L IW +λ2·LAl+λ3·L Tl , Among them, λ1, λ2 and λ3 are constant coefficients of each term of the loss function, L IW is the comprehensive light environment intensity loss of infrared image and visible light image, L Al is the auxiliary intensity loss of the light environment, L Tl is the image texture loss.

10. The infrared and visible light image fusion method based on illumination adaptation and attention guidance according to claim 9, characterized in that: The specific algorithms for the comprehensive light environment intensity loss, light environment auxiliary intensity loss, and image texture loss of infrared images and visible light images are defined as: L IW =w ir ·L IR +w vis ·L VIS Among them, w vis With w ir is the light environment adaptation weight of visible light image and infrared image, where M and N represent the width and height of the image, and I f ,I i with I v represent the fused image, infrared image and visible light image respectively, θ(·) represents the smooth L1 loss function, represents the gradient operator, L IR With L VIS are the intensity losses of infrared images and visible light images respectively, and the specific algorithm is defined as: In the formula, θ(·) represents the smooth L1 loss function, which is expressed as follows: Where x represents the numerical difference between the predicted box and the true box.

Citation Information

Patent Citations

  • Target detection method and system using illumination guidance and attention mechanism

    CN115131640A

  • Gastrointestinal metaplasia image classification system based on multi-feature fusion and weak supervision positioning

    CN118334456A

Cited By

  • Self-adaptive infrared and visible light image fusion method based on environmental perception

    CN119762360A

  • Infrared and visible light image fusion network based on illumination intensity perception

    CN119942296A

  • RGB and near-infrared image fusion method of intelligent reconnaissance equipment under weak light condition

    CN120070204A

  • Lightweight target detection method and system based on multi-modal image

    CN120107565A

  • Target detection method based on adaptive fusion of visible light and infrared features

    CN121214139A