Infrared and visible light image fusion algorithm based on hybrid expert system

Abstract

The invention relates to the technical field of image fusion, and provides an infrared and visible light image fusion algorithm based on a hybrid expert system, and the infrared and visible light image fusion algorithm comprises the following steps: S1, carrying out the cross feature extraction of an input infrared image and an input visible light image; s2, inputting a cross feature extraction result into a gating module composed of a full connection layer; S3, selecting two environment categories with the maximum probability value, and executing differential fusion processing in an expert model; S31, splitting an infrared image and a visible light image into local windows, then through a wavelet adaptive transformation module, performing multi-band decomposition on each window and applying an extra weight coefficient to generate a preliminary fusion image; s32, performing edge feature extraction, spot feature extraction and feature fusion processing on the preliminary fusion image to obtain a final fusion image; and S4, performing weighted fusion on the final fusion image through the trainable weight coefficient to obtain a final output image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image fusion, and particularly relates to an infrared and visible light image fusion algorithm based on a hybrid expert system. BACKGROUND

[0002] The infrared and visible light image fusion technology generates a more comprehensive fusion image by integrating the thermal target information of the infrared image and the texture details of the visible light image. At present, the technology mainly includes the following aspects: based on multi-scale transformation (such as pyramid and wavelet decomposition), sparse representation (such as dictionary coding), and deep learning (such as GAN and autoencoder). However, these existing methods have some significant defects, for example, the inference efficiency is low, the sparse coding calculation is complex, the deep network parameter quantity is large, and it is difficult to meet the real-time processing demand. In addition, the lack of universality is also a problem, because the existing models are designed for a single scene, and in fog, rain, snow and other complex weather conditions, the fusion effect will obviously decrease due to the degradation of the visible light image or the interference of the infrared features. Moreover, the data-driven deep learning model has weak generalization ability in extreme weather. What's worse, the traditional method relies on artificial setting of fusion rules, which is easy to cause information loss or artifacts. SUMMARY

[0003] In view of the above defects, the present application aims to provide an infrared and visible light image fusion algorithm based on a hybrid expert system, which aims to solve the problems of low inference efficiency, lack of universality, poor adaptability to complex weather and information loss or artifacts caused by relying on artificial setting of fusion rules by constructing a hybrid expert model, using cross-feature extraction, gate module selection, differential fusion processing and weighted fusion operation.

[0004] To achieve this purpose, the present application adopts the following technical solutions:

[0005] An infrared and visible light image fusion algorithm based on a hybrid expert system, the infrared and visible light image fusion algorithm comprising the following steps:

[0006] S1: cross-feature extraction is performed on the input infrared image and visible light image;

[0007] S2: the cross-feature extraction result is input into a gate module composed of a full connection layer, and an environment category probability distribution is output by a SoftMax classifier, the environment categories including a strong light environment, a weak light environment, a complex weather environment, a comprehensive complex environment and a normal environment;

[0008] S3: two environment categories with the largest probability values are selected, and the corresponding two expert models are activated, and differential fusion processing is performed in the activated expert models;

[0009] S31: split the infrared image and the visible light image into local windows, and then pass through a wavelet adaptive transformation module: perform multi-band decomposition on each window and apply additional weight coefficients to generate a preliminary fusion image;

[0010] S32: perform edge feature extraction, spot feature extraction and feature fusion processing on the preliminary fusion image to obtain a final fusion image, wherein additional processing on spatial attention and channel attention is performed in the complex environment and normal environment expert models;

[0011] S4: perform weighted fusion on the final fusion image obtained by the activated two expert models through trainable weight coefficients to obtain a final output image.

[0012] Preferably, step S1 comprises:

[0013] The infrared image is processed by the CBR module twice to obtain a fifth extraction result, and the visible light image is processed by the CBR module once to obtain a sixth extraction result;

[0014] The fifth extraction result is up-sampled and then concatenated with the sixth extraction result by channel, and processed by the CBR module once to obtain a seventh extraction result;

[0015] The seventh extraction result is concatenated with the fifth extraction result by channel through dimension convolution to obtain an eighth extraction result;

[0016] The CBR module comprises a Conv structure, a BN structure and a Relu activation function connected in sequence, and BN represents batch normalization.

[0017] Preferably, the SoftMax function in step S2 satisfies the relationship:

[0018]

[0019] The training loss function of step S2 satisfies the relationship:

[0020]

[0021] wherein z i represents the i-th output of the full connection layer, represents the true value of the k-th output, and k represents a certain expert model.

[0022] Preferably, the wavelet adaptive transformation module is composed of a wavelet decomposition transformation and a 1x1 convolution, and step S31 comprises:

[0023] After split-n 2The splitting is implemented by applying additional weight coefficients to different subbands to emphasize different subbands of the restored image. The restored infrared image and the visible light image are preliminarily fused to obtain a preliminary fusion image.

[0024] The average value of the pixels of the infrared light and the visible light image at each position is set as the pixel value at each position of the preliminary fusion image.

[0025] Preferably, the step S31 further comprises setting a loss function satisfying the relationship:

[0026]

[0027] wherein N is the area of the infrared salient region mask, M ir is the infrared salient region mask, F 0 is the preliminary fusion image of the expert, represents the loss function of the infrared image, I ir represents the infrared image;

[0028]

[0029] wherein, is a Scharr gradient operator, M is the area of the gradient region, represents the loss function of the visible light image;

[0030]

[0031] wherein DWT LL (·) represents the low-frequency component of the image, β is a learnable weight coefficient, I vi represents the visible light image, represents the loss function of the low-frequency component;

[0032]

[0033] wherein DWT HH (·) represents the high-frequency component of the image, represents the loss function of the high-frequency component;

[0034] The total loss function of the preliminary fusion image is:

[0035]

[0036] wherein λ1, λ2, λ3, λ4 are hyperparameters.

[0037] Preferably, the step S32 comprises:

[0038] If the strong light or weak light environment expert model is activated: the preliminary fusion image sequentially passes through the EEM module and the SEM module, and then passes through several times of CBR module for feature extraction to obtain a first final fusion image;

[0039] If the complex weather environment expert model is activated: the preliminary fusion image sequentially passes through the EEM module and the CBR module, and then passes through several times of SEM module, and then passes through the CBR module for feature extraction to obtain a second final fusion image.

[0040] Further, the step S32 comprises:

[0041] If the comprehensive complex environment expert model is activated: features are extracted through parallel processing branches, wherein a first branch sequentially passes through the CBAM, the EEM, the SEM and the CBR module for feature extraction to obtain a first extraction result;

[0042] A second branch extracts features through parallel processing branches, and a first sub-branch sequentially passes through the EEM module and the CBR module for feature extraction, and then adds the output after feature extraction of a second sub-branch through the SEM module, and then passes through the CBR module to obtain a second extraction result;

[0043] The first extraction result and the second extraction result are added, and after performing several times of feature extraction operation of the first branch and the second branch, the corresponding extraction results are comprehensively extracted through the CBR module to obtain a third final fusion image;

[0044] If the normal environment expert model is activated: features are extracted through parallel processing branches, wherein a third branch sequentially passes through the EEM, the CBAM, the SEM and the CBR module for feature extraction to obtain a third extraction result;

[0045] A fourth branch extracts features through parallel processing branches, and a third sub-branch sequentially passes through the SEM module and the CBR module for feature extraction, and then adds the output after feature extraction of a fourth sub-branch through the SEM module, and then passes through the CBR module to obtain a fourth extraction result;

[0046] The third extraction result and the fourth extraction result are added, and after performing several times of feature extraction operation of the third branch and the fourth branch, the corresponding extraction results are comprehensively extracted through the CBR module to obtain a fourth final fusion image.

[0047] Preferably, in the step S32, a loss function is set for the final fusion image, and the relationship is satisfied:

[0048]

[0049] Wherein N is the area of the infrared salient region mask, Mir F is an infrared saliency mask, 1 F is the final fused image output by the expert model, represents the intersection part of the two, I ir represents an infrared image;

[0050]

[0051] where M2 represents an infrared non-salient region mask area, M ir F is an infrared saliency mask, I represents a full white image, I vi represents a visible light image;

[0052]

[0053] where is a Sobel gradient operator, and M is a gradient region area;

[0054]

[0055] where SSIM(F 1 , I vi ) refers to a structural similarity index calculated for the final fused image F 1 output by the expert model and the visible light image I vi ;

[0056] In summary, the total loss function of the final fused image is as follows:

[0057]

[0058] where λ5, λ6, λ7 and λ8 are hyperparameters.

[0059] Preferably, step S4 satisfies the relationship:

[0060] Output = α·EXP 1 +(1-α)·EXP 2 ;

[0061] where Output is the final output image, α is a trainable learning weight parameter, EXP 1 and EXP 2 are the final fused images of the two selected expert models.

[0062] One of the technical solutions in the above technical solution has the following advantages or beneficial effects:

[0063] The application obtains shared feature information of infrared and visible light images through cross feature extraction measures, provides a basis for subsequent environment classification and image fusion, realizes effective integration of complementary information of two modal images, and helps to improve the accuracy and quality of image fusion. The environment category probability distribution is output by the gate module composed of the full connection layer and the SoftMax classifier, and the two environment categories with the maximum probability are selected to activate the corresponding expert model, so that the appropriate expert model is adaptively selected for image fusion according to different environment characteristics, the generality and robustness of the fusion algorithm are improved, and the fusion algorithm can adapt to various complex environments. By performing differential fusion processing in the expert model, the key information of the image is effectively extracted and fused, the detail and texture information of the fused image is enhanced, the clarity and quality of the fused image are improved, and the fused image is closer to the real scene effect. Through the measures of additionally performing spatial attention and channel attention processing in the comprehensive complex environment and normal environment expert models, the key information in the image is further highlighted, the quality and robustness of the fused image are improved, and the fused image can still maintain good visual effect and information integrity in a complex environment. Through the measures of weighting fusion of the final fused image obtained by the two activated expert models by using the trainable weight coefficient, the advantages of multiple expert models are integrated, the processing capability of each expert model in a specific environment is fully utilized, the best effect of the final output image in different environments is ensured, and the comprehensive quality and practicability of the image are improved. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only are the embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0065] Figure 1 is a flowchart of the infrared and visible light image fusion algorithm based on the hybrid expert system provided by the embodiment of the present application;

[0066] Figure 2 is an initial feature extraction network structure of the infrared and visible light image fusion algorithm based on the hybrid expert system provided by the embodiment of the present application;

[0067] Figure 3 is a structure diagram of the CBR module of the infrared and visible light image fusion algorithm based on the hybrid expert system provided by the embodiment of the present application;

[0068] Figure 4 is a structure diagram of the gate module of the infrared and visible light image fusion algorithm based on the hybrid expert system provided by the embodiment of the present application;

[0069] Figure 5 Figure 1 is a structural schematic diagram of a wavelet adaptive transform module of an infrared and visible light image fusion algorithm based on a hybrid expert system according to an embodiment of the present application;

[0070] Figure 6 Figure 2 is a structural schematic diagram of a strong and weak light environment expert model of an infrared and visible light image fusion algorithm based on a hybrid expert system according to an embodiment of the present application;

[0071] Figure 7 Figure 3 is a structural schematic diagram of a complex weather environment expert model of an infrared and visible light image fusion algorithm based on a hybrid expert system according to an embodiment of the present application;

[0072] Figure 8 Figure 4 is a structural schematic diagram of a comprehensive complex environment expert model of an infrared and visible light image fusion algorithm based on a hybrid expert system according to an embodiment of the present application;

[0073] Figure 9 Figure 5 is a structural schematic diagram of a normal environment expert model of an infrared and visible light image fusion algorithm based on a hybrid expert system according to an embodiment of the present application;

[0074] Figure 10 Figure 6 is a structural schematic diagram of a SEM module and an EEM module of an infrared and visible light image fusion algorithm based on a hybrid expert system according to an embodiment of the present application;

[0075] Figure 11 Figure 7 is a structural schematic diagram of a CBAM module, a SAM module and a CAM module of an infrared and visible light image fusion algorithm based on a hybrid expert system according to an embodiment of the present application. DETAILED DESCRIPTION

[0076] Embodiments of the present application are described in detail below with reference to the attached drawings, which are meant to be exemplary, and are not to be construed as limiting the present application.

[0077] In the present application, the terms "comprising", "containing" or any other variant thereof are intended to encompass non-exclusive inclusion, so that processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or device including the element.

[0078] An infrared and visible light image fusion algorithm based on a hybrid expert system, as shown in a preferred embodiment of the present application, comprises the following steps: Figure 1

[0079] S1: Cross-feature extraction is performed on the input infrared image and visible light image.

[0080] Cross-feature extraction is performed on the input infrared image and visible light image to obtain shared feature information of the two modal images. Cross-feature extraction is a method of extracting relevant features from two different types of images, which can capture complementary information between the two images and provide a basis for subsequent environment classification and image fusion.

[0081] Cross-feature extraction is a process of extracting common features from infrared images and visible light images, which can represent both thermal target information of infrared images and texture details of visible light images, providing basic features for environment classification and image fusion in subsequent steps. The fully connected layer is a neural network layer in which each neuron is connected to all neurons of the previous layer, used to convert input features into higher-level representations, which are used here to map the extracted cross-features to the probability space of environment categories. The SoftMax classifier is a commonly used classifier that can convert the input feature vector into a probability distribution of each category, used to determine the environment category of the image.

[0082] S2: The results of cross-feature extraction are input into a gating module composed of fully connected layers, and the SoftMax classifier outputs the environment category probability distribution, which includes strong light environment, weak light environment, complex weather environment, comprehensive complex environment, and normal environment.

[0083] Among them, according to the probability distribution output by the SoftMax classifier, each environment category is sorted from high to low according to the probability value, and the top two environment categories with the largest probability value are selected, and the corresponding expert models are activated. These expert models are designed for specific environment categories and can perform image fusion processing according to the characteristics of the environment.

[0084] S3: Select the two environment categories with the largest probability value and activate the corresponding two expert models to perform differential fusion processing in the activated expert models.

[0085] Among them, the two environment categories with the largest probability value are selected, and the corresponding two expert models are activated to perform differential fusion processing in the activated expert models. Differential fusion processing uses different fusion strategies according to different environment categories to adapt to the characteristics of images in various environments.

[0086] ​S31: Split the infrared image and the visible light image into local windows, and then pass through a wavelet adaptive transform module: multi-band decomposition is performed on each window and additional weight coefficients are applied to generate a preliminary fusion image;

[0087] The wavelet adaptive transform module uses wavelet decomposition to perform multi-band decomposition on the image and applies additional weight coefficients to highlight important features, and the preliminary fusion image is generated on this basis; edge feature extraction is used to obtain contour information in the image, and spot feature extraction is used to obtain detail information in the image, and feature fusion processing integrates these features to obtain a more complete fusion image. The local window is a small area into which the image is divided for local feature processing; the wavelet adaptive transform module decomposes the image into subbands of different frequency bands through wavelet decomposition, and applies additional weight coefficients to adjust the contribution of each subband to highlight important features and generate a preliminary fusion image.

[0088] S32: Perform edge feature extraction, spot feature extraction and feature fusion processing on the preliminary fusion image to obtain a final fusion image, wherein additional processing on spatial attention and channel attention is performed in the complex environment and normal environment expert models;

[0089] Edge feature extraction is used to identify contour and boundary information in the image, common methods include Sobel operator, Canny operator, etc.; spot feature extraction is used to capture local details and texture information in the image, common methods include Laplacian operator, etc.; feature fusion processing integrates edge features and spot features to form a more complete image representation; spatial attention mechanism is used to highlight important spatial regions in the image, and channel attention mechanism is used to emphasize key information in the feature channel, and the combination of the two can further improve the quality of the fusion image.

[0090] In summary, edge feature extraction and spot feature extraction can enhance the contour and detail information of the fusion image respectively, making the fusion image more rich and clear. Feature fusion processing effectively integrates these features to form a more complete image representation. The introduction of spatial attention and channel attention mechanisms can further highlight key information in the image, improve the quality and robustness of the fusion image, and make it closer to the real scene.

[0091] S4: Weighted fusion of the final fusion image obtained by the activated two expert models through trainable weight coefficients to obtain a final output image.

[0092] The trainable weight coefficient is a parameter used to adjust the contribution degree of the output of different expert models, and the optimal value is learned through the training process; weighted fusion is a linear combination of the outputs of different expert models according to the weight coefficients to generate the final fusion image.

[0093] Through the weighted fusion of trainable weight coefficients, the advantages of multiple expert models can be integrated, and the processing capabilities of each expert model in a specific environment can be fully utilized. This fusion method can preserve the thermal target information of the infrared image while retaining the rich details of the visible light image, improving the overall quality and practicality of the image, and ensuring that the final output image can achieve the best effect in different environments.

[0094] Preferably, step S1 comprises:

[0095] As shown in Figure 2 , the infrared image and the visible light image are input into the CFEM module, the infrared image is processed by two CBR modules to obtain a fifth extraction result, and the visible light image is processed by one CBR module to obtain a sixth extraction result;

[0096] The fifth extraction result is upsampled and concatenated with the sixth extraction result, and processed by one CBR module to obtain a seventh extraction result;

[0097] The seventh extraction result is concatenated with the fifth extraction result after a dimension convolution operation to obtain an eighth extraction result;

[0098] As shown in Figure 3 , the CBR module comprises a Conv structure, a BN structure and a Relu activation function connected in sequence, and BN represents a batch normalization operation.

[0099] Where U represents an upsample operation, Conv represents a two-dimensional convolution operation with a convolution kernel of 3, and C represents a channel concatenation operation.

[0100] For the fifth extraction result, the infrared image is processed by two CBR modules to obtain the fifth extraction result. At this time, the infrared image has been deeply extracted, and important information such as the contour and shape of the thermal target can be represented, providing key details of the infrared image for subsequent fusion, which helps to preserve the thermal target information of the infrared image in the fused image and improve the performance of the fused image in thermal target detection.

[0101] For the sixth extraction result, the visible light image is processed by one CBR module to obtain the sixth extraction result, which preliminarily extracts the basic features of the visible light image, including texture, edge and other information, providing a basic feature representation of the visible light image for subsequent fusion with the infrared image features. It can preserve the main texture and structure information of the visible light image, so that the fused image can inherit the rich details and texture features of the visible light image in the subsequent fusion process, enhancing the visual effect and information integrity of the fused image.

[0102] For the seventh extraction result, the fifth extraction result is up-sampled and concatenated with the sixth extraction result, and then processed by a CBR module to obtain the seventh extraction result. This process fuses the features of the infrared image and the visible light image, integrates the information of the two modal images, and preliminarily forms the cross-feature representation. The first fusion of the infrared image and the visible light image features is realized, so that the feature map contains both the thermal target information of the infrared image and the texture details of the visible light image.

[0103] For the eighth extraction result, the seventh extraction result is concatenated with the fifth extraction result after convolution operation to obtain the eighth extraction result. This step further integrates the deep features of the fused features and the infrared image, and strengthens the expression ability of the features, providing more powerful feature support for subsequent environment classification. The eighth extraction result fuses the deeper infrared image features and the previously fused feature information, so that the eighth extraction result can more comprehensively represent the shared features of the infrared and visible light images, further improving the discriminability and representativeness of the features.

[0104] After obtaining the seventh extraction result and the eighth extraction result, the seventh extraction result and the eighth extraction result are input into a gating module, and the structure diagram of the gating module is as shown in Figure 4 Therefore, the selected expert model satisfies the relationship: EXP=FC(Concat(I ir ,I vi ,CFEM(I ir ,I vi ))), where EXP represents the selected expert, FC(·) represents a fully connected layer, I ir represents an infrared image, and I vi represents a visible light image.

[0105] Preferably, the SoftMax function in step S2 satisfies the relationship:

[0106]

[0107] The SoftMax function is used to convert the output of the fully connected layer into an environment category probability distribution. The SoftMax function converts each output z i into a probability value between 0 and 1, and the sum of the probabilities of all environment categories is 1, which enables the model to represent the possibility of the input image belonging to each environment category in the form of probability.

[0108] The training loss function of step S2 satisfies the relationship:

[0109]

[0110] where z i represents the i-th output of the fully connected layer, Represents the true value of the k-th output, and k represents a certain expert model.

[0111] Preferably, the wavelet adaptive transformation module is composed of wavelet decomposition transformation and 1×1 convolution, and step S31 includes:

[0112] After split-n 2 Split, by applying additional weight coefficients to different sub-bands to achieve emphasis on different sub-bands of the restored image, and preliminarily fuse the restored infrared image and visible light image to obtain a preliminary fused image;

[0113] The average value of each pixel in the infrared light image and the visible light image is used as the pixel value in each location of the preliminary fusion image.

[0114] Specifically, the infrared image and the visible light image are split-n 2 The operation is split into multiple local windows, which helps to process local features of the image and improve processing efficiency and adaptability. Figure 5 As shown in the figure, each local window undergoes a wavelet decomposition transform, decomposing it into multiple subbands. These subbands include low-frequency subbands (such as the LL subband) and high-frequency subbands (such as the HL, LH, and HH subbands). The LL subband is the result of two low-pass filters (horizontally and vertically), the LH subband is the result of two low-pass filters (horizontally and vertically), the HL subband is the result of two high-pass filters (horizontally and vertically), the HH subband is the result of two high-pass filters (horizontally and vertically), and they represent the low-frequency and high-frequency features of the image, respectively. Additional weighting coefficients are applied to different subbands. These weighting coefficients can be adjusted based on specific needs, for example, enhancing high-frequency subbands to highlight edge information or low-frequency subbands to preserve overall structural information. The average value of the corresponding pixels in the infrared and visible light images is taken as the pixel value of the preliminary fused image. This simple average fusion method can initially integrate the information of the two modalities while maintaining the overall image structure.

[0115] Preferably, step S31 further includes setting a loss function to satisfy the relationship:

[0116]

[0117] Where N is the mask area of ​​infrared salient region, M ir is the infrared salient area mask, F 0 is the expert’s preliminary fusion image, Represents the loss function of infrared image, I ir Indicates infrared image;

[0118]

[0119] wherein, is the Scharr gradient operator, M is the gradient area of the region, denotes the loss function of the visible light image;

[0120]

[0121] wherein DWT LL denotes the low-frequency component of the image, β is a learnable weighting coefficient, I vi denotes the visible light image, denotes the loss function of the low-frequency component;

[0122]

[0123] wherein DWT HH denotes the high-frequency component of the image, denotes the loss function of the high-frequency component;

[0124] The total loss function for obtaining the preliminary fusion image is:

[0125]

[0126] wherein λ1, λ2, λ3, λ4 are hyperparameters.

[0127] The loss function set in step S31 aims to optimize the preliminary fusion image, ensuring that it is consistent with the original infrared and visible light images in multiple key aspects, thereby improving the quality and information integrity of the fusion image. Specifically:

[0128] The function is used to ensure that the preliminary fusion image is highly consistent with the original infrared image in the infrared salient region, by calculating the difference between the fusion image and the infrared image in the salient region and squaring it, The function emphasizes the importance of preserving key thermal target information in the infrared image.

[0129] The function focuses on the gradient information of the fusion image, ensuring that it matches the gradient features of the visible light image, using the Scharr gradient operator to calculate the gradients of the fusion image F 0 and the visible light image I vi , and then taking the absolute difference, which helps to preserve the details and edge information in the visible light image and enhances the visual clarity of the fusion image.

[0130] The function focuses on the low-frequency component of the image, i.e., the overall structure and large-area features. DWT LLrepresents the low-frequency component of the image after wavelet decomposition, and β is a learnable weighting coefficient to balance the low-frequency information of the infrared and visible light images. This loss function encourages the low-frequency part of the fused image to be a weighted average of the low-frequency information of the two modal images, so as to retain the overall structural information.

[0131] The function focuses on the high-frequency component, i.e. the details and edge features in the image, and DWT HH represents the high-frequency component of the image. This loss function encourages the high-frequency energy of the fused image to be at least equal to the sum of the high-frequency energies of the infrared and visible light images, so as to avoid losing important detail information.

[0132] Final total loss function is a weighted sum of the above four loss functions, where λ1, λ2, λ3, λ4 are hyperparameters for adjusting the weights of each loss in the total loss. By minimizing this total loss function, the preliminary fused image can achieve the best balance in terms of salient regions, gradient information, low-frequency and high-frequency components, etc., thereby improving the overall effect of image fusion. In general, the low-frequency loss function and the high-frequency loss function in the total loss function make the low-frequency / high-frequency information of the fused image constantly approach the low-frequency / high-frequency information of the original two images, so that the model automatically adjusts the wavelet sub-band attention parameters through backpropagation, finally obtains the optimal wavelet sub-band attention parameters, completes the preliminary fusion process, and obtains the optimal preliminary fused image.

[0133] Preferably, step S32 comprises:

[0134] As shown in Figure 6 , if the strong light or weak light environment expert model is activated: the preliminary fused image sequentially passes through the EEM module (edge extraction module) and the SEM module (spot extraction module), and then passes through several times of CBR module for feature extraction to obtain the first final fused image.

[0135] When the strong light or weak light environment expert model is activated, the preliminary fusion image needs to be first subjected to the EEM (edge feature extraction) module to highlight the contour and boundary information in the image, which is crucial for preserving the shape and structure of objects in strong light or weak light environments; then through the SEM (spot feature extraction) module, local details and texture information in the image are captured, which can make the fusion image more rich in detail performance and enhance the visual effect and information quantity of the image; then, the image is subjected to several times of CBR module (convolution, batch normalization and activation function) processing, which can further extract and integrate features, strengthen the key information of the image, and remove redundant information, and finally obtain the first final fusion image adapted to the strong light or weak light environment. The image can still maintain good clarity and detail performance in the environment with unsatisfactory lighting conditions. For the strong light or weak light environment expert model, split-n 2 The split window n is set to 4x4.

[0136] As shown in Figure 7 , if the complex weather environment expert model is activated: the preliminary fusion image is sequentially subjected to the EEM module and the CBR module, and then subjected to several times of SEM module and CBR module for feature extraction to obtain the second final fusion image.

[0137] If the complex weather environment expert model is activated, the preliminary fusion image is first subjected to the EEM module to extract edge features, which helps to identify the object contour disturbed by weather factors under complex weather conditions such as rain, snow and fog, and ensures that the basic shape information of the object in the image is preserved; then, the CBR module is used for feature integration, so that the extracted edge features are combined with other general features to form a more complete feature representation; then, several times of SEM module are used to extract spot features to strengthen the local details in the image, such as small objects in snowy weather or objects appearing and disappearing in foggy weather, thereby enhancing the information integrity of the image under complex weather conditions; finally, the CBR module is used for processing again to obtain the second final fusion image adapted to the complex weather environment, which can effectively present the key features and details of the object under complex weather conditions, and improve the usability and reliability of the image in practical applications. For the complex weather environment expert model, split-n 2 The split window n is set to 5x5, expecting to control more finely with more subdivided images to realize feature extraction and fusion under complex weather conditions.

[0138] As shown in Figure 10 , AP in SEM represents global average pooling, MP represents global maximum pooling, "+" operation represents element-wise addition, Dilated Conv in EEM represents dilated convolution, the original convolution kernel is 3, and the dilation rate is 2.

[0139] Further, the step S32 comprises:

[0140] If the integrated complex environment expert model is activated, features are extracted through parallel processing branches, wherein the first branch sequentially passes through the CBAM, EEM, SEM and CBR modules for feature extraction to obtain a first extraction result;

[0141] The second branch extracts features through parallel processing branches, and after the first sub-branch sequentially passes through the EEM module and the CBR module for feature extraction, the output thereof is added to the output of the second sub-branch after passing through the SEM module for feature extraction, and then the CBR module is passed through to obtain a second extraction result;

[0142] The first extraction result and the second extraction result are added to the output, and after performing a plurality of times of feature extraction operations of the first branch and the second branch, the corresponding extraction results are integrated through the CBR module for feature extraction to obtain a third final fusion image;

[0143] If the normal environment expert model is activated, features are extracted through parallel processing branches, wherein the third branch sequentially passes through the EEM, CBAM, SEM and CBR modules for feature extraction to obtain a third extraction result;

[0144] The fourth branch extracts features through parallel processing branches, and after the third sub-branch sequentially passes through the SEM module and the CBR module for feature extraction, the output thereof is added to the output of the fourth sub-branch after passing through the SEM module for feature extraction, and then the CBR module is passed through to obtain a fourth extraction result;

[0145] The third extraction result and the fourth extraction result are added to the output, and after performing a plurality of times of feature extraction operations of the third branch and the fourth branch, the corresponding extraction results are integrated through the CBR module for feature extraction to obtain a fourth final fusion image.

[0146] As shown in FIG. 1, Figure 8 For the integrated complex environment expert model, the processing process is divided into two main branches. In the first branch, the image sequentially passes through the CBAM, EEM, SEM and CBR modules. The CBAM is used to identify and enhance the key regions and channel information of the image, the EEM extracts the edge features of the image, the SEM captures the spot features of the image, and the CBR module further integrates these features to obtain a first extraction result.

[0147] In the second branch, the image is divided into two sub-branches for processing. The first sub-branch uses the EEM and CBR modules to extract edge features and integrate them, while the second sub-branch uses the SEM module to extract spot features. The outputs of the two sub-branches are added together and then processed by the CBR module to obtain the second extraction result. Then, the first extraction result and the second extraction result are added together, and after several times of branch feature extraction, the CBR module is used for comprehensive feature extraction to finally obtain the third final fusion image. The multi-branch parallel processing mode of the comprehensive complex environment expert model combines the advantages of different feature extraction modules, fully excavates the key information of the image in the comprehensive complex environment, and improves the robustness and adaptability of the fusion image. For the comprehensive complex environment expert model, the split-n 2 When splitting, n is also set to 5x5, and finer control is also expected to be able to meet the feature extraction requirements of various situations and perform fusion, and then a new round of feature extraction and learning is also needed to obtain the final fusion image.

[0148] As shown in Figure 9 For the normal environment expert model, the processing process is also divided into two main branches. In the third branch, the image passes through the EEM, CBAM, SEM, and CBR modules in turn. After the EEM extracts the edge features, the CBAM emphasizes the key regions and channel information in the image, the SEM captures the spot features, and finally the CBR module integrates these features to obtain the third extraction result.

[0149] In the fourth branch, the image is divided into a third sub-branch and a fourth sub-branch for processing. The third sub-branch uses the SEM and CBR modules to extract spot features and integrate them, and the fourth sub-branch further extracts spot features through the SEM module. The outputs of the two sub-branches are added together and then processed by the CBR module to obtain the fourth extraction result. Subsequently, the third extraction result and the fourth extraction result are added together, and after several times of branch feature extraction, the CBR module is used for comprehensive feature extraction to finally obtain the fourth final fusion image. This process effectively integrates edge and spot features, and the fusion image generated in the normal environment can retain rich details and clear contours.

[0150] The difference between the first branch and the third branch is the order of the EEM module and the CBAM module, because in normal environment, the data set is more complex than the feature, so the edge to be paid attention to is more and more chaotic, so the edge information needs to be extracted by EEM first (the receptive field is expanded in advance), and then the global feature is extracted by CBAM. In this way, the feature in the normal environment can be better extracted, and the optimal fusion image is obtained. In a comprehensive complex environment, although there is a lot of edge information in the weather environment, the comprehensive complex environment is also accompanied by strong light or weak light, which reduces the edge information to a certain extent, so the global feature needs to be extracted first and then the edge information is extracted, so as to obtain the optimal fusion image. For the normal environment expert model, the split-n 2 When splitting, n is 3*3, which is expected to reduce a certain amount of parameters and perform better feature extraction and fusion, and then a new round of feature extraction and learning is also needed to obtain the final fusion image.

[0151] For the CBAM module of Figures 8-9 , the structure is shown in Figure 11 , the input of the CBAM module is a feature map, and the output is a feature map enhanced by channel and spatial attention. It first calculates the channel attention through the CAM module, then inputs the result into the SAM module to calculate the spatial attention, and finally multiplies the two attention results to obtain the final output feature map.

[0152] The function of the CAM module is to calculate the importance weight of each channel. It receives the input feature map, respectively compresses the feature map through average pooling (AP) and maximum pooling (MP) operations, and obtains two different feature representations. These two feature representations are sent to a shared multi-layer perceptron (Shared MLP), which respectively outputs two channel attention branches. The results of the two branches are added and then passed through a Sigmoid activation function to obtain the channel attention weight. These weights are then multiplied with the input feature map to enhance important channel features and suppress unimportant channel features.

[0153] The function of the SAM module is to calculate the spatial attention of each position in the feature map. It receives the feature map processed by the CAM, respectively compresses each channel through average pooling and maximum pooling operations, and obtains two different feature representations. These two feature representations are fused through a 1*1 convolution layer, and then a Sigmoid activation function is used to obtain the spatial attention weight. These weights are multiplied with the input feature map to enhance important spatial region features and suppress unimportant spatial region features.

[0154] CBAM can capture important information in the feature map more comprehensively by combining channel attention and spatial attention, thereby improving the feature expression ability of the network and the performance of the model.

[0155] Preferably, in step S32, after the forward propagation of the two expert models selected adaptively, the loss function of the final fused image of each expert model is set correspondingly at the same time, including setting the loss function of the final fused image, satisfying the relationship:

[0156]

[0157] Where N is the area of the infrared salient region mask, M ir is the infrared salient region mask, F 1 is the final fused image output by the expert model, represents the intersection part, I ir represents the infrared image;

[0158]

[0159] Where M2 represents the area of the infrared non-salient region mask, M ir is the infrared salient region mask, I represents the all-white image, I vi represents the visible light image;

[0160]

[0161] Where is the Sobel gradient operator, and M is the gradient area;

[0162]

[0163] Where SSIM(F 1 ,I vi ) represents the structural similarity index of the final fused image F 1 output by the expert model and the visible light image I vi ;

[0164] Therefore, the total loss function of the final fused image is:

[0165]

[0166] Where λ5, λ6, λ7, and λ8 are hyperparameters.

[0167] The function is used to measure the consistency of the final fused image F 1 output by the expert model and the original infrared image I ir in the infrared salient region. It calculates the difference between the two, and takes the difference in the infrared salient region mask Mir The absolute value of the intersection part of the range, divided by the infrared significant region mask area N, helps to ensure that the fused image is highly consistent with the infrared image in the key hot target region.

[0168] Infrared non-significant region loss function The infrared non-significant region is concerned, that is, those parts that are not so prominent in the infrared image. It is achieved by calculating the absolute value of the intersection part of the range of the difference between the fused image F 1 and the visible light image I vi in the non-significant region mask (obtained by subtracting the infrared significant region mask M ir from the all-white image I ), and dividing it by the non-significant region mask area M2. This helps to ensure that the fused image is consistent with the visible light image in the non-significant region, thereby preserving more details and textures.

[0169] Gradient loss function To ensure that the gradient information of the fused image F 1 is consistent with that of the visible light image I vi , the gradient difference between the two is calculated using the Sobel gradient operator, and the absolute value of the difference is taken, and then divided by the gradient region area M, which helps to preserve the edge and detail information in the fused image and enhance the clarity of the image.

[0170] Structural similarity loss function By calculating the structural similarity index (SSIM) between the fused image F 1 and the visible light image I vi , and subtracting the value from 1 to represent the loss. SSIM measures the similarity in brightness, contrast and structure between two images, so this loss function helps to maintain the consistency of the fused image with the visible light image in the overall structure.

[0171] Total loss function of the final fused image is the weighted sum of the above four loss functions, where λ5, λ6, λ7 and λ8 are hyperparameters used to balance the weights of each loss function in the total loss. By minimizing this total loss function, the model can be trained to generate high-quality final fused images that not only preserve the thermal target information of the infrared image, but also preserve the details and structure of the visible light image, thereby improving the overall quality and practicality of the fused image.

[0172] Preferably, step S4 satisfies the relationship:

[0173] Output = a·EXP 1 +(1-a)·EXP 2 ;

[0174] wherein Output is the final output image, a is a trainable learning weight parameter, EXP 1 and EXP 2 are the final fused images of the two selected expert models.

[0175] In this formula, when a is close to 1, the output image is more inclined to the result of the first expert model; while when a is close to 0, the output image is closer to the result of the second expert model.

[0176] Such a weighted fusion mechanism can integrate the advantages of the two expert models. For example, in certain environments, one expert model can perform better in preserving the thermal target information of the infrared image, while the other expert model can be more excellent in preserving the details of the visible light image. By learning the optimal a value through the training process, the final output image can achieve the best fusion effect in different environments, both preserving the key thermal target information of the infrared image and preserving the rich details of the visible light image, thereby improving the overall quality and practicality of image fusion.

[0177] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0178] Although the embodiments of the present application have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, replacements and variations can be made to these embodiments without departing from the principles and spirit of the present application, and the scope of the present application is defined by the claims and their equivalents.

Claims

1. An infrared and visible light image fusion algorithm based on a hybrid expert system, characterized in that: The infrared and visible light image fusion algorithm includes the following steps: S1: Perform cross feature extraction on the input infrared image and visible light image; S2: The result of cross-feature extraction is input into a gating module composed of fully connected layers, and the SoftMax classifier outputs the probability distribution of environmental categories, including strong light environment, weak light environment, complex weather environment, comprehensive complex environment and normal environment; S3: Select the two environmental categories with the largest probability values, activate the corresponding two expert models, and perform differential fusion processing in the activated expert models: S31: Split the infrared image and visible light image into local windows, and then pass them through the wavelet adaptive transform module: perform multi-band decomposition on each window and apply additional weight coefficients to generate a preliminary fusion image; S32: performing edge feature extraction, spot feature extraction, and feature fusion processing on the preliminary fused image to obtain a final fused image, wherein the complex environment and normal environment expert models are combined to perform additional processing on spatial attention and channel attention; S4: The final fused images obtained by the activated two expert models are weightedly fused through the trainable weight coefficient to obtain the final output image.

2. The infrared and visible light image fusion algorithm based on hybrid expert system according to claim 1 is characterized in that: Step S1 includes: The infrared image is processed by the CBR module twice to obtain the fifth extraction result, and the visible light image is processed by the CBR module once to obtain the sixth extraction result; The fifth extraction result is upsampled and concatenated with the sixth extraction result by channel, and processed by the CBR module once to obtain the seventh extraction result. The seventh extraction result is subjected to a dimensional convolution operation and then concatenated with the fifth extraction result by channels to obtain an eighth extraction result. Among them, the CBR module includes a Conv structure, a BN structure and a Relu activation function connected in sequence, and BN represents a batch normalization operation.

3. The infrared and visible light image fusion algorithm based on hybrid expert system according to claim 1 is characterized in that: The SoftMax function in step S2 satisfies the relationship: The training loss function of step S2 satisfies the relationship: Among them, z i represents the i-th output of the fully connected layer, Represents the true value of the k-th output, and k represents a certain expert model.

4. The infrared and visible light image fusion algorithm based on hybrid expert system according to claim 1 is characterized in that: The wavelet adaptive transform module is composed of wavelet decomposition transform and 1×1 convolution. Step S31 includes: After split-n 2 Split, by applying additional weight coefficients to different sub-bands to achieve emphasis on different sub-bands of the restored image, and preliminarily fuse the restored infrared image and visible light image to obtain a preliminary fused image; The average value of each pixel in the infrared light image and the visible light image is used as the pixel value in each location of the preliminary fusion image.

5. The infrared and visible light image fusion algorithm based on hybrid expert system according to claim 1 is characterized in that: Step S31 also includes setting a loss function to satisfy the relationship: Where N is the mask area of ​​infrared salient region, M ir is the infrared salient area mask, F 0 is the expert’s preliminary fusion image, Represents the loss function of infrared image, I ir Indicates infrared image; in, is the Scharr gradient operator, M is the gradient area, Represents the loss function for visible light images; where DWT LL (·) represents the low-frequency component of the image, β is a learnable weighting coefficient, I vi represents a visible light image, Represents the loss function of the low-frequency component; where DWT HH (·) represents the high-frequency component of the image, Represents the loss function of high-frequency components; The total loss function of the preliminary fusion image is: Among them, λ1, λ2, λ3, and λ4 are hyperparameters.

6. The infrared and visible light image fusion algorithm based on hybrid expert system according to claim 1 is characterized in that: Step S32 includes: If the strong light or weak light environment expert model is activated: the preliminary fused image passes through the EEM module and the SEM module in sequence, and then passes through the CBR module several times for feature extraction to obtain the first final fused image; If the complex weather environment expert model is activated: the initial fused image passes through the EEM module and the CBR module in sequence, then passes through the SEM module several times, and then passes through the CBR module for feature extraction to obtain the second final fused image.

7. The infrared and visible light image fusion algorithm based on hybrid expert system according to claim 6 is characterized in that: Step S32 includes: If the integrated complex environment expert model is activated: features are extracted by parallel processing branches, where the first branch extracts features through the CBAM, EEM, SEM and CBR modules in sequence to obtain the first extraction result; The second branch extracts features through the parallel processing branch. After the features are extracted by the EEM module and the CBR module in the first sub-branch, the output of the feature extraction by the SEM module in the second sub-branch is added and passed through the CBR module to obtain the second extraction result. The first extraction result and the second extraction result are output and added together. After performing the feature extraction operations of the first branch and the second branch several times, the corresponding extraction results are subjected to comprehensive feature extraction by the CBR module to obtain a third final fused image. If the normal environment expert model is activated: features are extracted through parallel processing branches, wherein the third branch extracts features through the EEM, CBAM, SEM and CBR modules in sequence to obtain the third extraction result; The fourth branch extracts features through the parallel processing branch. After the features are extracted by the SEM module and the CBR module in the third sub-branch, the output of the feature extraction by the SEM module in the fourth sub-branch is added together and passed through the CBR module to obtain the fourth extraction result. The third extraction result and the fourth extraction result are output and added. After performing the feature extraction operations of the third branch and the fourth branch several times, the corresponding extraction results are subjected to comprehensive feature extraction by the CBR module to obtain the fourth final fused image.

8. The infrared and visible light image fusion algorithm based on hybrid expert system according to claim 1 is characterized in that: In step S32, a loss function is set for the final fused image to satisfy the relationship: Where N is the mask area of ​​infrared salient region, M ir is the infrared salient area mask, F 1 is the final fused image output by the expert model, Indicates taking the intersection of the two, I ir Indicates infrared image; Where M2 represents the mask area of ​​the infrared non-salient region, M ir is the infrared salient area mask, I represents the all-white image, I vi Represents a visible light image; in is the Sobel gradient operator, M is the area of ​​the gradient region; Among them, SSIM(F 1 ,I vi ) refers to the final fusion image F output by the expert model 1 and visible light image I vi The structural similarity index of In summary, the total loss function of the final fused image is: Among them, λ5, λ6, λ7 and λ8 are hyperparameters.

9. The infrared and visible light image fusion algorithm based on hybrid expert system according to claim 1 is characterized in that: Step S4 satisfies the relationship: Output=α·EXP 1 +(1-α)·EXP 2 ; Output is the final output image, α is the trainable weight parameter, EXP 1 and EXP 2 The final fusion image of the two selected expert models.

Images