An infrared and visible light fusion method based on knowledge distillation

Abstract

The application discloses an infrared and visible light fusion method based on knowledge distillation, which comprises the following steps: using a fusion model, generating a fusion image by providing a pair of visible light and infrared images to the fusion model, creating a dataset with a visible light-infrared image pair as input and a fusion image as a label; then, using an adaptive DenseNet and a Huber loss network model for convergence and supervised learning on the fusion model; in order to select the best DenseNet network architecture for the fitting task, an adaptive DenseNet set is constructed. The application introduces an adaptive knowledge distillation network, realizes the performance equivalent to a larger model, and has a simpler structure and fewer parameters; the application can adaptively adjust hyperparameters and utilize different kinds of existing fusion models, such as a CNN-based model, a Transformer-based model and a high visual level-based model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to an infrared and visible light fusion method based on knowledge distillation. BACKGROUND

[0002] In image fusion, the combination of infrared and visible light imaging sensors can take full advantage of the strengths of each sensor. Infrared sensors can detect infrared radiation and temperature changes, while visible light sensors can capture the details of texture structures. In recent years, deep learning techniques have been widely applied to traditional methods to effectively utilize multi-modal features and promote efficient image fusion. End-to-end fusion methods based on deep learning can be roughly divided into four types: autoencoder (AE), convolutional neural network (CNN), generative adversarial network (GAN) and Transformer structure. AE-based techniques use autoencoders to extract and reconstruct features and use specific strategies for fusion. In contrast, CNN-based methods use convolutional neural networks for feature extraction, fusion and reconstruction, achieving different fusion results through carefully designed network structures and loss functions. Unlike CNN-based techniques, GAN-based methods introduce a generative adversarial mechanism in infrared and visible light image fusion, usually using a generator and a discriminator. On the other hand, Transformer-based methods use self-attention mechanisms to extract global features to solve the fusion task.

[0003] Although current image fusion methods have shown impressive results, their focus on visual quality and statistical indicators often leads to overly complex and computationally demanding models that are not suitable for real-time applications on edge devices. SUMMARY

[0004] To solve this problem, the present application proposes an infrared and visible light fusion method based on knowledge distillation, which is an adaptive knowledge distillation network suitable for infrared and visible light image fusion tasks. This method uses the knowledge of a well-trained large-scale model as a teacher model and transfers this knowledge to a smaller, lighter student model. The present application collects a dataset of 3288 pairs of infrared and visible light images covering various real-world scenarios for training and evaluating the method proposed in the present application.

[0005] The infrared and visible light fusion method based on knowledge distillation disclosed in the present application comprises the following steps: using a fusion model to generate a fusion image by providing pairs of visible light and infrared images to the fusion model, creating a dataset with visible light-infrared image pairs as input and fusion images as labels;

[0006] Subsequently, the adaptive DenseNet and Huber loss network model are used for convergence, and supervised learning is performed on the fusion model.

[0007] In order to select the best DenseNet network architecture for fitting tasks, an adaptive DenseNet set is constructed.

[0008] Further, the input visible light and infrared image pairs are connected by the adaptive DenseNet, and processed by ConvBlock in denselayers(n), wherein n represents the number of denselayers; the denselayer includes ConvBlock and concatenate module, and the concatenate module helps the integration of the input of the next layer and the ConvBlock features; the output channel number generated by the ConvBlock of each denselayer is m, and the final output is obtained through convolution and Tanh activation function, thereby generating a fusion output with three channels; the adaptive DenseNet integrates the number n of denselayers and the output channel number m generated by the ConvBlock of the denselayer, and is used to adjust the network structure and hyperparameters of the adaptive DenseNet, so as to realize knowledge distillation.

[0009] Further, the convolution step stride=1 is set in all convolution processes, so that the output fusion image and the input visible light or infrared image have the same size.

[0010] Further, the HuberLoss function is used, and its mathematical expression is as follows:

[0011]

[0012] Wherein, x i is the output fusion image sequence of the proposed adaptive model, y i is the fusion output sequence of the original model, and δ is the label; for the sample set, the average loss is used to represent the loss, which can be defined as follows:

[0013] L(x,y)=mean{l1,l2…l i}。

[0014] Further, the mathematical model of adaptive optimization is as follows:

[0015] Min:{Inference time|AdaptiveDenseNet(n,m)}

[0016] And:|f(n,m)-f(n best ,mbest )| <s,(n,m)?N

[0017] Where m is the number of output channels of the denselayer, n is the number of denselayers, and f(n,m) represents the fitting performance, f(n best ,m best ) represents the theoretical optimal fit performance, s represents the design threshold, and N is a natural number.

[0018] Furthermore, DenseNet's adaptive optimal strategy consists of three steps:

[0019] Step 1: Calculate the inference time of different DenseNet structures and remove models with a certain (n,m) value and low cost-effectiveness;

[0020] The second step introduces assumptions from machine learning, including: after removing low-cost and efficient (n,m) pairs, model complexity is positively correlated with inference time; the higher the model complexity, the better its generalization and fitting ability; if two models, A and B, converge through function fitting under appropriate training and validation data, A will perform at least as well as B if A is more complex than B.

[0021] Step 3: To obtain the optimal adaptive DenseNet architecture and dual-light image fusion effect, a binary search strategy is used to adaptively obtain the optimal hyperparameters (n, m). and For network structures corresponding to s equal to 1, 8, and 16, S low S mid S high The threshold values ​​are low, medium, and high color, respectively. The initial hyperparameters are set to (n0,m0)=(5,16) and (n1,m1)=(2,4), and the initial threshold is set to σ=0.01*[f(n0,m0)-f(n1,m1)];

[0022] Iterative execution steps: when S low Not equal to S high If |f(n,m) s_mid -f(n0,m0)| < initial threshold, then S high Assigned to S mid and S low S high Half of the sum is assigned to S. mid Otherwise, S mid +1 is assigned to S low , will S low S high Half of the sum is assigned to S. mid ; until Slow Equal to S high Exit the iteration;

[0023] When the value of s is minimized, its value is considered to be the optimal case, and the adaptive DenseNet constructed based on the (n, m) value obtained from the solution is the optimal network.

[0024] The beneficial effects of the present application are as follows:

[0025] 1) The adaptive knowledge distillation network is introduced, which realizes the performance equivalent to a larger model, while having a simpler structure and fewer parameters;

[0026] 2) The method of the present application can adaptively adjust the hyperparameters and utilize different kinds of existing fusion models, such as CNN-based, Transformer-based and high-level vision-based models;

[0027] 3) The present application creates a new dataset suitable for training and evaluating models for infrared and visible image fusion tasks, covering a wide range of practical scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The overall framework diagram of the present application;

[0029] Figure 2 The structure diagram of the adaptive DenseNet of the present application;

[0030] Figure 3 Test results of all models on "man.jpg". DETAILED DESCRIPTION

[0031] The present application will be further described below in conjunction with the drawings, but in no way limits the present application, and any transformation or replacement based on the teaching of the present application belongs to the protection scope of the present application.

[0032] The framework of the present application is as follows Figure 1The method of the present application regards the fusion of visible light and infrared images as a neural network fitting problem using knowledge distillation. Specifically, the present application utilizes existing fusion models by providing them with pairs of visible light and infrared images to generate fused images. This enables the present application to create a dataset with visible light-infrared image pairs as input and fused images as labels. Subsequently, the present application uses an adaptive DenseNet and Huber loss network model to converge and perform supervised learning on the fusion model of the present application. In order to facilitate the present application to select the optimal DenseNet network architecture for the fitting task, the present application constructs an adaptive DenseNet set. Notably, compared with existing trained models, the network model of the present application has lower complexity, smaller storage size and fewer parameters, while achieving comparable or superior image fusion performance.

[0033] The structure of the adaptive DenseNet proposed by the present application is shown in Figure 2 , which integrates two variables m and n to adjust the network structure and hyperparameters of the adaptive DenseNet to achieve knowledge distillation. The input visible light and infrared image pairs are connected and processed through ConvBlock within denselayers(n), where n represents the number of denselayers, similar to the depth of the YOLO network. The denselayer includes ConvBlock and concatenate modules, which help integrate the input of the next layer and the ConvBlock features. Here, m represents the number of output channels generated by each denselayer ConvBlock, similar to the width in the YOLO network. The final output is obtained through convolution and Tanh activation function, thereby generating a fused output with three channels. Notably, due to the setting of stride = 1 in all convolution processes, the output fused image and the input visible light or infrared image have the same size.

[0034] In addition, the present application adopts HuberLoss function, which has the advantages of small influence of outliers and fast convergence speed, and its mathematical expression is as follows:

[0035]

[0036] where x i is the output fused image sequence of the proposed adaptive model, y i is the fused output sequence of the original model, and δ is the label, usually set to a threshold of 1. For the sample set, the average loss is used to represent the loss, which can be defined as follows:

[0037] L(x,y)=mean{l1,l2…l i} (2)

[0038] Best strategy for adaptive DenseNet:

[0039] The present invention aims to achieve knowledge distillation by constructing an optimal network structure to adapt to existing fusion algorithms. This involves finding the best value of (n, m) based on simplified use of DenseNet under discrete conditions to achieve good fitting performance and minimize inference time. The mathematical model of adaptive optimization is as follows:

[0040]

[0041] Where m is the number of dense layer output channels, n is the number of dense layers, f(n, m) represents the fitting performance, f(n best ,m best ) represents the optimal theoretical fitting performance, s represents the design threshold, and N is a natural number. The adaptive best strategy of DenseNet consists of three steps.

[0042] Step 1: Calculate the inference time of different DenseNet structures using the proposed method, and select low-cost and high-performance models as shown in Table 1. Models with certain (n, m) values and low cost-effectiveness are excluded. Interestingly, when n is constant, increasing m does not necessarily increase inference time, and the power of 2 of m can lead to higher cost-effectiveness.

[0043] Table 1: Selectable adaptive DenseNet network architecture

[0044]

[0045] Step 2: The proposed method introduces various common assumptions in machine learning before performing the adaptive search process to find the optimal solution. These assumptions include recognizing that after excluding low-cost and effective (n, m) pairs, the complexity of the model is positively correlated with the inference time. In other words, as the sequence number s increases, the complexity of the model also increases, resulting in longer inference time. In addition, the higher the complexity of the model, the better its generalization fitting ability. Therefore, if two models, A and B, converge through function fitting under appropriate training and validation data, and if A is more complex than B, its performance will at least be comparable to B. However, insufficient training and validation data may result in both A and B models not converging.

[0046] Step 3: The pseudo-code for the adaptive optimal search strategy is shown in Algorithm 1. To obtain the optimal adaptive DenseNet architecture and double-light image fusion effect, a binary search strategy is used to adaptively obtain the best hyperparameters (n, m). Hyperparameters and Network structures corresponding to s equal to 1, 8 and 16. After several experiments, the initialization hyperparameters of the model architecture were set to (n0, m0) = (5, 16) and (n1, m1) = (2, 4), while the initial threshold was set to σ = 0.01 * [f(n0, m0) - f(n1, m1)]. The value of s was considered optimal when its value was minimized. The adaptive DenseNet constructed based on the (n, m) values obtained from the solution was referred to as the optimal network.

[0047]

[0048] The present application uses 3288 pairs of visible and infrared images from the MSRS, M3FD_fusion and LLVIP datasets, which are divided into 2562 training and 726 validation sets. The images are cropped to 640x480 and strictly spatially aligned. To compare the performance of the proposed adaptive optimal DenseNet and the original model network, 21 pairs of dual optical images in the Visible Infrared Image Fusion Benchmark (VIFB) are used as training and testing data. 17 popular evaluation metrics in 5 categories are adopted, including entropy (EN), mutual information (MI), pixel feature mutual information (FMI_pixel), wavelet feature mutual information (FMI_w), discrete cosine feature mutual information (FMI_dct), peak signal-to-noise ratio (PSNR), edge information-based metrics (Qabf), artifact-based metrics (Nabf), structural similarity index measurement (SSIM), multi-scale structural similarity index measurement (MS-SSIM), mean square error (MSE), spatial frequency (SF), standard deviation (SD), average gradient (AG), visual information fidelity (VIF), correlation coefficient (CC) and sum of correlation differences (SCD). As shown in Table 2.

[0049] Table 2. Adaptive DenseNet network architecture

[0050]

[0051] Experimental setup and comparative analysis

[0052] By using the method proposed in the present application, i.e. optimizing the adaptive DenseNet structure of 6 popular dual-optical fusion models through knowledge distillation, the stable loss function value is converged within 20 epochs. Through at most five searches, the optimal (n, m) combination of each model when s is 1 to 16 can be found. Using 640x480 resolution images including three-channel infrared and visible light image pairs for testing, Table 3 shows that the adaptive DenseNet generated through knowledge distillation can significantly reduce the inference time and the size of model parameters. In Table 3, T represents the inference time, in ms, and S represents the weight size of the deep learning model, in KB. Among them, SwinFusion achieves the largest reduction, compressing the inference time to 0.0006 times of the original algorithm and the model parameters to 0.002 times of the original. Although the network models in DenseFuse and U2Fusion are both based on DenseNet, their inference times can still be reduced to within 0.4 times. The inference time of U2Fusion is reduced the least, to 0.305 times, but compared with other algorithm models, it has less redundancy and a more reasonable network structure.

[0053] Table 3. Comparison of model size and inference time

[0054]

[0055] For qualitative analysis, the present application selects the "tricycle.jpg" image in VIFB to show the differences between the 6 original fusion models and their corresponding distillation models in terms of contrast and texture details of infrared and visible light fusion images. Figure 3 The test results of all models on "man.jpg" are shown, and the present application selects the typical area of the figure and the ground mark for magnified comparison. The results show that the original DenseNet and U2Fusion models do not significantly highlight the figure area, while the U2Fusion distillation model using the knowledge distillation adaptive DenseNet method proposed in the present application shows higher overall contrast and is more consistent with human visual perception. In the ground mark area, RFN-NEST and its corresponding distillation model both lack obvious texture features and cannot show clear ground marks. In general, even if the defects in the original fusion models also appear in the distillation models, the knowledge distillation adaptive DenseNet method proposed in the present application can effectively fit the performance of the original fusion models to achieve the goal of model compression with shorter inference time.

[0056] The present application evaluates the performance of the proposed distillation model and dual image fusion by using 6 popular fusion models and their corresponding distillation models on 21 pairs of VIFB images. The performance evaluation uses 17 commonly used evaluation metrics, and the distribution of the fusion results obtained using the proposed distillation model is similar to the results obtained using the original model in terms of evaluation metrics. Based on the EN index, the algorithm combined with the U2Fusion method performs best on most image pairs, which indicates that the proposed distillation method can still enrich the information in the image when adapting to the U2Fusion fusion algorithm. This is because the corresponding adaptive DenseNet not only regresses to the results of the U2Fusion algorithm, but also obtains information from it.

[0057] In terms of structural similarity (SSIM) and multi-scale structural similarity (MS_SSIM) indexes, the distillation model of the DenseFuse method shows a very high similarity to the original model and is overall superior to the original model. This high degree of similarity can be attributed to the similarity in network structure between the proposed model and DenseFuse, resulting in better fitting. In addition, the traditional fusion strategy adopted by DenseFuse is specific to a specific application scenario, while the knowledge distillation model proposed in this study has greater generality and is not limited by a specific application scenario. In contrast, the original RFN-NEST algorithm performs poorly on all three SSIM indexes. Similarly, among the other models evaluated in this study, the distillation model of RFN-NEST also does not exhibit relatively strong performance.

[0058] In terms of VIF indexes based on human visual perception, the original RFN-NEST algorithm exhibits the best performance, while the results of the distillation algorithm proposed in this application rank second, but the performance of both is significantly better than that of other algorithms. The quality of the input image pairs also has a significant impact on the fusion results. For example, for image pair 17, the RFN-NEST algorithm obtains a very low score on the MS_SSIM index, which directly leads to poor performance of the corresponding distillation adaptive DenseNet on image pair 17.

[0059] The beneficial effects of the present application are as follows:

[0060] The present application proposes a new infrared and visible light image fusion method, which compresses the existing six dual light image fusion models based on different algorithms by using adaptive DenseNet and knowledge distillation technology. The method learns from the existing fusion model by using the designed adaptive DenseNet, and adjusts the hyperparameters of the model (such as inference time and the number of dense layers), and realizes the fusion of visible light and infrared images as a solution to the neural network fitting problem. Then, the present application generates the best adaptive DenseNet network for each existing fusion model. In addition, the present application also carefully selects some dual light image pairs from the public data set and the newly generated representative multi-scene infrared and visible light data set, and evaluates the performance of the various fusion models after knowledge distillation by using 17 popular evaluation indicators of dual light fusion. The proposed method is evaluated by qualitative and quantitative experiments, which shows that it can match the original fusion model and realize model compression with shorter inference time.

[0061] The word "preferred" is used herein as a term of art to denote features that can be used in one or more embodiments. The use of the word "preferred" herein does not indicate a preference of one aspect over another aspect. Rather, the word "preferred" is used to indicate a particular feature(s) that can be used in one or more embodiments. The term "or" as used herein is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless specified otherwise, or as is clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. In addition, the articles "a" and "an" as used in this application and the appended claims should generally be construed to mean "one or more" unless specified otherwise or clear from context to be directed to a singular form.

[0062] Moreover, although the present disclosure has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based on the foregoing description and illustrations. The present disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular, with reference to the various functions performed by the elements (components, etc.) described above, the terms used to describe such elements are intended to correspond, unless otherwise indicated or as is plain from the context, to any element which performs the specified function of that element (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the exemplary implementations of the present disclosure described herein. In addition, while a particular feature of the disclosure can have been disclosed with respect to only one of several implementations, such feature can be combined with one or other features of the other implementations as can be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "including", "includes", "having", "has", "containing", or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term "comprising". Moreover, although the present disclosure has been illustrated and described with respect to one or more implementations, equivalent alterations and modifications will occur to others skilled in the art based on the foregoing description and illustrations. The present disclosure includes all such modifications and alterations and is limited only by the scope of the following claims. In particular, with reference to the various functions performed by the elements (components, etc.) described above, the terms used to describe such elements are intended to correspond, unless otherwise indicated or as is plain from the context, to any element which performs the specified function of that element (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the exemplary implementations of the present disclosure described herein. In addition, while a particular feature of the disclosure can have been disclosed with respect to only one of several implementations, such feature can be combined with one or other features of the other implementations as can be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "including", "includes", "having", "has", "containing", or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term "comprising".

[0063] The various functional units in the embodiments of the present application can be integrated in one processing module, or each unit can exist physically, or multiple or more units can be integrated in one module. The integrated module can be realized in the form of hardware, or in the form of a software functional module. When the integrated module is realized in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. The various devices or systems mentioned above can execute the storage method in the corresponding method embodiments.

[0064] In summary, the above embodiments are one embodiment of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made under the spirit and principles of the present application are equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. An infrared and visible light fusion method based on knowledge distillation, characterized in that, The method comprises the following steps: A fusion model is used to generate a fused image by providing pairs of visible light and infrared images to the fusion model, creating a dataset with pairs of visible light-infrared images as input and a fused image as a label; Then, the adaptive DenseNet and Huber loss network model are used for convergence and supervised learning on the fusion model; In order to select the best DenseNet network architecture for the fitting task, an adaptive DenseNet set is constructed; The input pairs of visible light and infrared images are connected by the adaptive DenseNet and processed by ConvBlock in denselayers(n), where n represents the number of denselayers; the denselayer includes a ConvBlock and a concatenate module, which helps the integration of the input of the next layer and the ConvBlock features, and the output channel number generated by the ConvBlock of each denselayer is m, and the final output is obtained through convolution and Tanh activation function, thereby generating a fused output with three channels; The adaptive DenseNet integrates the number n of denselayers and the output channel number m generated by the ConvBlock of the denselayer to adjust the network structure and hyperparameters of the adaptive DenseNet, so as to realize knowledge distillation.

2. The knowledge distillation based infrared and visible light fusion method of claim 1, wherein, The convolution step stride is set to 1 in all convolution processes, so that the output fused image and the input visible light or infrared image have the same size.

3. The knowledge distillation based infrared and visible light fusion method of claim 1, wherein, The HuberLoss function is used, and its mathematical expression is as follows: wherein, is the output fusion image sequence of the proposed adaptive model, is the fusion output sequence of the original model, as a label; for the sample set, the loss is expressed using the average loss, which can be defined as follows: 。 4. The knowledge distillation based infrared and visible light fusion method of claim 1, wherein, The mathematical model of adaptive optimization is as follows: where m is the number of denselayer output channels, n is the number of denselayers, denotes the fit performance, denotes the theoretical optimal fit performance, design threshold, N is a natural number.

5. The knowledge distillation based infrared and visible light fusion method of claim 4, wherein, The adaptive optimal strategy of DenseNet consists of three steps: Step 1: Calculate the inference time of different DenseNet structures, and eliminate the models with certain (n, m) values and low cost-effectiveness; Step 2: Introduce the hypotheses in machine learning, which include: after eliminating the low-cost effective (n, m) pairs, the complexity of the model is positively correlated with the inference time; the higher the complexity of the model, the better its generalization fitting ability; if two models, A and B, converge through function fitting under appropriate training and validation data, if A is more complex than B, its performance will at least be comparable to B; Third step: In order to obtain the optimal adaptive DenseNet architecture and the double image fusion effect, the dichotomy search strategy is used to adaptively obtain the best super parameters (n, m), and the super parameters , and correspond to the network structures when s is equal to 1, 8 and 16, S low , S mid , S high are respectively low design threshold, medium design threshold and high color threshold, the initial super parameter is set to and , and the initial threshold is set to ; The step is iteratively performed: while S low is not equal to S high , if |f(n,m) s_mid -f(n0,m0)|< initial threshold, then S high is assigned the value of S mid , and S low is assigned the value of S high , and S mid is assigned the value of half the sum of S mid , else S low is assigned the value of S low +1, and S high is assigned the value of half the sum of S mid ; until S low is equal to S high , exit iteration; When the value of s is minimized, it is considered to be the optimal case, and the adaptive DenseNet constructed based on the (n, m) value obtained from the solution is the optimal network.

Images