A detection method of infrared and visible light image fusion based on interactive optimization strategy

Abstract

The present invention discloses a method for fusion detection of infrared and visible light images based on an interactive optimization strategy, comprising: inputting a visible light image to be detected and an infrared image into a neural network for fusion detection, obtaining a fused image, and obtaining the category and position of a target based on the fused image; wherein the neural network is processed by an interactive optimization strategy, and the interactive optimization strategy includes a fusion stage and a detection stage; the neural network is obtained by training a training data set, and the training data set includes processed visible light images and infrared images. The present invention adopts an interactive optimization strategy to train and optimize the fusion detection neural network, effectively solving the open-loop optimization problem of the fusion detection method and improving the detection accuracy of the detection model.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion detection, and in particular to an infrared and visible light image fusion detection method based on an interactive optimization strategy. Background Art

[0002] Image fusion is a technology that combines different spectral information. This fusion method aims to utilize the unique information content under different spectra and avoid the influence of other factors. The fused single image can express the target more comprehensively and provide a more reliable basis for subsequent image target detection. Therefore, fusion detection technology has been widely used in military, security, medicine and other fields.

[0003] Among the numerous infrared and visible light image fusion detection methods, the infrared and visible light fusion detection method based on the two-step method has achieved remarkable results and has many advantages. For example, it can well balance the infrared intensity information and visible light texture information, and improve the accuracy of the detection model. However, in the two-step fusion detection method, the fusion model and the detection model are open-loop optimization, that is, the fusion model cannot know whether the image it outputs can improve the accuracy of the detection model, so it is easy to fuse irrelevant feature information and lose feature information that is conducive to detection, thereby reducing the accuracy of the detection model. Therefore, the development of an infrared and visible light image fusion detection method with closed-loop optimization of the fusion and detection models and strong correlation between feature content is conducive to the application and development of infrared and visible light fusion detection. Summary of the invention

[0004] In view of the problem that the fused features interfere with detection during open-loop optimization in the infrared and visible light image fusion detection method, the present invention provides an infrared and visible light image fusion detection method based on an interactive optimization strategy. The method solves the open-loop optimization problem of the fusion detection model, enhances the feature information with high correlation, weakens the feature information with low correlation, and improves the detection accuracy of the detection model.

[0005] To achieve the above object, the present invention provides an infrared and visible light image fusion detection method based on an interactive optimization strategy, comprising:

[0006] Inputting the visible light image and the infrared image to be detected into the neural network for fusion detection to obtain a fused image, and obtaining the category and position of the target based on the fused image;

[0007] The neural network is processed by an interactive optimization strategy, which includes a fusion stage and a detection stage. The fusion stage is used to form a fused image, and the detection stage is used to detect the target in the fused image. The neural network is obtained by training a training data set, and the training data set includes processed visible light images and infrared images.

[0008] Preferably, the neural network includes a fusion model and a detection model, the fusion model is used to extract feature information and output the fused image, the detection model is used to detect the target object in the fused image and output a feedback factor, and the feedback factor is used to adjust the optimization step size of the fusion model.

[0009] Preferably, the fusion model includes a feature filtering and extraction module and a feature reconstruction and conversion module. The feature reconstruction and conversion module is used to reconstruct a grayscale fusion image and convert the grayscale fusion image into a color fusion image. The feature filtering and extraction module is used to enhance the feature information with strong correlation in the color fusion image, weaken the feature information with weak correlation, and output a fusion image that is conducive to detection.

[0010] Preferably, the processing process of the feature reconstruction conversion module includes:

[0011] After the infrared intensity features and visible light detail features are input into the channel for splicing operation, they are input into several layers of convolution to reconstruct the Y channel of the fused image;

[0012] The conversion factor is used to convert the Cb channel of the visible light image, the Cr channel of the visible light image and the Y channel of the fused image into the color fused image in the RGB space domain.

[0013] Preferably, the infrared intensity feature I is obtained int and visible light detail features I text The method is:

[0014] I int =FFE(I inf )I text =FFE(I Y ) where FFE(·) represents the abstract function of the feature filtering and extraction module, I inf is the infrared image, I Y is the Y channel of the visible light image, I int is the infrared intensity characteristic, I text It is the visible light detail feature.

[0015] Preferably, the processing process of the feature filtering and extraction module includes:

[0016] Extracting feature information of the infrared image or visible light image through convolution, and calculating the correlation between channels in the feature information based on covariance. The larger the absolute value of the covariance, the stronger the correlation between channels in the feature information.

[0017] The covariance value is mapped to a preset interval using a hyperbolic tangent function to obtain a feature filtering weight, and the feature filtering weight is scaled to enhance features with strong correlation and weaken features with weak correlation.

[0018] Preferably, the method for extracting the characteristic information of the infrared image or the visible light image is:

[0019] I t_a =CBR(CBR(I inf ))

[0020] In the formula, I t_a is the feature information, CBR(·) is the joint abstraction function of normalization and activation function, I inf is the input infrared image or visible light image;

[0021] The calculation method of the feature filtering weight is:

[0022]

[0023] Where Tanh(·) represents the hyperbolic tangent function, I t_f is the feature filtering weight, I t_t For feature content, For I t_t The transposed matrix of , h is the height of the channel, and w is the width of the channel.

[0024] Preferably, the loss function of the fusion model includes:

[0025] Content loss function: used to preliminarily fuse and restore the texture information of the source image by minimizing the square difference between the grayscale fused image and the source image;

[0026] Color loss function: used to keep the color fusion image similar to the color visible light image;

[0027] Feedback loss function: used to adjust the average structural similarity loss size of the fusion network optimization image.

[0028] Preferably, the content loss function is:

[0029]

[0030] In the formula, is the content loss function, ||·||2 represents the two-norm, and represents the balance weight, N is the total number of iterations, n∈[1,N], is the grayscale fusion image, For infrared images, is the Y channel of the visible light image;

[0031] The color loss function is:

[0032]

[0033] In the formula, is the color loss function, is the color fusion image, is a visible light image;

[0034] The feedback loss function is:

[0035]

[0036]

[0037] In the formula, is the feedback loss function, MSSIM(·) represents the abstract function of average structural similarity, ε n is the feedback factor, is the detection loss value of this iteration, is the detection loss value of the previous iteration.

[0038] Preferably, the training data set includes a plurality of sub-image sets, and the process of generating the sub-image sets includes:

[0039] The captured visible light image and infrared image are spatially registered and aligned, and the category and position of the target object are marked on the visible light image and infrared image to obtain the detected marked data;

[0040] Finally, after the data is expanded by cutting and randomly flipping and rotating, a number of training samples, namely the sub-image set, are formed.

[0041] Compared with the prior art, the present invention has the following advantages and technical effects:

[0042] (1) The present invention adopts an interactive optimization strategy to train and optimize the fusion detection neural network, effectively solving the open-loop optimization problem of the fusion detection method and improving the detection accuracy of the detection model;

[0043] (2) The present invention adopts a feature filtering and extraction module to enhance the feature content with strong correlation and weaken the feature content with weak correlation, so as to fuse a fused image that is conducive to detection. The conversion factor after iterative optimization is adopted to effectively output a higher quality color fused image. By feedback loss function, the quality of fused image is enhanced and the detection accuracy is improved. The present invention is conducive to the application research of infrared and visible light image fusion detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 A schematic diagram of an interactive optimization strategy architecture according to an embodiment of the present invention;

[0046] Figure 2 A schematic diagram of a fusion detection neural network architecture according to an embodiment of the present invention;

[0047] Figure 3 Schematic diagram of a feature filtering and extraction module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0049] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] The present invention proposes an infrared and visible light image fusion detection method based on an interactive optimization strategy, and the steps of implementing the method are as follows:

[0051] In order to solve the problem of open-loop optimization of the infrared and visible light image fusion detection model and improve the detection accuracy of the detection model, an interactive optimization strategy was used to train the neural network.

[0052] Interaction optimization strategies such as Figure 1 As shown in Figure 1, the interactive optimization strategy is divided into a fusion stage and a detection stage. The fusion stage mainly fuses feature information that is conducive to detection and outputs a high-quality fused image. The detection stage mainly detects the target in the fused image and outputs a feedback factor ε. The feedback factor ε is used to adjust the optimization step size of the fusion model, thereby forming a closed-loop optimization of the fusion and detection models.

[0053] Specifically, in the fusion stage, the visible light image and infrared images Input into the fusion model to obtain a fusion image that is conducive to detection Among them, C is the number of channels, G is the height, and W is the width. Therefore, the optimization formula of the fusion model is as follows:

[0054]

[0055]

[0056] In the above formula, and They represent the fusion loss abstract function, the fusion model abstract function and the fusion model parameters respectively, and ε is the feedback factor.

[0057] The output fused image is input into the detection model to detect the target object in the fused image. Therefore, the optimization formula of the detection model is as follows:

[0058]

[0059] In the above expression, and θ represent the detection loss abstract function, the detection model abstract function and the detection model parameters, respectively.

[0060] Fusion detection network model such as Figure 2 As shown in the figure, the fusion detection network model consists of a feature filtering and extraction module and a feature reconstruction and conversion module. inf and the Y channel I of the visible light image Y The infrared intensity features I are output to the feature filtering and extraction modules respectively. int and visible light detail features I text .

[0061] It can be abstractly expressed as:

[0062] I int =FFE(I inf )

[0063] I text =FFE(I Y )

[0064] In the above formula, FFE(·) represents the abstract function of the feature filtering and extraction module.

[0065] Feature reconstruction transformation module such as Figure 2 As shown, specifically, the feature reconstruction process is: the infrared intensity feature I int and visible light detail features I text After being input into the channel splicing operation Cat, it is input into the four-layer convolution to reconstruct the Y channel of the fused image Its mathematical formula is as follows:

[0066] I G =Conv(CBR(VBR(CBR(CBR(Cat(I text , I int ))

[0067] In the above formula, Conv(·), Cat(·) and CBR(·) represent the abstract function of convolution, the abstract function of channel splicing operation and the joint abstract function of convolution, normalization and activation function respectively.

[0068] In order to transform a higher quality color fusion image, a transformation process is introduced in the feature reconstruction transformation module. Specifically, the transformation factor The Cb channel I of the visible light image CB , Cr channel I of visible light image CR and the Y channel I of the fusion image G Convert to color fusion image in RGB space

[0069] Therefore, the calculation formula for space conversion is as follows:

[0070]

[0071] In the above formula, E represents an all-1 matrix, and × represents a matrix multiplication operation.

[0072] Initialize to And through continuous optimization of the loss function, the optimal conversion factor is obtained

[0073] In order to enhance the highly correlated feature information, weaken the less correlated feature information, and fuse a fused image that is conducive to detection, a feature filtering and extraction module is introduced into the fusion model. Figure 3 As shown in the figure, the calculation process of the feature filtering and extraction module is explained by taking the intensity feature extraction of infrared images as an example.

[0074] First, two layers of convolution are used to extract the infrared image I inf All feature information in The mathematical calculation formula is as follows:

[0075] I t_a =CBR(CBR(I inf ))

[0076] Secondly, a layer of convolution is used to extract the feature content of each channel The covariance is introduced into the feature filtering and extraction module to calculate the feature content I t_t The correlation between the channels in the feature content is the larger the absolute value of the covariance, the stronger the correlation between the channels in the feature content. The covariance value is mapped to [-1,1] using the hyperbolic tangent function to obtain the feature filtering weight.

[0077] Feature Filter Weight I t_f The calculation formula is as follows:

[0078] I t_t =Conv(I inf )

[0079]

[0080] In the above formula, Tanh(·) represents the hyperbolic tangent function.

[0081] Finally, the feature filtering weight I t_f Zoom I t_a Feature weights, used to enhance I t_a Strongly correlated features weaken I t_a Weakly correlated features.

[0082] Infrared intensity feature I int The calculation formula is as follows:

[0083] I int =I t_f ×I t_a

[0084] The same calculation process is used for the Y channel I of the visible light image Y , we can get the highly correlated visible light detail feature I text .

[0085] In order to fuse a high-quality fused image that is conducive to detection, three parts of loss are used to optimize the fusion network, namely content loss, color loss and feedback loss.

[0086] The content loss preliminarily fuses and restores the texture information of the source image by minimizing the square difference between the grayscale fused image and the source image.

[0087] The grayscale loss function is defined as follows:

[0088]

[0089] In the above formula, ||·||2 represents the two-norm, and Represents the balance weight.

[0090] The color loss is designed to keep the color fused image similar to the color visible light image, which avoids the fusion model from focusing too much on the grayscale and destroying the color channel.

[0091] The color loss function is defined as follows,

[0092]

[0093] In order to fuse a fused image suitable for detection and thus improve detection accuracy, feedback loss is used to construct the optimal connection between the fusion model and the detection model.

[0094] The detection loss is used as a feedback factor ε to adjust the average structural similarity loss of the fusion network to optimize the image. The average structural similarity loss function is used to further integrate more image content. Specifically, by comparing the detection loss of two adjacent iterations when Will use The average structural similarity loss value is scaled, otherwise, the original loss value will be kept to optimize the fusion network. Under the premise of keeping the loss optimization direction unchanged, the average structural similarity loss value is adjusted according to the detection loss to enhance the quality of the fused image and thus improve the detection accuracy.

[0095] Therefore, the feedback loss function is defined as follows:

[0096]

[0097]

[0098] In the above formula, MSSIM(·) represents the abstract function of average structural similarity. Therefore, the loss function of the fusion model is is defined as follows:

[0099]

[0100] In the above formula, λ1=1.25 and λ2=1.25 represent the balancing weights.

[0101] In the training data set The process of generating a sub-image set is as follows:

[0102] The captured visible light image is of size C1×H×W=3×420×512 and an infrared image of size C2×G×W=1×420×512 Perform spatial registration and alignment. Label the category and position of the target object on the visible light image and infrared image to obtain the detected labeled data. Finally, after expanding the data by cutting, random flipping and rotation, multiple training samples are formed, each training sample is recorded as S i , and finally get the complete training data set S.

[0103] After the fusion detection model is trained for M = 300 iterations as shown in Table 1, the optimized parameters of the fusion model can be obtained. And the optimization parameters of the detection model

[0104] For an infrared image X inf With visible light image X vis Forming paired sub-image sets, we can use Calculate to get the fused image Y, and pass Calculation is performed to obtain the category and position Z of the target.

[0105] Table 1

[0106]

[0107]

[0108] The present invention adopts an interactive optimization strategy to train and optimize the fusion detection neural network, effectively solves the open-loop optimization problem of the fusion detection method, and improves the detection accuracy of the detection model. The present invention adopts a feature filtering and extraction module to enhance the feature content with strong correlation, weaken the feature content with weak correlation, and fuse a fusion image that is conducive to detection. The present invention adopts the conversion factor after iterative optimization to effectively output a higher quality color fusion image. The present invention adopts a feedback loss function to effectively enhance the quality of the fusion image, thereby improving the detection accuracy. The present invention is conducive to the application research of infrared and visible light image fusion detection methods.

[0109] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for infrared and visible light image fusion detection based on interactive optimization strategy, characterized in that: include: Inputting the visible light image and the infrared image to be detected into the neural network for fusion detection to obtain a fused image, and obtaining the category and position of the target based on the fused image; The neural network is processed by an interactive optimization strategy, which includes a fusion stage and a detection stage, wherein the fusion stage is used to form a fused image, and the detection stage is used to detect the target object in the fused image; the neural network is obtained by training a training data set, and the training data set includes a processed visible light image and an infrared image; The neural network includes a fusion model and a detection model, the fusion model is used to extract feature information and output the fused image, and the detection model is used to detect the target object in the fused image and output a feedback factor; The loss function of the fusion model includes: Content loss function: used to preliminarily fuse and restore the texture information of the source image by minimizing the square difference between the grayscale fused image and the source image; Color loss function: used to keep the color fusion image similar to the color visible light image; Feedback loss function: used to adjust the average structural similarity loss of the fusion network optimized image; The content loss function is: In the formula, is the content loss function, ‖·‖2 represents the bi-norm, and represents the balance weight, N is the total number of iterations, n∈[1,N], is the grayscale fusion image, For infrared images, is the Y channel of the visible light image; The color loss function is: In the formula, is the color loss function, is the color fusion image, is a visible light image; The feedback loss function is: In the formula, is the feedback loss function, MSSIM(·) represents the abstract function of average structural similarity, ε n is the feedback factor, is the detection loss value of this iteration, is the detection loss value of the previous iteration.

2. The infrared and visible light image fusion detection method based on interactive optimization strategy according to claim 1 is characterized in that: The fusion model includes a feature filtering and extraction module and a feature reconstruction and conversion module. The feature reconstruction and conversion module is used to reconstruct a grayscale fusion image and convert the grayscale fusion image into a color fusion image. The feature filtering and extraction module is used to enhance the feature information with strong correlation in the color fusion image, weaken the feature information with weak correlation, and output a fusion image that is conducive to detection.

3. The infrared and visible light image fusion detection method based on interactive optimization strategy according to claim 2 is characterized in that: The processing process of the feature reconstruction conversion module includes: After the infrared intensity features and visible light detail features are input into the channel for splicing operation, they are input into several layers of convolution to reconstruct the Y channel of the fused image; The conversion factor is used to convert the Cb channel of the visible light image, the Cr channel of the visible light image and the Y channel of the fused image into the color fused image in the RGB space domain.

4. The infrared and visible light image fusion detection method based on interactive optimization strategy according to claim 3 is characterized in that: Obtain the infrared intensity feature I int and visible light detail features I text The method is: I int =FFE(I inf ) I text =FFE(I Y ) Where FFE(·) represents the abstract function of the feature filtering and extraction module, I inf is the infrared image, I Y is the Y channel of the visible light image, I int is the infrared intensity characteristic, I text It is the visible light detail feature.

5. The infrared and visible light image fusion detection method based on interactive optimization strategy according to claim 2 is characterized in that: The processing process of the feature filtering and extraction module includes: Extracting feature information of the infrared image or visible light image through convolution, and calculating the correlation between channels in the feature information based on covariance. The larger the absolute value of the covariance, the stronger the correlation between channels in the feature information. The covariance value is mapped to a preset interval using a hyperbolic tangent function to obtain a feature filtering weight, and the feature filtering weight is scaled to enhance features with strong correlation and weaken features with weak correlation.

6. The infrared and visible light image fusion detection method based on interactive optimization strategy according to claim 5 is characterized in that: The method for extracting the characteristic information of the infrared image or the visible light image is: I t_a =CBR(CBR(I inf )) In the formula, I t_a is the feature information, CBR(·) is the joint abstraction function of normalization and activation function, I inf is the input infrared image or visible light image; The calculation method of the feature filtering weight is: Where Tanh(·) represents the hyperbolic tangent function, I t_f is the feature filtering weight, I t_t For feature content, For I t_t The transposed matrix of , h is the height of the channel, and w is the width of the channel.

7. The infrared and visible light image fusion detection method based on interactive optimization strategy according to claim 1 is characterized in that: The training data set includes a plurality of sub-image sets, and the process of generating the sub-image sets includes: The captured visible light image and infrared image are spatially registered and aligned, and the category and position of the target object are marked on the visible light image and infrared image to obtain the detected marked data; Finally, after the data is expanded by cutting and randomly flipping and rotating, a number of training samples, namely the sub-image set, are formed.

Images