Fusion method and device of infrared image and visible light image
The infrared and visible images are processed through high-pass filtering and non-downsampled shear wave transformation, and the weighted feature processing is performed in the fusion network using the attention mechanism module, which solves the insufficient generalization of the fusion network and the poor fusion quality of the fusion object in the prior art. The best problem achieves high-quality image fusion.
Patent Information
- Application Number
- CN202410961644.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-17
- Publication Date
- 2024-11-08
AI Technical Summary
The existing infrared and visible image fusion methods are insufficiently generalized, and it is difficult to extract the feature information of the image that enriches the image. It is impossible to balance the fusion quality of large-size objects and extremely small-sized objects in the image, and it is impossible to highlight the importance of foreground information.
High-frequency images of infrared and visible light images are obtained through high-pass filtering, and non-downsampled shear wave transformation is performed to obtain the stacking of infrared high-frequency subband images and visible light high-frequency images. It is inputted into the fusion network and uses the attention mechanism module to perform high-frequency high-frequency results. Frequency-weighted feature processing and short-connection with visible light images through long jump connection.
The generalization of the fusion network is improved, the prospect information is highlighted, the fusion quality of objects of different sizes is enhanced, and the purpose of improving the fusion image quality is achieved.
Smart Images

Figure CN118918018A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for fusing an infrared image and a visible light image. Background Art
[0002] Infrared and visible light image fusion technology is to fuse infrared images with visible light images to improve image quality and target recognition capabilities.
[0003] The existing fusion method mainly utilizes the infrared and visible light fusion algorithm of deep learning, and fuses the images by learning the nonlinear mapping relationship between the input and output images. This method directly inputs the visible light image and the infrared image as the input image into the fusion network, resulting in insufficient generalization of the fusion network. It also makes it difficult to completely extract the feature information of images with rich texture information, and cannot balance the fusion quality of large objects and very small objects in the image. In addition, the existing fusion network does not focus on different image areas. During the fusion process, the foreground information and background information are fused in the same way, which cannot highlight the importance of foreground information.
[0004] In summary, the fusion results of existing fusion methods are not good. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides a method and device for fusing an infrared image and a visible light image, so as to achieve the purpose of improving the quality of the fused image.
[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0007] A first aspect of an embodiment of the present invention discloses a method for fusing an infrared image and a visible light image, the method comprising:
[0008] Acquire the infrared image and visible light image to be fused;
[0009] Performing high-pass filtering on the infrared image and the visible light image respectively to obtain an infrared high-frequency image and a visible light high-frequency image;
[0010] Performing a non-subsampled shearlet transform on the infrared high-frequency image to obtain an infrared high-frequency directional sub-band image;
[0011] Stacking the infrared high-frequency directional sub-band image and the visible light high-frequency image to obtain a target image;
[0012] The target image is input into a pre-built fusion network to obtain a high-frequency weighted feature map output by the fusion network; the fusion network includes an attention mechanism module; the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, and each channel of the high-frequency weighted feature map has a corresponding weight;
[0013] The high-frequency weighted feature map and the visible light image are short-circuited by using a long skip connection to obtain a fused image.
[0014] Preferably, performing a non-subsampled shearlet transform on the infrared high-frequency image to obtain an infrared high-frequency directional sub-band image includes:
[0015] Based on the non-subsampled pyramid, the infrared high-frequency image is decomposed into K levels, and the number of directions of each level is set to M, so as to obtain K high-frequency sub-band coefficients, wherein each of the high-frequency sub-band coefficients includes target high-frequency sub-band coefficients in M directions; K and M are positive integers;
[0016] Setting the preset target high frequency subband coefficients that do not need to be reconstructed to 0;
[0017] Each of the target high-frequency sub-band coefficients is transformed into the image domain to obtain an infrared high-frequency directional sub-band image.
[0018] Preferably, the fusion network includes: a feature extraction network and a feature injection network, the feature extraction network and the feature injection network respectively include multiple residual modules, and each of the residual modules includes an attention mechanism module.
[0019] Preferably, the step of inputting the target image into a pre-built fusion network to obtain a high-frequency weighted feature map output by the fusion network comprises:
[0020] Utilizing the plurality of residual modules in the feature extraction network, extracting low-frequency features in the target image to obtain a low-frequency feature map, and utilizing the attention mechanism module to perform weighted processing on the low-frequency feature map to obtain and output a low-frequency weighted feature map;
[0021] Eliminating the low-frequency features contained in the low-frequency weighted feature map from the target image to obtain a high-frequency target image;
[0022] The high-frequency features in the high-frequency target image are nonlinearly adjusted by using the multiple residual modules in the feature injection network to obtain a high-frequency feature map, and the high-frequency feature map is weighted by using the attention mechanism module to obtain and output a high-frequency weighted feature map.
[0023] Preferably, the attention mechanism module includes: a global average pooling layer, a normal convolution layer with a dilation rate of 1, a dilated convolution layer with a dilation rate greater than 1, a fully connected layer and a Sigmoid function layer.
[0024] Preferably, the step of performing weighted processing on the low-frequency feature map by using the attention mechanism module to obtain and output a low-frequency weighted feature map comprises:
[0025] Processing the low-frequency feature map sequentially through the global average pooling layer, the common convolution layer, the dilated convolution layer, the fully connected layer and the Sigmoid function layer to obtain a first weight coefficient output by the Sigmoid function layer;
[0026] Performing weighted processing on the low-frequency feature map based on the first weight coefficient to obtain and output a low-frequency weighted feature map;
[0027] Correspondingly, the use of the attention mechanism module to perform weighted processing on the high-frequency feature map to obtain and output a high-frequency weighted feature map includes:
[0028] Processing the high-frequency feature map sequentially through the global average pooling layer, the ordinary convolution layer, the dilated convolution layer, the fully connected layer and the Sigmoid function layer to obtain a second weight coefficient output by the Sigmoid function layer;
[0029] The high-frequency feature map is weighted based on the second weight coefficient to obtain and output a high-frequency weighted feature map.
[0030] A second aspect of an embodiment of the present invention discloses a device for fusing infrared images and visible light images, the device comprising:
[0031] An acquisition unit, used for acquiring the infrared image and the visible light image to be fused;
[0032] A filtering unit, used to perform high-pass filtering on the infrared image and the visible light image respectively to obtain an infrared high-frequency image and a visible light high-frequency image;
[0033] A transform unit, configured to perform a non-subsampled shearlet transform on the infrared high-frequency image to obtain an infrared high-frequency directional sub-band image;
[0034] A stacking unit, used for stacking the infrared high-frequency directional sub-band image and the visible light high-frequency image to obtain a target image;
[0035] An extraction unit is used to input the target image into a pre-built fusion network to obtain a high-frequency weighted feature map output by the fusion network; the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, and each channel of the high-frequency weighted feature map has a corresponding weight;
[0036] The fusion unit is used to short-circuit the high-frequency weighted feature map and the visible light image by using a long jump connection method to obtain a fused image.
[0037] Preferably, the transformation unit is specifically used for:
[0038] Based on the non-subsampled pyramid, the infrared high-frequency image is decomposed into K levels, and the number of directions of each level is set to M, so as to obtain K high-frequency sub-band coefficients, wherein each of the high-frequency sub-band coefficients includes target high-frequency sub-band coefficients in M directions; K and M are positive integers;
[0039] Setting the preset target high frequency subband coefficients that do not need to be reconstructed to 0;
[0040] Each of the target high-frequency sub-band coefficients is transformed into the image domain to obtain an infrared high-frequency directional sub-band image.
[0041] Preferably, the fusion network includes: a feature extraction network and a feature injection network, the feature extraction network and the feature injection network respectively include multiple residual modules, and each of the residual modules includes an attention mechanism module.
[0042] Preferably, the extraction unit comprises:
[0043] An extraction subunit, configured to extract low-frequency features in the target image using the plurality of residual modules in the feature extraction network to obtain a low-frequency feature map, and to perform weighted processing on the low-frequency feature map using the attention mechanism module to obtain and output a low-frequency weighted feature map;
[0044] A removal subunit, used for removing the low-frequency features contained in the low-frequency weighted feature map from the target image to obtain a high-frequency target image;
[0045] The adjustment subunit is used to use the multiple residual modules in the feature injection network to perform nonlinear adjustment on the high-frequency features in the high-frequency target image to obtain a high-frequency feature map, and use the attention mechanism module to perform weighted processing on the high-frequency feature map to obtain and output a high-frequency weighted feature map.
[0046] Based on the above-mentioned method and device for fusing infrared images and visible light images provided by the embodiment of the present invention, an infrared image and a visible light image to be fused are obtained; the infrared image and the visible light image are respectively subjected to high-pass filtering to obtain an infrared high-frequency image and a visible light high-frequency image; the infrared high-frequency image is subjected to non-subsampled shearlet transform to obtain an infrared high-frequency directional sub-band image; the infrared high-frequency directional sub-band image is stacked with the visible light high-frequency image to obtain a target image; the target image is input into a pre-constructed fusion network to obtain a high-frequency weighted feature map output by the fusion network; the fusion network includes an attention mechanism module; the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, and each channel of the high-frequency weighted feature map has a corresponding weight; the high-frequency weighted feature map is short-circuited with the visible light image by using a long jump connection method to obtain a fused image. In this scheme, high-pass filtering improves the generalization of the fusion network, the attention mechanism module can highlight the foreground information in the fusion process, and the non-subsampled shearlet transform has the characteristics of multi-scale and translation invariance, which can improve the fusion quality of objects of different sizes, and finally achieve the purpose of improving the quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0048] Figure 1 A flowchart of a method for fusing an infrared image and a visible light image disclosed in an embodiment of the present invention;
[0049] Figure 2 A structural diagram of a fusion network disclosed in an embodiment of the present invention;
[0050] Figure 3 A structural diagram of a residual module disclosed in an embodiment of the present invention;
[0051] Figure 4 A structural diagram of an attention mechanism module disclosed in an embodiment of the present invention;
[0052] Figure 5 The present invention is a structural diagram of a device for fusing infrared images and visible light images disclosed in an embodiment of the present invention. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.
[0055] As can be seen from the background technology, the existing fusion method mainly utilizes the infrared and visible light fusion algorithm of deep learning, and fuses by learning the nonlinear mapping relationship between the input and output images. This method directly inputs the visible light image and the infrared image as the input image into the fusion network, resulting in insufficient generalization of the fusion network. It also makes it difficult to completely extract the feature information of images with rich texture information, and cannot balance the fusion quality of large-sized objects and extremely small-sized objects in the image. In addition, the existing fusion network does not focus on different image areas, and in the fusion process, the foreground information and the background information are fused in the same way, which cannot highlight the importance of the foreground information.
[0056] Therefore, an embodiment of the present invention discloses a method and device for fusing infrared images and visible light images. In this solution, high-pass filtering improves the generalization of the fusion network, the attention mechanism module can highlight the foreground information during the fusion process, and the non-subsampled shearlet transform has the characteristics of multi-scale and translation invariance, which can improve the fusion quality of objects of different sizes, and ultimately achieve the purpose of improving the quality of the fused image.
[0057] like Figure 1 FIG. 1 is a flow chart of a method for fusing an infrared image and a visible light image disclosed in an embodiment of the present invention, comprising the following steps:
[0058] Step S101: Acquire the infrared image and visible light image to be fused.
[0059] Step S102: performing high-pass filtering on the infrared image and the visible light image respectively to obtain an infrared high-frequency image and a visible light high-frequency image.
[0060] It should be noted that deep learning methods are essentially about learning the nonlinear mapping relationship between input and output data distributions. Although the image distributions obtained by different sensors vary greatly, the high-frequency features of each image have the statistical characteristics of zero mean.
[0061] In step S102, high-pass filtering is performed on the infrared image and the visible light image respectively to obtain an infrared high-frequency image containing high-frequency features of the infrared image, and to obtain a visible light high-frequency image containing high-frequency features of the visible light image.
[0062] Compared with the prior art method of directly inputting infrared images and visible light images into the fusion network, the present application performs high-pass filtering on the infrared image and the visible light image respectively to obtain the infrared high-frequency image and the visible light high-frequency image, which has the following beneficial effects:
[0063] Since the high-frequency features of images have similar distribution, the high-frequency features of infrared images and visible light images are used as the input of the fusion network. Even if the data distribution of visible light images and infrared images in the training set and the test set is quite different, the input data distribution of the fusion network is similar. The mapping relationship learned from the training set can be effectively applied to the test set, which enhances the robustness of the fusion network structure and thus improves the generalization of the fusion network.
[0064] In addition, since most of the high-frequency eigenvalues of the infrared high-frequency image and the visible light high-frequency image are approximately equal to 0, part of the value of the fusion network input is 0, thereby reducing part of the mapping space and reducing the network calculation complexity, where the high-frequency eigenvalue is the numerical value corresponding to the data distribution of the infrared high-frequency image and the visible light high-frequency image.
[0065] Step S103: performing a non-subsampled shearlet transform on the infrared high-frequency image to obtain an infrared high-frequency directional sub-band image.
[0066] Among them, the non-subsampled shearlet transform (NSST) has the characteristics of multi-scale and multi-directional, and performs multi-scale decomposition through the non-subsampled pyramid (NSP) during the transformation process, making it translation invariant. The infrared high-frequency directional subband image obtained by the transformation has the characteristics of multi-scale, multi-directional and translation invariance through the sampling shearlet transform, which can enable the fusion network to improve the fusion quality of objects of different sizes.
[0067] The specific implementation process of step S103 includes the following steps:
[0068] Step S1031: Decompose the infrared high-frequency image into K levels based on the non-subsampled pyramid (NSP), and set the number of directions at each level to M to obtain K high-frequency sub-band coefficients, where each high-frequency sub-band coefficient contains target high-frequency sub-band coefficients in M directions, and K and M are positive integers.
[0069] Step S1032: setting the preset target high-frequency subband coefficients that do not need to be reconstructed to 0.
[0070] In step S1032, the target high-frequency sub-band coefficients to be reconstructed are transformed into the image domain to obtain an infrared high-frequency directional sub-band image to achieve reconstruction.
[0071] Step S1033: transform each target high-frequency sub-band coefficient into the image domain to obtain an infrared high-frequency directional sub-band image.
[0072] In the specific implementation process of step S1033, each target high-frequency sub-band coefficient is subjected to an inverse non-subsampled shearlet transform, thereby transforming each target high-frequency sub-band coefficient into the image domain to obtain an infrared high-frequency directional sub-band image, wherein the size of the obtained infrared high-frequency directional sub-band image is the same as the size of the original infrared image.
[0073] It should be noted that the reconstructed multi-scale and multi-directional infrared high-frequency directional sub-band image is input into the fusion network, so that the fusion process can not only take into account the directional characteristics of the image, but also be trained according to the high-frequency features of different scales, thereby taking into account the fusion quality of objects of different sizes in the image.
[0074] Step S104: stacking the infrared high-frequency directional sub-band image and the visible light high-frequency image to obtain a target image.
[0075] In step S104, stacking refers to stacking along the channel direction, that is, the visible light high-frequency image is three-channel, and the infrared high-frequency direction sub-band image is one channel. After stacking along the channel direction, a four-channel target image is obtained.
[0076] Step S104: input the target image into the pre-built fusion network to obtain a high-frequency weighted feature map output by the fusion network.
[0077] In step S104, the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, and each channel of the high-frequency weighted feature map has a corresponding weight.
[0078] like Figure 2 FIG. 1 is a structural diagram of a fusion network disclosed in an embodiment of the present invention.
[0079] Among them, the fusion network FusNet includes: feature extraction network Enet and feature injection network Inet.
[0080] The feature extraction network can fully extract the deep feature maps of high-frequency features of infrared and visible light images, that is, extract relatively low-frequency features from high-frequency features.
[0081] Then in the fusion network, the extracted low-frequency features are removed from the input feature image.
[0082] The feature injection network performs nonlinear adjustment on the remaining high-frequency features in the input feature image after removing the low-frequency features, thereby obtaining the feature information that is missing from the visible light image, that is, the feature information that differs between the ideal fusion image and the visible light image.
[0083] The expression of the fusion network is as follows:
[0084] F output =Inet(F input -Enet(F input ))
[0085] In the formula, F output represents the output feature image, F input Represents the input feature image, Enet represents feature extraction through the feature extraction network, and Inet represents feature nonlinear adjustment through the feature injection network.
[0086] Furthermore, the feature extraction network and the feature injection network respectively include multiple residual modules, and each residual module includes an attention mechanism module.
[0087] Among them, the attention mechanism module enables the fusion network training to learn adaptive weight coefficients.
[0088] like Figure 3 , which is a structural diagram of a residual module disclosed in an embodiment of the present invention.
[0089] The residual module includes: a first convolutional layer, a first ReLU layer, a second convolutional layer, an attention mechanism module and a second ReLU layer, wherein the attention mechanism module is arranged after the second convolutional layer to calibrate the residual features and highlight important information.
[0090] Based on the structure of the above fusion network, the target image is input into the pre-built fusion network, and the specific process of obtaining the high-frequency weighted feature map output by the fusion network is as follows:
[0091] Multiple residual modules in the feature extraction network are used to extract low-frequency features in the target image to obtain a low-frequency feature map, and the low-frequency feature map is weighted by the attention mechanism module to obtain and output a low-frequency weighted feature map.
[0092] The low-frequency features contained in the low-frequency weighted feature map are removed from the target image to obtain a high-frequency target image.
[0093] The high-frequency features in the high-frequency target image are nonlinearly adjusted by using multiple residual modules in the feature injection network to obtain a high-frequency feature map. The high-frequency feature map is weighted by using the attention mechanism module to obtain and output a high-frequency weighted feature map.
[0094] like Figure 4 , which is a structural diagram of an attention mechanism module disclosed in an embodiment of the present invention.
[0095] The attention mechanism module includes: global average pooling layer (GAP), ordinary convolution layer, two dilated convolution layers, fully connected layer (FC) and Sigmoid function layer (Sigmoid).
[0096] Among them, ordinary convolution and dilated convolution are combined. Ordinary convolution can capture the nonlinear relationship between adjacent channel features, and dilated convolution can capture the nonlinear relationship between non-adjacent channel features. The expansion rate of the ordinary convolution layer is smaller than that of the dilated convolution layer. The number of ordinary convolution layers and dilated convolution layers is set according to demand.
[0097] Among them, the expansion rate of ordinary convolution is 1, and the expansion rates of the two expanded convolution layers are 2 and 3 respectively. The expansion of convolution increases the size of the convolution kernel, which leads to an increase in the receptive field, and convolution layers with different convolution rates are used to obtain multi-scale receptive fields. At the same time, the expanded convolution makes the convolution kernel sparse, so it will not increase the computational burden of the network.
[0098] All output nodes in ordinary convolution and dilated convolution are connected through the fully connected layer, and the calculated features are integrated into one value, so as to comprehensively learn the relationship between the features of different channels. The fully connected layer also plays a role in dimensionality reduction. Finally, the output of the fully connected layer is input into the Sigmoid function, and the Sigmoid function is used to map the output of the fully connected layer. The output value after the Sigmoid function mapping is used as the adaptive weight coefficient, and then the channels of the feature map are weighted to distinguish the importance.
[0099] The attention mechanism module is embedded into the second convolution layer of the residual module to calibrate the residual features and highlight the foreground information, thereby solving the problem that the existing fusion network does not focus on different image regions and fuses the foreground characters and background information in the same way during the fusion process, thereby highlighting the importance of foreground information in the fused image.
[0100] Based on the specific structure of the attention mechanism module described above, the specific implementation process of using the attention mechanism module to weight the low-frequency feature map and obtain and output the low-frequency weighted feature map includes:
[0101] The low-frequency feature map is processed in sequence through the global average pooling layer, the normal convolution layer, the dilated convolution layer, the fully connected layer and the Sigmoid function layer to obtain the first weight coefficient output by the Sigmoid function layer.
[0102] The low-frequency feature map is weighted based on the first weight coefficient to obtain and output a low-frequency weighted feature map.
[0103] Correspondingly, the attention mechanism module is used to perform weighted processing on the high-frequency feature map to obtain and output the high-frequency weighted feature map. The specific implementation process includes:
[0104] The high-frequency feature map is processed in sequence through the global average pooling layer, the ordinary convolution layer, the dilated convolution layer, the fully connected layer and the Sigmoid function layer to obtain the second weight coefficient output by the Sigmoid function layer.
[0105] The high-frequency feature map is weighted based on the second weight coefficient to obtain and output a high-frequency weighted feature map.
[0106] The embodiment of the present invention also discloses a training process of a fusion network, which is as follows:
[0107] After the network is built, a public training set is used to train the network. 90% of the data in the training set is randomly selected as training samples, and 10% of the data is used as verification samples to prevent overfitting of the network.
[0108] The Adam optimizer is used during the training process to optimize the loss function to minimize it, thereby updating the network parameters.
[0109] After the network training is completed, the test visible light image and infrared image are input into the trained network to obtain the corresponding fused image.
[0110] Step S105: short-circuit the high-frequency weighted feature map and the visible light image using a long skip connection to obtain a fused image.
[0111] In step S105, since the input of the fusion network is the high-frequency features of the image, the long jump connection can utilize the low-frequency part of the visible light image to be fused, thereby realizing compensation for the spatial information of the fused image. And as the number of fusion network layers increases, the image spectral information will be lost, which can be compensated by the spectral information of the visible light image to be fused, thereby obtaining a fused image with higher fusion quality.
[0112] Among them, the high-frequency weighted feature map has the same size as the visible light image.
[0113] In the specific implementation process of step S105, for each pixel point in the visible light image, the pixel value of the pixel point is added to the pixel value of the pixel point corresponding to the position of the pixel point in the feature image to obtain a fused image.
[0114] For example, the high-frequency weighted feature map output by the network can be understood as: the part of the detail information that is missing compared to the ideal fused image. The high-frequency weighted feature map has the same size as the visible light image, for example, both are 256×256 pixels, and the pixel value represents the image information. Adding the pixel value of each pixel point of the high-frequency weighted feature map to the pixel value of the corresponding position in the visible light image is equivalent to filling in the missing part of the image information of the visible light image, thereby obtaining a fused image.
[0115] Based on the above-mentioned method for fusing infrared images and visible light images disclosed in the embodiment of the present invention, an infrared image and a visible light image to be fused are obtained, and high-pass filtering is performed on the infrared image and the visible light image respectively to obtain an infrared high-frequency image and a visible light high-frequency image, and a non-subsampled shearlet transform is performed on the infrared high-frequency image to obtain an infrared high-frequency directional subband image, and the infrared high-frequency directional subband image and the visible light high-frequency image are stacked to obtain a target image, and the target image is input into a pre-constructed fusion network to obtain a high-frequency weighted feature map output by the fusion network, wherein the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, and each channel of the high-frequency weighted feature map has a corresponding weight, and the high-frequency weighted feature map is short-circuited with the visible light image by using a long jump connection method to obtain a fused image. In this scheme, high-pass filtering improves the generalization of the fusion network, the attention mechanism module can highlight the foreground information in the fusion process, and the non-subsampled shearlet transform has the characteristics of multi-scale and translation invariance, which can improve the fusion quality of objects of different sizes, and finally achieve the purpose of improving the quality of the fused image.
[0116] Corresponding to the method for fusing an infrared image and a visible light image disclosed in the above embodiment of the present invention, as Figure 5 FIG. 1 is a structural diagram of a device for fusing infrared images and visible light images disclosed in an embodiment of the present invention. The device for fusing infrared images and visible light images includes:
[0117] The acquisition unit 501 is used to acquire the infrared image and the visible light image to be fused.
[0118] The filtering unit 502 is used to perform high-pass filtering on the infrared image and the visible light image respectively to obtain an infrared high-frequency image and a visible light high-frequency image.
[0119] The transform unit 503 is used to perform a non-subsampled shearlet transform on the infrared high-frequency image to obtain an infrared high-frequency directional sub-band image.
[0120] In one embodiment, the transform unit 503 is specifically configured to:
[0121] Based on the non-subsampled pyramid, the infrared high-frequency image is decomposed into K levels, and the number of directions at each level is set to M, to obtain K high-frequency sub-band coefficients, where each high-frequency sub-band coefficient contains the target high-frequency sub-band coefficients in M directions; K and M are positive integers;
[0122] Set the preset target high-frequency subband coefficients that do not need to be reconstructed to 0;
[0123] The high-frequency sub-band coefficients of each target are transformed into the image domain to obtain the infrared high-frequency directional sub-band image.
[0124] The stacking unit 504 is used to stack the infrared high-frequency directional sub-band image and the visible light high-frequency image to obtain a target image.
[0125] The extraction unit 505 is used to input the target image into the pre-built fusion network to obtain a high-frequency weighted feature map output by the fusion network; the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, and each channel of the high-frequency weighted feature map has a corresponding weight.
[0126] In one embodiment, the fusion network includes: a feature extraction network and a feature injection network, the feature extraction network and the feature injection network respectively include a plurality of residual modules, each residual module includes an attention mechanism module, and the extraction unit 505 includes:
[0127] The extraction subunit is used to use multiple residual modules in the feature extraction network to extract low-frequency features in the target image to obtain a low-frequency feature map, and use the attention mechanism module to weight the low-frequency feature map to obtain and output a low-frequency weighted feature map.
[0128] The elimination subunit is used to eliminate the low-frequency features contained in the low-frequency weighted feature map from the target image to obtain a high-frequency target image.
[0129] The adjustment subunit is used to use multiple residual modules in the feature injection network to nonlinearly adjust the high-frequency features in the high-frequency target image to obtain a high-frequency feature map, and use the attention mechanism module to weight the high-frequency feature map to obtain and output a high-frequency weighted feature map.
[0130] In one embodiment, the attention mechanism module includes: a global average pooling layer, a normal convolution layer, a dilated convolution layer, a fully connected layer and a Sigmoid function layer, wherein the dilation rate of the normal convolution layer is smaller than that of the dilated convolution layer, and is used to perform weighted processing on the low-frequency feature map using the attention mechanism module to obtain and output the extraction subunit of the low-frequency weighted feature map, specifically for:
[0131] The low-frequency feature map is processed sequentially through a global average pooling layer, a normal convolution layer, a dilated convolution layer, a fully connected layer, and a Sigmoid function layer to obtain the first weight coefficient output by the Sigmoid function layer;
[0132] Performing weighted processing on the low-frequency feature map based on the first weight coefficient to obtain and output a low-frequency weighted feature map;
[0133] Correspondingly, the adjustment subunit for performing weighted processing on the high-frequency feature map by using the attention mechanism module to obtain and output the high-frequency weighted feature map is specifically used for:
[0134] The high-frequency feature map is processed in sequence through a global average pooling layer, a normal convolution layer, a dilated convolution layer, a fully connected layer, and a Sigmoid function layer to obtain a second weight coefficient output by the Sigmoid function layer;
[0135] The high-frequency feature map is weighted based on the second weight coefficient to obtain and output a high-frequency weighted feature map.
[0136] The fusion unit 506 is used to short-circuit the high-frequency weighted feature map and the visible light image by using a long skip connection to obtain a fused image.
[0137] In one embodiment, the fusion unit 506 is specifically configured to:
[0138] For each pixel in the visible light image, the pixel value of the pixel is added to the pixel value of the pixel corresponding to the pixel position in the feature image to obtain a fused image.
[0139] Based on the above-mentioned infrared image and visible light image fusion device disclosed in the embodiment of the present invention, the infrared image and visible light image to be fused are obtained, the infrared image and the visible light image are respectively subjected to high-pass filtering to obtain an infrared high-frequency image and a visible light high-frequency image, the infrared high-frequency image is subjected to non-subsampled shearlet transform to obtain an infrared high-frequency directional subband image, the infrared high-frequency directional subband image is stacked with the visible light high-frequency image to obtain a target image, the target image is input into a pre-constructed fusion network, and a high-frequency weighted feature map output by the fusion network is obtained, the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, each channel of the high-frequency weighted feature map has a corresponding weight, and the high-frequency weighted feature map is short-circuited with the visible light image by a long jump connection method to obtain a fused image. In this scheme, high-pass filtering improves the generalization of the fusion network, the attention mechanism module can highlight the foreground information in the fusion process, and the non-subsampled shearlet transform has the characteristics of multi-scale and translation invariance, which can improve the fusion quality of objects of different sizes, and finally achieve the purpose of improving the quality of the fused image.
[0140] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0141] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0142] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for fusing infrared images and visible light images, characterized in that: The method comprises: Acquire the infrared image and visible light image to be fused; Performing high-pass filtering on the infrared image and the visible light image respectively to obtain an infrared high-frequency image and a visible light high-frequency image; Performing a non-subsampled shearlet transform on the infrared high-frequency image to obtain an infrared high-frequency directional sub-band image; Stacking the infrared high-frequency directional sub-band image and the visible light high-frequency image to obtain a target image; The target image is input into a pre-built fusion network to obtain a high-frequency weighted feature map output by the fusion network; the fusion network includes an attention mechanism module; the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, and each channel of the high-frequency weighted feature map has a corresponding weight; The high-frequency weighted feature map and the visible light image are short-circuited by using a long skip connection to obtain a fused image.
2. The method according to claim 1, characterized in that The step of performing a non-subsampled shearlet transform on the infrared high-frequency image to obtain an infrared high-frequency directional sub-band image includes: Based on the non-subsampled pyramid, the infrared high-frequency image is decomposed into K levels, and the number of directions of each level is set to M, so as to obtain K high-frequency sub-band coefficients, wherein each of the high-frequency sub-band coefficients includes target high-frequency sub-band coefficients in M directions; K and M are positive integers; Setting the preset target high frequency subband coefficients that do not need to be reconstructed to 0; Each of the target high-frequency sub-band coefficients is transformed into the image domain to obtain an infrared high-frequency directional sub-band image.
3. The method according to claim 1, characterized in that: The fusion network includes: a feature extraction network and a feature injection network, the feature extraction network and the feature injection network respectively include multiple residual modules, and each of the residual modules includes an attention mechanism module.
4. The method according to claim 3, characterized in that The step of inputting the target image into a pre-built fusion network to obtain a high-frequency weighted feature map output by the fusion network comprises: Utilizing the plurality of residual modules in the feature extraction network, extracting low-frequency features in the target image to obtain a low-frequency feature map, and utilizing the attention mechanism module to perform weighted processing on the low-frequency feature map to obtain and output a low-frequency weighted feature map; Eliminating the low-frequency features contained in the low-frequency weighted feature map from the target image to obtain a high-frequency target image; The high-frequency features in the high-frequency target image are nonlinearly adjusted by using the multiple residual modules in the feature injection network to obtain a high-frequency feature map, and the high-frequency feature map is weighted by using the attention mechanism module to obtain and output a high-frequency weighted feature map.
5. The method according to claim 4, characterized in that The attention mechanism module includes: a global average pooling layer, a normal convolution layer with a dilation rate of 1, a dilated convolution layer with a dilation rate greater than 1, a fully connected layer and a Sigmoid function layer.
6. The method according to claim 5, characterized in that The step of using the attention mechanism module to perform weighted processing on the low-frequency feature map to obtain and output a low-frequency weighted feature map includes: Processing the low-frequency feature map sequentially through the global average pooling layer, the common convolution layer, the dilated convolution layer, the fully connected layer and the Sigmoid function layer to obtain a first weight coefficient output by the Sigmoid function layer; Performing weighted processing on the low-frequency feature map based on the first weight coefficient to obtain and output a low-frequency weighted feature map; Correspondingly, the use of the attention mechanism module to perform weighted processing on the high-frequency feature map to obtain and output a high-frequency weighted feature map includes: Processing the high-frequency feature map sequentially through the global average pooling layer, the ordinary convolution layer, the dilated convolution layer, the fully connected layer and the Sigmoid function layer to obtain a second weight coefficient output by the Sigmoid function layer; The high-frequency feature map is weighted based on the second weight coefficient to obtain and output a high-frequency weighted feature map.
7. A device for fusing infrared images and visible light images, characterized in that: The device comprises: An acquisition unit, used for acquiring the infrared image and the visible light image to be fused; A filtering unit, used to perform high-pass filtering on the infrared image and the visible light image respectively to obtain an infrared high-frequency image and a visible light high-frequency image; A transform unit, configured to perform a non-subsampled shearlet transform on the infrared high-frequency image to obtain an infrared high-frequency directional sub-band image; A stacking unit, used for stacking the infrared high-frequency directional sub-band image and the visible light high-frequency image to obtain a target image; An extraction unit is used to input the target image into a pre-built fusion network to obtain a high-frequency weighted feature map output by the fusion network; the high-frequency weighted feature map includes high-frequency features in the target image that have been nonlinearly adjusted, and each channel of the high-frequency weighted feature map has a corresponding weight; The fusion unit is used to short-circuit the high-frequency weighted feature map and the visible light image by using a long jump connection method to obtain a fused image.
8. The device according to claim 7, characterized in that The transformation unit is specifically used for: Based on the non-subsampled pyramid, the infrared high-frequency image is decomposed into K levels, and the number of directions of each level is set to M, so as to obtain K high-frequency sub-band coefficients, wherein each of the high-frequency sub-band coefficients includes target high-frequency sub-band coefficients in M directions; K and M are positive integers; Setting the preset target high frequency subband coefficients that do not need to be reconstructed to 0; Each of the target high-frequency sub-band coefficients is transformed into the image domain to obtain an infrared high-frequency directional sub-band image.
9. The device according to claim 7, characterized in that The fusion network includes: a feature extraction network and a feature injection network, the feature extraction network and the feature injection network respectively include multiple residual modules, and each of the residual modules includes an attention mechanism module.
10. The device according to claim 9, characterized in that The extraction unit comprises: An extraction subunit, configured to extract low-frequency features in the target image using the plurality of residual modules in the feature extraction network to obtain a low-frequency feature map, and to perform weighted processing on the low-frequency feature map using the attention mechanism module to obtain and output a low-frequency weighted feature map; A removal subunit, used to remove the low-frequency features contained in the low-frequency weighted feature map from the target image to obtain a high-frequency target image; The adjustment subunit is used to use the multiple residual modules in the feature injection network to perform nonlinear adjustment on the high-frequency features in the high-frequency target image to obtain a high-frequency feature map, and use the attention mechanism module to perform weighted processing on the high-frequency feature map to obtain and output a high-frequency weighted feature map.
Citation Information
Patent Citations
Infrared image and visible image fusion method based on guide filtering
CN104809734A
A remote sensing image fusion method based on a dual-channel neural network
CN109886870A
Infrared-visible light image target level fusion method based on full convolutional neural network
CN116246138A
