Image fusion method and device, storage medium, electronic device and computer program product

Abstract

The invention discloses an image fusion method and device, a storage medium, an electronic device and a computer program product, and relates to the field of image fusion, and the image fusion method comprises the steps: carrying out the feature extraction of an infrared image and a visible light image through a multi-scale dual-channel encoder, a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image are obtained, the first feature set and the second feature set both comprise features of N scales, and N is an integer greater than or equal to 2; performing enhancement processing on each feature in the first feature set and the second feature set through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, the group attention module having a plurality of mixed attention layers; and performing feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image.

Description

Technical Field

[0001] The present application relates to the field of image fusion, and more specifically, to an image fusion method and device, a storage medium, an electronic device, and a computer program product. Background Art

[0002] In the field of modern image processing, a single type of image sensor is limited by its hardware structure or imaging method and can usually only capture part of the information in the scene, which limits its application in complex environments. For example, infrared image sensors can effectively capture information about target objects in harsh environments such as low light and haze, but due to their imaging mechanism, they often lack rich texture details and color information. Visible light image sensors have higher resolution and richer texture information, but their performance will be significantly reduced when there is insufficient light or harsh environmental conditions. Therefore, how to take advantage of the advantages of different types of image sensors and effectively fuse multi-source images has become an important research direction in the current image processing field.

[0003] Traditional infrared and visible light image fusion methods usually rely on manually designed rules and algorithms, such as multi-scale transformation, sparse representation, and saliency detection. However, these manually designed rules and algorithms can only capture limited feature information in the source image and cannot fully express the richness of the input image. In addition, the fused images generated by traditional methods also have problems in information balance, and it is difficult to retain all the key information in the infrared and visible light images at the same time.

[0004] In recent years, with the rapid development of deep learning technology, deep learning methods have made significant progress in the field of infrared and visible light image fusion. These methods automatically learn the feature information in the image by constructing a deep neural network model, thereby achieving more accurate image fusion. However, although these deep learning-based methods are superior to traditional methods in performance, they also have some problems. On the one hand, there are differences in feature extraction between infrared and visible light images, and the use of a unified feature extraction method may not give full play to their respective advantages. On the other hand, this method can only obtain features of a limited local perception domain, and it is difficult to establish long-distance dependencies between features. These problems lead to unsatisfactory quality of the final fused image, which is difficult to meet the needs of practical applications. At the same time, the self-attention mechanism can also be used to achieve infrared and visible light image fusion. However, a single self-attention mechanism may be insufficient in modeling image features, and it is difficult to fully explore potential features. At the same time, the application of multiple self-attention mechanisms for feature modeling requires a lot of computing resources, which increases the complexity and time cost of processing.

[0005] With regard to the problem of poor fusion effect of infrared images and visible light images in related technologies, no effective solution has been proposed so far.

[0006] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0007] The embodiments of the present application provide an image fusion method and device, a storage medium, an electronic device, and a computer program product to at least solve the problem of poor fusion effect of infrared images and visible light images.

[0008] According to one aspect of an embodiment of the present application, an image fusion method is provided, comprising: performing feature extraction on an infrared image and a visible light image respectively through a multi-scale dual-channel encoder to obtain a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image, wherein the first feature set and the second feature set both include features of N scales, and N is an integer greater than or equal to 2; performing enhancement processing on each feature in the first feature set and the second feature set respectively through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, wherein the group attention module has a plurality of mixed attention layers, and each mixed attention layer has a stripe attention mechanism, a channel attention mechanism, and a window attention mechanism; performing feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image.

[0009] In an exemplary embodiment, feature extraction is performed on the infrared image and the visible light image respectively by using a multi-scale dual-channel encoder, including: extracting features from the infrared image by using the multi-scale dual-channel encoder using the following formula: in, is the i-th layer encoder corresponding to the infrared image in the multi-scale dual-channel encoder, I r is the infrared image, Φ r,i is the extracted feature of the infrared image at the i-th scale, i is 1, ..., N-1, N in sequence; and the feature of the visible light image is extracted using the following formula through the multi-scale dual-channel encoder: in, is the i-th layer encoder corresponding to the visible light image in the multi-scale dual-channel encoder, I v is the visible light image, Φ v,i is the extracted feature of the visible light image at the i-th scale.

[0010] In an exemplary embodiment, each feature in the first feature set and the second feature set is enhanced respectively by a group attention module, including: enhancing the j-th feature in the first feature set and the second feature set in the following manner, wherein j is 1, ..., 2N-1, 2N respectively: processing the j-th feature by M mixed attention layers in the group attention module to obtain a reference feature, and convolving the reference feature to obtain a target feature, wherein the input of the first mixed attention layer in the M mixed attention layers is the j-th feature, the input of the m-th mixed attention layer in the M mixed attention layers is the output of the m-1-th mixed attention layer, M is an integer greater than or equal to 2, and m is an integer greater than or equal to 1 and less than or equal to M; concatenating the target feature and the j-th feature to obtain an enhanced j-th feature, wherein the reference feature is the output of the M-th mixed attention layer in the M mixed attention layers.

[0011] In an exemplary embodiment, the j-th feature is processed by the M mixed attention layers in the group attention module to obtain a processed feature, including: looping the following operations until m is equal to M, wherein, in the first case of performing the following operations, m is equal to 1: performing the following operations by the m-th mixed attention layer among the M mixed attention layers, and after performing the following operations, updating the value of m to m+1: processing the input of the m-th mixed attention layer based on the stripe attention mechanism to obtain a first feature; processing the input of the m-th mixed attention layer based on the window attention mechanism to obtain a second feature; processing the input of the m-th mixed attention layer based on the channel attention mechanism to obtain a third feature; concatenating the first feature, the second feature and the third feature to obtain the output of the m-th mixed attention layer.

[0012] In an exemplary embodiment, processing the input of the m-th mixed attention layer based on the stripe attention mechanism to obtain a first feature includes: obtaining the first feature based on the stripe attention mechanism using the following formula: Y=M h ·(M v ·V), where Softmax is the activation function and Q = W Q Φ Z , K=W K Φ Z , V=W V Φ Z , W Q , W K , W Vare weight matrices used to calculate queries, keys, and values, respectively. A is the anchor point, Φ Z is the input of the mth mixed attention layer, and d is Φ Z The feature dimension of Y is the first feature.

[0013] In an exemplary embodiment, the enhanced first feature set and the enhanced second feature set are subjected to feature fusion to obtain a fused image, including: splicing the features of the i-th scale in the enhanced first feature set and the features of the i-th scale in the enhanced second feature set to obtain N fused features, where i is 1, ..., N-1, N in sequence; and fusing the N fused features to obtain the fused image.

[0014] In an exemplary embodiment, the N fusion features are fused to obtain the fusion image, including: when N is equal to 4, the N fusion features are processed in the following manner to obtain N fusion features after processing, and the N fusion features after processing are spliced ​​to obtain the fusion image: i,1 =CBR(M i,0 );M 4,2 =ASPP(M 4,1 ); Among them, M i,0 is the i-th fused feature among the N fused features. CBR includes a convolution layer, a batch normalization layer, and an activation function layer. ASPP is used to expand the receptive field of the feature. UP represents an upsampling operation. Represents a feature concatenation operation, and the N fused features after the processing include: M 1,2 , M 2,2 , M 3,2 , M 4,2 .

[0015] In an exemplary embodiment, after performing feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image, the method further includes: determining a loss of the fused image in the following manner, wherein the loss of the fused image is used to represent a fusion effect of the fused image: Where H is the height of the image, W is the width of the image, h is an integer greater than or equal to 0 and less than or equal to H-1, w is an integer greater than or equal to 0 and less than or equal to W-1, S represents the Sobel operator, and Δ represents the L1 loss calculation. represents the pixel value of the fused image at the (h,w) position, is the pixel value of the infrared image at the (h,w) position, is the pixel value of the visible light image at position (h, w), λ is the weight parameter, L total is the loss of the fused image.

[0016] According to another aspect of an embodiment of the present application, an image fusion device is also provided, including: an extraction module, used to perform feature extraction on an infrared image and a visible light image respectively through a multi-scale dual-channel encoder to obtain a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image, wherein the first feature set and the second feature set both include features of N scales, and N is an integer greater than or equal to 2; an enhancement module, used to perform enhancement processing on each feature in the first feature set and the second feature set respectively through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, wherein the group attention module has multiple mixed attention layers, and each mixed attention layer has a stripe attention mechanism, a channel attention mechanism and a window attention mechanism; a fusion module, used to perform feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image.

[0017] According to another aspect of the embodiments of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored program, wherein the program is configured to execute the above-mentioned image fusion method when running.

[0018] According to another aspect of the embodiments of the present application, there is further provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, wherein the processor is configured to execute the above-mentioned image fusion method through the computer program.

[0019] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and the computer program is executed by a processor to perform the above-mentioned image fusion method.

[0020] In the present application, a multi-scale dual-channel encoder is used to extract features of infrared images and visible light images at multiple scales respectively, and a group attention module is used to enhance the features of each scale in terms of stripe attention, channel attention and window attention, and finally the enhanced features are fused, thereby achieving the fusion of features of infrared images and features of visible light images, improving the fusion effect of infrared images and visible light images, and solving the problem of poor fusion effect of infrared images and visible light images. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 It is a hardware structure block diagram of a mobile terminal of an image fusion method according to an embodiment of the present application;

[0024] Figure 2 is a flow chart of an image fusion method according to an embodiment of the present application;

[0025] Figure 3 is a schematic diagram of an optional hybrid attention layer mechanism according to an embodiment of the present application;

[0026] Figure 4 is a schematic diagram of an optional group mixed attention layer module according to an embodiment of the present application;

[0027] Figure 5 It is a structural block diagram of an image fusion device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of an image fusion method according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (MP) or a programmable logic device (FPGA)) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for detecting pneumatic imbalance in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In order to solve the above problem, this embodiment provides an image fusion method, including but not limited to being applied to the above mobile terminal. Figure 2 is a flow chart of an image fusion method according to an embodiment of the present application. Figure 2 As shown, the process includes the following steps S202-S206:

[0034] Step S202: extracting features from the infrared image and the visible light image respectively by a multi-scale dual-channel encoder to obtain a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image, wherein the first feature set and the second feature set both include features of N scales, where N is an integer greater than or equal to 2;

[0035] It should be noted that the multi-scale dual-channel encoder is a neural network architecture for image processing. In the infrared image and visible light image fusion technology, it is designed to independently and effectively process two different types of images (such as infrared images and visible light images), and can extract and fuse features of different scales.

[0036] In the embodiment of the present application, the design of the multi-scale dual-channel encoder is as follows:

[0037] 1) Dual-channel design: The encoder contains two independent channels, each for processing infrared images and visible light images respectively. This design ensures that each type of image can be processed by a convolutional layer suitable for its characteristics, thereby retaining their respective advantages and details; Multi-scale feature extraction: The encoder part in each channel consists of a multi-layer convolutional neural network, and each layer can extract features of different scales, which means that everything from the global information of the image to the local details can be effectively captured and represented. Through this multi-scale processing, the network can better understand the multi-level structure of the image;

[0038] 2) Independent processing: Infrared images and visible light images are processed through their own channels, which means that their feature extraction processes are independent. This design helps avoid the interference of features of the two images and ensures that the characteristics of each image can be accurately extracted and preserved;

[0039] 3) Feature fusion: Although the feature extraction processes are independent, at the end of the encoder, the multi-scale features extracted from the two channels are fed into subsequent modules (such as the group attention module and the dense fusion block) for fusion to generate a fused image that contains the advantages of both.

[0040] The multi-scale dual-channel encoder can efficiently process and fuse features from different types of images, providing a solid foundation for high-quality image fusion. This architecture is particularly effective when processing image pairs with complementary information, as it ensures that the characteristics of each image are fully preserved and utilized.

[0041] Optionally, the first feature set and the second feature set both include N features, and each feature corresponds to a scale.

[0042] Step S204: enhancing each feature in the first feature set and the second feature set respectively through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, wherein the group attention module has a plurality of mixed attention layers, and each mixed attention layer has a stripe attention mechanism, a channel attention mechanism, and a window attention mechanism;

[0043] Step S206: performing feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image.

[0044] In the above steps, a multi-scale dual-channel encoder is used to extract features of infrared images and visible light images at multiple scales respectively, and a group attention module is used to enhance the features of each scale in terms of stripe attention, channel attention and window attention. Finally, the enhanced features are fused, thereby realizing the fusion of the features of the infrared image and the features of the visible light image, improving the fusion effect of the infrared image and the visible light image, and solving the problem of poor fusion effect of the infrared image and the visible light image.

[0045] In an exemplary embodiment, feature extraction is performed on the infrared image and the visible light image respectively by using a multi-scale dual-channel encoder, which can be achieved by the following steps S11 to S12:

[0046] Step S11: Using the multi-scale dual-channel encoder, extract features from the infrared image using the following formula: in, is the i-th layer encoder corresponding to the infrared image in the multi-scale dual-channel encoder, I r is the infrared image, Φ r,i is the extracted feature of the infrared image at the i-th scale, where i is 1, ..., N-1, N in sequence;

[0047] Optionally, the i-th layer encoder corresponding to the infrared image in the multi-scale dual-channel encoder extracts a feature of the infrared image at the i-th scale.

[0048] Step S12: Using the multi-scale dual-channel encoder, extract features from the visible light image using the following formula: in, is the i-th layer encoder corresponding to the visible light image in the multi-scale dual-channel encoder, I v is the visible light image, Φ v,i is the extracted feature of the visible light image at the i-th scale.

[0049] Optionally, the i-th layer encoder corresponding to the visible light image in the multi-scale dual-channel encoder extracts a feature of the visible light image at the i-th scale.

[0050] That is to say, the multi-scale dual-channel encoder will extract the features corresponding to the infrared image and the visible light image respectively at one scale.

[0051] It should be noted that the above-mentioned multi-scale feature extraction strategy can not only capture local details of the image, such as texture and edges, but also extract global information of the image, such as the shape and position of the object, thereby providing a rich and multi-level feature basis for subsequent image fusion.

[0052] In addition, the independent dual-channel design effectively avoids mutual interference between infrared image and visible light image features, ensuring the purity and accuracy of each image feature, which is particularly important for image fusion in complex environments. At the same time, this design can also better adapt to the differences in imaging principles and image characteristics between infrared and visible light images, further improving the visual effect and application performance of the fused image.

[0053] In an exemplary embodiment, each feature in the first feature set and the second feature set is enhanced by the group attention module, including: enhancing the j-th feature in the first feature set and the second feature set by the following steps S21-S22, wherein j is 1, ..., 2N-1, 2N in sequence:

[0054] Step S21: Processing the j-th feature through the M hybrid attention layers in the group attention module to obtain a reference feature, and convolving the reference feature to obtain a target feature, wherein the input of the first hybrid attention layer in the M hybrid attention layers is the j-th feature, the input of the m-th hybrid attention layer in the M hybrid attention layers is the output of the m-1-th hybrid attention layer, M is an integer greater than or equal to 2, and m is an integer greater than or equal to 1 and less than or equal to M;

[0055] Step S22: Concatenate the target feature and the j-th feature to obtain an enhanced j-th feature, wherein the reference feature is the output of the M-th mixed attention layer among the M mixed attention layers.

[0056] Alternatively, if Figure 4 As shown in FIG. 1 , there are multiple mixed attention layers in the group attention module. Taking four mixed attention layers as an example, first, the jth feature is input into the first mixed attention layer, and the first mixed attention layer will enhance the jth feature, thereby outputting a j* enhanced feature, and then the j* enhanced feature is input into the second mixed attention layer, and then the second mixed attention layer will enhance the j* enhanced feature, and output a feature that further enhances the enhanced feature, and input it into the third mixed attention layer, the third mixed attention layer and the fourth mixed attention layer, and so on. The output of the fourth mixed attention layer is the reference feature, and the group attention module convolves the obtained reference feature ( Figure 4 Conv in to obtain the target feature, and then concatenate the target feature with the j-th feature to obtain the enhanced j-th feature.

[0057] It should be noted that, through the above steps, in this embodiment, the multi-scale features of the infrared and visible light images are enhanced by the group attention module, which can significantly improve the quality and information richness of the fused image.

[0058] In an exemplary embodiment, processing the j-th feature through the M mixed attention layers in the group attention module to obtain the processed feature includes: looping the following step S31 until m is equal to M, wherein, in the case of the first execution of the following step S31, m is equal to 1:

[0059] Step S31: The following steps S311-S314 are performed by the mth mixed attention layer in the M mixed attention layers, and after performing the following steps S311-S314, the value of m is updated to m+1:

[0060] Step S311: Processing the input of the m-th mixed attention layer based on the stripe attention mechanism to obtain a first feature;

[0061] Step S312: Processing the input of the m-th hybrid attention layer based on the window attention mechanism to obtain a second feature;

[0062] Step S313: Processing the input of the m-th mixed attention layer based on the channel attention mechanism to obtain a third feature;

[0063] Step S314: concatenate the first feature, the second feature and the third feature to obtain the output of the mth mixed attention layer.

[0064] Alternatively, if Figure 3 As shown in the figure, each hybrid attention layer will enhance the features of the input of this layer through the stripe attention mechanism, window attention mechanism and channel attention mechanism.

[0065] Optionally, at each scale, the group attention module is composed of multiple stacked hybrid attention layers, each of which models local, regional, and global features respectively. The stripe attention mechanism achieves long-range dependency modeling of similar features across regions through anchor points with fixed directions, with low computational complexity; channel attention and window attention are processed in parallel, which improves the efficiency and accuracy of feature extraction.

[0066] Optionally, the stripe attention mechanism performs long-range dependency modeling through anchor points, the channel attention mechanism performs feature enhancement through inter-channel attention, and the window attention performs feature modeling within a local window.

[0067] It should be noted that the stripe attention mechanism is a method for long-range dependency modeling for anchor points with fixed directions, which can effectively establish connections between different regions of an image and is particularly suitable for processing directional features such as textures or lines; the channel attention mechanism can enhance important features related to the task by focusing on the correlation between different features, while suppressing irrelevant features, thereby improving the representation ability of features; the window attention mechanism performs feature modeling within a local window, which can capture local details in the image and improve efficiency through parallel processing. This is similar to self-attention in a local area and helps to understand the local structure of an image.

[0068] In this embodiment, multiple mixed attention layers are stacked in the group attention module. Each mixed attention layer can perform multi-level enhancement on image features by combining the above methods, thereby improving the quality of the fused image.

[0069] In an exemplary embodiment, processing the input of the m-th mixed attention layer based on the stripe attention mechanism to obtain a first feature includes: obtaining the first feature based on the stripe attention mechanism using the following formula:

[0070]

[0071] Y=M h ·(M v ·V)

[0072] Among them, Softmax is the activation function, Q = W Q ΦZ , K=W K Φ Z , V=W V Φ Z , W Q , W K , W V are weight matrices used to calculate queries, keys, and values, respectively. A is the anchor point, Φ Z is the input of the mth mixed attention layer, and d is Φ Z The feature dimension of Y is the first feature.

[0073] In an exemplary embodiment, processing the input of the mth mixed attention layer based on the window attention mechanism to obtain a second feature includes: based on the window attention mechanism, using the following formula to obtain the second feature: Φ WA =WindowAttention(Φ);

[0074] Among them, Φ WA is the second feature, WindowAttention is the window attention mechanism, and Φ is equal to Φ Z , which is the input of the mth mixed attention layer.

[0075] In an exemplary embodiment, processing the input of the mth mixed attention layer based on the channel attention mechanism to obtain a third feature includes: based on the channel attention mechanism, using the following formula to obtain the third feature: Φ CA = ChannelAttention(Φ);

[0076] Among them, Φ CA is the third feature, ChannelAttention is the channel attention mechanism, and Φ is equal to Φ Z , which is the input of the mth mixed attention layer.

[0077] In an exemplary embodiment, the enhanced first feature set and the enhanced second feature set are subjected to feature fusion to obtain a fused image, which can be achieved by the following steps S41-S42:

[0078] Step S41: splicing the features of the i-th scale in the enhanced first feature set and the features of the i-th scale in the enhanced second feature set to obtain N fused features, where i is 1, ..., N-1, N in sequence;

[0079] Step S42: fusing the N fusion features to obtain the fused image.

[0080] Optionally, after each feature in the first feature set and the second feature set is enhanced respectively by the group attention module, the multi-scale features are fused by the dense fusion block, and the dense fusion block adopts a dense connection strategy to ensure that the feature information of each scale is fully utilized in the fusion process: first, the feature group of each scale is input into the fusion block for further feature processing, and the features are transformed and optimized using the convolution layer, batch normalization layer and activation function layer; then, the adaptive pooling strategy is adopted to expand the receptive field of the feature and enhance the expressive ability of the feature; finally, the multi-scale features are fused layer by layer in a cascade manner to generate the final fused image.

[0081] In an exemplary embodiment, fusing the N fusion features to obtain the fusion image can be achieved by the following steps: when N is equal to 4, the N fusion features are processed in the following manner to obtain N fusion features after processing, and the N fusion features after processing are spliced ​​to obtain the fusion image:

[0082] M i,1 =CBR(M i,0 );

[0083] M 4,2 =ASPP(M 4,1 );

[0084]

[0085] Among them, M i,0 is the i-th fused feature among the N fused features. CBR includes a convolution layer, a batch normalization layer, and an activation function layer. ASPP is used to expand the receptive field of the feature. UP represents an upsampling operation. Represents a feature concatenation operation, and the N fused features after the processing include: M 1,2 , M 2,2 , M 3,2 , M 4,2 .

[0086] Optionally, when N is equal to 4, the multi-scale dual-channel encoder extracts features of the infrared image and the visible light image at four scales respectively, and obtains four features of the infrared image and four features of the visible light image respectively, then M 1,0 That is, it is the first fusion feature of the four fusion features (that is, the fusion feature of the feature of the infrared image at the first scale and the feature of the visible light image at the first scale). By analogy, M 2,0 That is, the fusion features of the infrared image at the second scale and the visible light image at the second scale, M 3,0That is, the fusion features of the infrared image at the third scale and the visible light image at the third scale, M 4,0 That is, the fusion features of the infrared image at the fourth scale and the visible light image at the fourth scale.

[0087] Optionally, M 4,0 For example, M 4,1 =CBR(M 4,0 );M 4,2 =ASPP(M 4,1 ).

[0088] Optionally, M 3,0 For example, M 3,1 =CBR(M 3,0 );but

[0089] Optionally, M 2,0 For example, M 2,1 =CBR(M 2,0 );but

[0090] Optionally, M 1,0 For example, M 1,1 =CBR(M 1,0 );but

[0091] Using M 1,0 、M 2,0 、M 3,0 、M 4,0 Through the above method, four fusion features are obtained after processing, namely M 1,2 , M 2,2 , M 3,2 , M 4,2 , M 1,2 , M 2,2 , M 3,2 , M 4,2 The fused image is obtained by stitching.

[0092] It should be noted that the Convolution-Batch Normalization-Activation (CBR) module is used to process and convert feature maps, which specifically includes the following three steps: Convolution layer: linearly transform the input feature map through the learned convolution kernel to extract local features of the image; Batch Normalization layer (BN): normalizes the feature map output by the convolution layer to make the features more refined; Activation function layer: usually uses a similar nonlinear activation function to increase the nonlinear expression ability of the features.

[0093] It should be noted that Atrous Spatial Pyramid Pooling (ASPP) can better understand objects or structures of different sizes in images and enhance the ability to capture features of different scales.

[0094] In an exemplary embodiment, after the enhanced first feature set and the enhanced second feature set are subjected to feature fusion to obtain a fused image, the method further comprises the following steps: determining a loss of the fused image in the following manner, wherein the loss of the fused image is used to represent a fusion effect of the fused image:

[0095]

[0096] Where H is the height of the image, W is the width of the image, h is an integer greater than or equal to 0 and less than or equal to H-1, w is an integer greater than or equal to 0 and less than or equal to W-1, S represents the Sobel operator, and Δ represents the L1 loss calculation. represents the pixel value of the fused image at position (h, w), is the pixel value of the infrared image at the position (h, w), is the pixel value of the visible light image at position (h, w), λ is the weight parameter, L total is the loss of the fused image.

[0097] Obviously, the above-described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. In order to better understand the above method, the above process is described below in conjunction with the embodiments, but it is not intended to limit the technical solutions of the embodiments of the present invention. Figure 3 is a schematic diagram of an optional hybrid attention layer mechanism according to an embodiment of the present application, specifically:

[0098] 1. Feature extraction of multi-scale dual-channel encoder;

[0099] First, a multi-scale dual-channel encoder is used to extract multi-scale features from infrared and visible light images respectively. The multi-scale dual-channel encoder can process infrared and visible light images independently. Specifically, the encoder extracts feature maps of different scales through a multi-layer convolutional neural network. Assume that the input infrared image is I r and the visible light image is I v , the encoder processes them separately to obtain the multi-scale feature representation Φ r,i and Φ v,i , where i represents different scales, i.e. in, and represents the i-th layer encoder.

[0100] Second, feature enhancement of group attention module;

[0101] After extracting multi-scale features, the group attention module is used to enhance the features. Specifically, at each scale, the group attention module is composed of multiple mixed attention layers stacked together, and each mixed attention layer models local, regional, and global features respectively: the stripe attention mechanism achieves long-range dependency modeling of similar features across regions through anchor points in fixed directions, with low computational complexity; channel attention and window attention are processed in parallel, which improves the efficiency and accuracy of feature extraction. The calculation process of the mixed attention layer is as follows:

[0102] 1. Stripe attention mechanism: long-range dependency modeling through anchor points;

[0103] 2. Channel attention: feature enhancement through the attention mechanism between channels;

[0104] 3. Window attention: feature modeling within a local window;

[0105] Q=W Q Φ Z ,K=W K Φ Z ,V=W V Φ Z ;

[0106] Stripe Attention:

[0107] Channel Attention: Φ CA = ChannelAttention(Φ);

[0108] Window attention: Φ WA =WindowAttention(Φ);

[0109] Final output: Φ out =Concat(Y,Φ CA,Φ WA );

[0110] Among them, W Q ,W K ,W V is the weight matrix used to calculate query, key and value, A is the anchor point, and d is the feature dimension.

[0111] 3. Feature fusion of dense fusion blocks;

[0112] After feature enhancement is completed, multi-scale features are fused through dense fusion blocks. The dense fusion block adopts a dense connection strategy to ensure that the feature information of each scale is fully utilized during the fusion process. First, the feature group of each scale is input into the fusion block for further feature processing, and the features are transformed and optimized using convolutional layers, batch normalization layers, and activation function layers. Then, an adaptive pooling strategy is used to expand the receptive field of the feature and enhance the expressive power of the feature. Finally, the multi-scale features are fused layer by layer in a cascade manner to generate the final fused image. The calculation process of the dense fusion block is as follows:

[0113] M i,1 =CBR(M i,0 );

[0114] M 4,2 =ASPP(M 4,1 );

[0115]

[0116] Final output: M fused =Concat(M 1,2 ,M 2,2 ,M 3,2 ,M 4,2 );

[0117] Among them, M i,1 (i=1,2,3,4), the CBR module contains convolutional layers, batch normalization layers and activation function layers. ASPP is used to expand the receptive field of features, and UP represents upsampling operation. Represents a feature concatenation operation.

[0118] In order to ensure the contrast and texture details of the fused image, the present invention designs a loss function that includes intensity loss and gradient loss. The intensity loss is based on the intensity information of the source image to ensure that the overall contrast of the fused image is consistent with human visual perception; the gradient loss prevents the fused image from generating blurred edges by maintaining the texture information of the visible light image. The calculation process is as follows:

[0119]

[0120] Among them, H and W represent the height and width of the image, S represents the Sobel operator, and Δ represents the L1 loss calculation. represents the fused image, and λ is the weight parameter. This process ensures that the fused image presents a large gradient for the change of the target edge pixel value and retains rich texture details.

[0121] It should be noted that the present application also has the following advantages: effectively fusing the information in the infrared image and the visible light image to generate a composite image that contains both the target object information in the infrared image and the rich texture details in the visible light image, thereby overcoming the limitations of a single image sensor.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0123] In this embodiment, an image fusion device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0124] Figure 5 is a structural block diagram of an image fusion device according to an embodiment of the present application, the device comprising:

[0125] An extraction module 502 is used to perform feature extraction on the infrared image and the visible light image respectively through a multi-scale dual-channel encoder to obtain a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image, wherein the first feature set and the second feature set both include features of N scales, where N is an integer greater than or equal to 2;

[0126] An enhancement module 504 is used to enhance each feature in the first feature set and the second feature set through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, wherein the group attention module has a plurality of mixed attention layers, and each mixed attention layer has a stripe attention mechanism, a channel attention mechanism, and a window attention mechanism;

[0127] The fusion module 506 is used to perform feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image.

[0128] The above device uses a multi-scale dual-channel encoder to extract features of infrared images and visible light images at multiple scales respectively, and uses a group attention module to enhance the features of each scale in terms of stripe attention, channel attention and window attention, and finally fuses the enhanced features, thereby realizing the fusion of the features of the infrared image and the features of the visible light image, improving the fusion effect of the infrared image and the visible light image, and solving the problem of poor fusion effect of the infrared image and the visible light image.

[0129] In an exemplary embodiment, the extraction module 502 is further configured to extract features from the infrared image using the following formula through the multi-scale dual-channel encoder: in, is the i-th layer encoder corresponding to the infrared image in the multi-scale dual-channel encoder, I r is the infrared image, Φ r,i is the extracted feature of the infrared image at the i-th scale, i is 1, ..., N-1, N in sequence; and the feature of the visible light image is extracted using the following formula through the multi-scale dual-channel encoder: in, is the i-th layer encoder corresponding to the visible light image in the multi-scale dual-channel encoder, I v is the visible light image, Φ v,i is the extracted feature of the visible light image at the i-th scale.

[0130] In an exemplary embodiment, the enhancement module 504 is also used to enhance the j-th feature in the first feature set and the second feature set in the following manner, where j is 1, ..., 2N-1, 2N in sequence: the j-th feature is processed by the M mixed attention layers in the group attention module to obtain a reference feature, and the reference feature is convolved to obtain a target feature, wherein the input of the first mixed attention layer in the M mixed attention layers is the j-th feature, the input of the m-th mixed attention layer in the M mixed attention layers is the output of the m-1-th mixed attention layer, M is an integer greater than or equal to 2, and m is an integer greater than or equal to 1 and less than or equal to M; the target feature and the j-th feature are concatenated to obtain an enhanced j-th feature, wherein the reference feature is the output of the M-th mixed attention layer in the M mixed attention layers.

[0131] In an exemplary embodiment, the enhancement module 504 is also used to loop the following operations until m is equal to M, wherein, in the case where the following operations are performed for the first time, m is equal to 1: perform the following operations through the mth mixed attention layer among the M mixed attention layers, and after performing the following operations, update the value of m to m+1: process the input of the mth mixed attention layer based on the stripe attention mechanism to obtain a first feature; process the input of the mth mixed attention layer based on the window attention mechanism to obtain a second feature; process the input of the mth mixed attention layer based on the channel attention mechanism to obtain a third feature; concatenate the first feature, the second feature and the third feature to obtain the output of the mth mixed attention layer.

[0132] In an exemplary embodiment, the enhancement module 504 is further configured to obtain the first feature based on the stripe attention mechanism using the following formula: Y=M h ·(M v ·V), where Softmax is the activation function and Q = W Q Φ Z , K=W K Φ Z , V=W V Φ Z , W Q , W K , W V are weight matrices used to calculate queries, keys, and values, respectively. A is the anchor point, Φ Z is the input of the mth mixed attention layer, and d is Φ Z The feature dimension of Y is the first feature.

[0133] In an exemplary embodiment, the fusion module 506 is further used to splice the features of the i-th scale in the enhanced first feature set and the features of the i-th scale in the enhanced second feature set to obtain N fused features, where i is 1,...,N-1,N in sequence; and fuse the N fused features to obtain the fused image.

[0134] In an exemplary embodiment, the fusion module 506 is further configured to, when N is equal to 4, process the N fusion features in the following manner to obtain N fusion features after processing, and splice the N fusion features after processing to obtain the fusion image: i,1 =CBR(M i,0 );M 4,2 =ASPP(M 4,1 ); Among them, M i,0 is the i-th fused feature among the N fused features. CBR includes a convolution layer, a batch normalization layer, and an activation function layer. ASPP is used to expand the receptive field of the feature. UP represents an upsampling operation. Represents a feature concatenation operation, and the N fused features after the processing include: M 1,2 , M 2,2 , M 3,2 , M 4,2 .

[0135] In an exemplary embodiment, the apparatus further includes: a determination module, configured to determine the loss of the fused image by the following method after the enhanced first feature set and the enhanced second feature set are fused to obtain a fused image, wherein the loss of the fused image is used to represent the fusion effect of the fused image: total =L intensity +λL gradient ; Where H is the height of the image, W is the width of the image, h is an integer greater than or equal to 0 and less than or equal to H-1, w is an integer greater than or equal to 0 and less than or equal to W-1, S represents the Sobel operator, and Δ represents the L1 loss calculation. represents the pixel value of the fused image at the (h,w) position, is the pixel value of the infrared image at the (h,w) position, is the pixel value of the visible light image at position (h, w), λ is the weight parameter, L total is the loss of the fused image.

[0136] It should be noted that in order to ensure the contrast and texture details of the fused image, a loss function including intensity loss and gradient loss is designed in the embodiment of the present application. The intensity loss is based on the intensity information of the source image to ensure that the overall contrast of the fused image is consistent with human visual perception; the gradient loss prevents the fused image from generating blurred edges by maintaining the texture information of the visible light image.

[0137] It should be noted that, through the above steps, it can be ensured that the fused image presents a large gradient for the change of the target edge pixel value and retains rich texture details.

[0138] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0139] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0140] S1, extracting features from the infrared image and the visible light image respectively by a multi-scale dual-channel encoder to obtain a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image, wherein the first feature set and the second feature set both include features of N scales, where N is an integer greater than or equal to 2;

[0141] S2, respectively enhancing each feature in the first feature set and the second feature set through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, wherein the group attention module has a plurality of mixed attention layers, and each mixed attention layer has a stripe attention mechanism, a channel attention mechanism, and a window attention mechanism;

[0142] S3, performing feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image.

[0143] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0144] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0145] An embodiment of the present application further provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps in any one of the above method embodiments are performed.

[0146] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0147] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0148] S1, extracting features from the infrared image and the visible light image respectively by a multi-scale dual-channel encoder to obtain a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image, wherein the first feature set and the second feature set both include features of N scales, where N is an integer greater than or equal to 2;

[0149] S2, respectively enhancing each feature in the first feature set and the second feature set through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, wherein the group attention module has a plurality of mixed attention layers, and each mixed attention layer has a stripe attention mechanism, a channel attention mechanism, and a window attention mechanism;

[0150] S3, performing feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image.

[0151] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0152] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0153] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0154] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image fusion method, characterized in that: include: Performing feature extraction on the infrared image and the visible light image respectively through a multi-scale dual-channel encoder to obtain a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image, wherein the first feature set and the second feature set both include features of N scales, where N is an integer greater than or equal to 2; Performing enhancement processing on each feature in the first feature set and the second feature set respectively through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, wherein the group attention module has a plurality of mixed attention layers, and each mixed attention layer has a stripe attention mechanism, a channel attention mechanism, and a window attention mechanism; The enhanced first feature set and the enhanced second feature set are subjected to feature fusion to obtain a fused image.

2. The method according to claim 1, characterized in that: The infrared image and visible light image are respectively feature extracted through a multi-scale dual-channel encoder, including: The multi-scale dual-channel encoder is used to extract features from the infrared image using the following formula: in, is the i-th layer encoder corresponding to the infrared image in the multi-scale dual-channel encoder, I r is the infrared image, Φ r,i is the extracted feature of the infrared image at the i-th scale, where i is 1, ..., N-1, N in sequence; and The multi-scale dual-channel encoder is used to extract features from the visible light image using the following formula: in, is the i-th layer encoder corresponding to the visible light image in the multi-scale dual-channel encoder, I v is the visible light image, Φ v,i is the extracted feature of the visible light image at the i-th scale.

3. The method according to claim 1, characterized in that Performing enhancement processing on each feature in the first feature set and the second feature set respectively through a group attention module includes: The j-th feature in the first feature set and the second feature set is enhanced in the following manner, where j is 1, ..., 2N-1, 2N in sequence: The j-th feature is processed by the M mixed attention layers in the group attention module to obtain a reference feature, and the reference feature is convolved to obtain a target feature, wherein the input of the first mixed attention layer in the M mixed attention layers is the j-th feature, the input of the m-th mixed attention layer in the M mixed attention layers is the output of the m-1-th mixed attention layer, M is an integer greater than or equal to 2, and m is an integer greater than or equal to 1 and less than or equal to M; The target feature and the j-th feature are concatenated to obtain an enhanced j-th feature, wherein the reference feature is the output of the M-th mixed attention layer among the M mixed attention layers.

4. The method according to claim 3, characterized in that: The j-th feature is processed by the M mixed attention layers in the group attention module to obtain a processed feature, including: The following operations are performed cyclically until m is equal to M, where m is equal to 1 in the first case of the following operations: The following operations are performed through the mth mixed attention layer in the M mixed attention layers, and after performing the following operations, the value of m is updated to m+1: Processing the input of the m-th mixed attention layer based on the stripe attention mechanism to obtain a first feature; Processing the input of the m-th mixed attention layer based on the window attention mechanism to obtain a second feature; Processing the input of the m-th mixed attention layer based on the channel attention mechanism to obtain a third feature; The first feature, the second feature and the third feature are concatenated to obtain the output of the mth mixed attention layer.

5. The method according to claim 4, characterized in that The input of the m-th mixed attention layer is processed based on the stripe attention mechanism to obtain a first feature, including: Based on the stripe attention mechanism, the first feature is obtained using the following formula: Y=M h ·(M v ·V) Among them, Softmax is the activation function, Q = W Q Φ Z , K=W K Φ Z , V=W V Φ Z , W Q , W K , W V are weight matrices used to calculate queries, keys, and values, respectively. A is the anchor point, Φ Z is the input of the mth mixed attention layer, and d is Φ Z The feature dimension of Y is the first feature.

6. The method according to claim 1, characterized in that The method of fusing the enhanced first feature set and the enhanced second feature set to obtain a fused image includes: The features of the i-th scale in the enhanced first feature set and the features of the i-th scale in the enhanced second feature set are concatenated to obtain N fused features, where i is 1, ..., N-1, N in sequence; The N fusion features are fused to obtain the fused image.

7. The method according to claim 6, characterized in that The N fusion features are fused to obtain the fused image, including: When N is equal to 4, the N fusion features are processed in the following manner to obtain N fusion features after processing, and the N fusion features after processing are spliced ​​to obtain the fusion image: M i,1 =CBR(M i,0 ); M 4,2 =ASPP(M 4,1 ); Among them, M i,0 is the i-th fused feature among the N fused features. CBR includes a convolution layer, a batch normalization layer, and an activation function layer. ASPP is used to expand the receptive field of the feature. UP represents an upsampling operation. Represents a feature concatenation operation, and the N fused features after the processing include: M 1,2 , M 2,2 , M 3,2 , M 4,2 。 8. The method according to claim 1, characterized in that: After fusing the enhanced first feature set and the enhanced second feature set to obtain a fused image, the method further includes: determining a loss of the fused image in the following manner, wherein the loss of the fused image is used to represent a fusion effect of the fused image: Where H is the height of the image, W is the width of the image, h is an integer greater than or equal to 0 and less than or equal to H-1, w is an integer greater than or equal to 0 and less than or equal to W-1, S represents the Sobel operator, and Δ represents the L1 loss calculation. represents the pixel value of the fused image at position (h, w), is the pixel value of the infrared image at the position (h, w), is the pixel value of the visible light image at position (h, w), λ is the weight parameter, L total is the loss of the fused image.

9. An image fusion device, characterized in that: include: an extraction module, configured to perform feature extraction on the infrared image and the visible light image respectively through a multi-scale dual-channel encoder to obtain a first feature set corresponding to the infrared image and a second feature set corresponding to the visible light image, wherein the first feature set and the second feature set both include features of N scales, where N is an integer greater than or equal to 2; An enhancement module, configured to enhance each feature in the first feature set and the second feature set respectively through a group attention module to obtain an enhanced first feature set and an enhanced second feature set, wherein the group attention module has a plurality of mixed attention layers, and each mixed attention layer has a stripe attention mechanism, a channel attention mechanism, and a window attention mechanism; The fusion module is used to perform feature fusion on the enhanced first feature set and the enhanced second feature set to obtain a fused image.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images