Infrared image fusion method, system and device based on multi-scale feature extraction

Abstract

The present application relates to the field of image processing, and provide an infrared image fusion method, system and equipment based on multi-scale feature extraction, including using multi-scale pyramid for extraction, obtaining layer image features; obtaining first layer image features from the layer image features, first cross-modal interaction collaboration and cross injection are carried out on the first layer image features, then difference feature calculation and cross injection are carried out, to obtain refined first layer image features; first feature fusion and channel attention extraction are carried out on the second layer image features, then channel attention re-extraction and final difference enhancement are carried out, to obtain refined second layer image features; cross-channel attention and cross-space attention are carried out on the third layer image features and the fourth layer image features respectively in turn, to obtain refined third layer image features and refined fourth layer image features respectively; cross-layer up sampling and residual feature distillation are carried out, to obtain reconstruction features, and the reconstruction features are reconstructed to obtain information complementary fusion image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an infrared image fusion method, system, and device based on multi-scale feature extraction. Background Technology

[0002] Due to hardware limitations in imaging equipment, sensors of a single type or with a single setup typically cannot fully characterize an imaging scene. For example, visible light images usually contain rich textural details but are susceptible to extreme environments and occlusion, causing them to lose targets within the scene. Conversely, infrared sensors, by capturing thermal radiation information emitted by objects, can effectively highlight prominent targets such as pedestrians and vehicles, but lack detailed descriptions of the scene. Sensors of different types or with different optical setups often contain a wealth of complementary information; therefore, image fusion techniques have emerged.

[0003] Before deep learning swept through the field of computer vision, image fusion had already been extensively studied. Traditional image fusion algorithms typically involve performing activity level measurements in the spatial domain or transform domain (transforming the image to the transform domain through some mathematical transformation) and manually designing fusion rules to achieve image fusion. In recent years, deep learning has dominated the development of computer vision with its powerful feature extraction and representation capabilities. To overcome the shortcomings of traditional algorithms, researchers in the field of image fusion have explored a large number of deep learning-based image fusion algorithms, which has become a hot topic in current image fusion research. However, existing techniques generally suffer from problems such as not considering the differences in features at various scales and the lack of noise suppression processing during the upsampling process, leading to chessboard artifacts. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides an infrared image fusion method, system, and device based on multi-scale feature extraction, which realizes the fusion of infrared images and visible light images.

[0005] This invention provides an infrared image fusion method based on multi-scale feature extraction, comprising:

[0006] S1: Acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images;

[0007] S2: Obtain the first layer image features from the layer image features. First, perform cross-modal interactive collaboration and cross-injection on the first layer image features, and then perform differential feature calculation and cross-injection to obtain the refined first layer image features.

[0008] S3: Obtain the second layer of image features. First, perform feature fusion and channel attention extraction on the second layer of image features. Then, perform channel attention re-extraction and final difference enhancement to obtain refined second layer of image features.

[0009] S4: Obtain the third-layer image features and the fourth-layer image features. Perform cross-channel attention and cross-space attention on the third-layer image features and the fourth-layer image features respectively to obtain the refined third-layer image features and the refined fourth-layer image features respectively.

[0010] S5: Perform cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. Reconstruct the reconstructed features to obtain a complementary and fused image.

[0011] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, step S1 further includes:

[0012] S11: Acquire infrared and visible light images respectively, establish a shallow feature extraction module, a medium feature extraction module and a deep feature extraction module, and establish the multi-scale pyramid through the shallow feature extraction module, the medium feature extraction module and the deep feature extraction module;

[0013] S12: Input the infrared image and the visible light image into the multi-scale pyramid and extract the infrared image and the visible light image layer by layer to obtain the layer image features.

[0014] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S2, after obtaining the first layer of image features including the first layer of infrared features and the first layer of visible light features, the first layer of infrared features and the first layer of visible light features are subjected to cross-modal interactive collaboration through cascaded convolution operation to obtain a channel attention map. The channel attention map is then cross-injected with the first layer of image features to obtain the intermediate first layer of image features.

[0015] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S2, the intermediate first layer image features are spliced ​​and the difference features are calculated to obtain a difference attention map. The difference attention map and the intermediate first layer image features are cross-injected to obtain the refined first layer image features.

[0016] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S3, after obtaining the second layer image features including the second layer infrared features and the second layer visible light features, the second layer infrared features and the second layer visible light features are fused to obtain a feature fusion map. Channel attention extraction is performed on the feature fusion map to obtain a channel feature map. Channel modulation is performed based on the channel feature map to obtain the intermediate second layer image features.

[0017] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S3, channel attention re-extraction is performed on the intermediate second layer image features to obtain a re-extracted channel feature map, and final difference enhancement is performed based on the re-extracted channel feature map to obtain the refined second layer image features.

[0018] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, step S4 further includes:

[0019] S41: Obtain the third-layer image features and the fourth-layer image features, perform pyramid pooling and convolution projection on the third-layer image features and the fourth-layer image features respectively, and obtain the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively, and obtain the intermediate third-layer image features and the intermediate fourth-layer image features based on the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively.

[0020] S42: Perform pyramid pooling and convolution projection on the intermediate third-layer image features and the intermediate fourth-layer image features respectively to obtain third-layer image key-value pairs and fourth-layer image key-value pairs. Based on the third-layer image key-value pairs and the fourth-layer image key-value pairs, obtain refined third-layer image features and refined fourth-layer image features respectively.

[0021] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S5, deformable convolution is performed on the refined fourth-layer image features and the refined third-layer image features to obtain reconstructed fourth-layer image features, and residual feature distillation is performed on the reconstructed fourth-layer image features and the refined third-layer image features to obtain intermediate reconstructed third-layer image features.

[0022] Deformable convolution is performed on the intermediate reconstructed third-layer image features and the refined second-layer image features to obtain the reconstructed third-layer image features. Residual feature distillation is then performed on the reconstructed third-layer image features and the refined second-layer image features to obtain the intermediate reconstructed second-layer image features.

[0023] Deformable convolution is performed on the intermediate reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed second-layer image features. Residual feature distillation is then performed on the reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed features.

[0024] This invention also provides an infrared image fusion system based on multi-scale feature extraction, comprising:

[0025] Layer image feature module: used to acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images;

[0026] The first-layer image feature refinement module is used to obtain the first-layer image features from the layer image features. The first-layer image features are first subjected to cross-modal interactive collaboration and cross-injection, and then differential feature calculation and cross-injection are performed to obtain the refined first-layer image features.

[0027] The second-layer image feature refinement module is used to obtain the second-layer image features. The second-layer image features are first fused and channel attention extracted, then channel attention is extracted again and finally difference enhancement is performed to obtain the refined second-layer image features.

[0028] The module for refining the third and fourth layer image features is used to obtain the third and fourth layer image features. Cross-channel attention and cross-spatial attention are performed on the third and fourth layer image features respectively to obtain the refined third layer image features and the refined fourth layer image features respectively.

[0029] Image reconstruction module: This module performs cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. The reconstructed features are then used to reconstruct the image to obtain a complementary and fused image.

[0030] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the infrared image fusion method based on multi-scale feature extraction as described above.

[0031] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0032] The infrared image fusion method, system, and device based on multi-scale feature extraction provided by this invention designs different feature extraction modules for image features at different scales, thereby fully considering the features of each scale in both spatial and channel dimensions, greatly improving the overall performance of the fusion network framework, effectively suppressing noise in the image, making full use of the information in the image, and preserving the details in the image.

[0033] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating the infrared image fusion method based on multi-scale feature extraction provided by the present invention.

[0036] Figure 2 This is a comparison image of the fusion results of the infrared image fusion method based on multi-scale feature extraction provided by this invention.

[0037] Figure 3 This is a schematic diagram of the infrared image fusion system based on multi-scale feature extraction provided by the present invention.

[0038] Figure 4 This is a schematic diagram of the structure of the infrared image fusion device based on multi-scale feature extraction provided by the present invention.

[0039] Figure label:

[0040] 100. Layer image feature module; 200. First layer image feature refinement module; 300. Second layer image feature refinement module; 400. Third and fourth layer image feature refinement modules; 500. Image reconstruction module; 810. Processor; 820. Communication interface; 830. Memory; 840. Communication bus. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0042] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second" and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0043] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0044] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0045] The following is combined Figures 1 to 4 Specific embodiments of the present invention are described below. Figure 1 This is a flowchart illustrating the infrared image fusion method based on multi-scale feature extraction provided by the present invention. It includes:

[0046] S1: Acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images;

[0047] Furthermore, the objective of this stage is to extract layer image features from infrared and visible light images using a multi-scale pyramid. Specifically, step S1 further includes:

[0048] S11: Acquire infrared and visible light images respectively, establish a shallow feature extraction module, a medium feature extraction module and a deep feature extraction module, and establish the multi-scale pyramid through the shallow feature extraction module, the medium feature extraction module and the deep feature extraction module;

[0049] S12: Input the infrared image and the visible light image into the multi-scale pyramid and extract the infrared image and the visible light image layer by layer to obtain the layer image features.

[0050] The specific implementation method for the above steps in this embodiment is as follows:

[0051] First, it is necessary to acquire infrared images of the same target separately. and visible light images Next, shallow feature extraction, mid-level feature extraction, and deep feature extraction modules need to be established and stacked layer by layer to obtain a multi-scale pyramid. Finally, the infrared and visible light images are input into the multi-scale pyramid, which can then extract features layer by layer from the infrared and visible light images respectively.

[0052]

[0053]

[0054]

[0055]

[0056] in, The first layer of infrared features, The first layer of visible light characteristics, For the second layer of infrared features, This represents the second layer of visible light characteristics. The third layer of infrared features, This represents the third layer of visible light characteristics. This is the fourth layer of infrared features. This represents the fourth layer of visible light characteristics. This is a shallow infrared feature extraction module. This is a shallow visible light feature extraction module. This is a mid-layer infrared feature extraction module. This is a mid-layer visible light feature extraction module. This is the first deep infrared feature extraction module. This is the first deep visible light feature extraction module. This is the second deep infrared feature extraction module. This is the second deep visible light feature extraction module.

[0057] Each layer of the multi-scale pyramid is configured with different parameters, allowing image features to be extracted layer by layer from shallow to deep. This enables the low-level structural information of infrared and visible light images to establish a cross-scale feature synergy with the high-level semantic representation, thereby enhancing the ability of the fused image to retain detailed information and perceive salient targets.

[0058] S2: Obtain the first layer image features from the layer image features. First, perform cross-modal interactive collaboration and cross-injection on the first layer image features, and then perform differential feature calculation and cross-injection to obtain the refined first layer image features.

[0059] Furthermore, the objective of this stage is to refine the first-layer image features to obtain refined first-layer image features. Specifically, in step S2, after obtaining the first-layer image features including first-layer infrared features and first-layer visible light features, the first-layer infrared features and first-layer visible light features are subjected to cross-modal interactive collaboration through cascaded convolution operations to obtain a channel attention map. The channel attention map is then cross-injected with the first-layer image features to obtain intermediate first-layer image features.

[0060] In step S2, the intermediate first layer image features are spliced ​​and the difference features are calculated to obtain a difference attention map. The difference attention map and the intermediate first layer image features are cross-injected to obtain the refined first layer image features.

[0061] The specific implementation method for the above steps in this embodiment is as follows:

[0062] First, after obtaining the first layer of image features, which includes the first layer of infrared features and the first layer of visible light features, it is necessary to perform cross-modal interactive collaboration on the first layer of infrared features and the first layer of visible light features through cascaded convolution operations to obtain a channel attention map α that fuses multi-source information:

[0063]

[0064]

[0065] The `Concat()` function concatenates the contents within the parentheses. This represents a 3-level cascaded convolution operation. Note the diagram indicating the middle passage. As the activation function, GAP() indicates that the content within the parentheses is subjected to global average pooling.

[0066] Then, cross-injection is performed to obtain the first layer of image features in the middle, in order to further promote cross-modal interaction and collaborative representation:

[0067]

[0068]

[0069] in, This represents the first visible light image feature in the middle layer. The first layer of infrared image features is located in the middle. This is an element-wise multiplication operation.

[0070] Subsequently, the image features of the first intermediate layer are stitched together and difference features are calculated to obtain the difference attention map. This allows for the extraction of differential features and explicit modeling of differences between modalities, capturing highly complementary differential information.

[0071]

[0072]

[0073]

[0074] in, Note the first difference in the diagram. Note the second difference in the diagram. This is an element-by-element subtraction operation. This is a two-stage cascaded convolution operation. Cross-injection is then performed to supplement missing semantic information related to modality differences, resulting in refined first-layer image features:

[0075]

[0076]

[0077] in, This is an element-wise addition operation. To refine the features of the first layer of infrared image, To refine the features of the first layer of visible light image.

[0078] S3: Obtain the second layer of image features. First, perform feature fusion and channel attention extraction on the second layer of image features. Then, perform channel attention re-extraction and final difference enhancement to obtain refined second layer of image features.

[0079] Furthermore, the objective of this stage is to obtain refined second-layer image features. Specifically, in step S3, after obtaining the second-layer image features including second-layer infrared features and second-layer visible light features, feature fusion is performed on the second-layer infrared features and the second-layer visible light features to obtain a feature fusion map. Channel attention extraction is then performed on the feature fusion map to obtain a channel feature map. Based on the channel feature map, channel modulation is performed to obtain the intermediate second-layer image features.

[0080] In step S3, channel attention is re-extracted from the intermediate second layer image features to obtain a re-extracted channel feature map. Based on the re-extracted channel feature map, final difference enhancement is performed to obtain the refined second layer image features.

[0081] The specific implementation method for the above steps in this embodiment is as follows:

[0082] First, to enhance channel selectivity, a second layer of image features, including a second layer of infrared features and a second layer of visible light features, is obtained. Then, the second layer of infrared features and the second layer of visible light features are fused to obtain a feature fusion map. This allows for the simultaneous characterization of the overall activation intensity and fluctuations of each channel. Channel attention extraction is then performed to obtain channel feature map A.

[0083]

[0084]

[0085] in, For the first channel, pay attention to extracting the weight coefficients. For the second channel, weighting coefficients are extracted, and SD represents the standard deviation within parentheses. Subsequently, to preserve basic structural information, channel modulation is performed to obtain the intermediate second-layer image features. :

[0086]

[0087] Next, to further construct explicit intermodal differences, a parallel channel attention mechanism is introduced in the channel dimension to re-extract the image features of the intermediate second layer, resulting in a re-extracted channel feature map. :

[0088]

[0089] in, For the first channel, pay attention to extracting coefficients again. To further extract coefficients for the second channel, and finally to effectively suppress redundant information accumulation and improve the discriminative power of the fusion result, highlighting areas of significant difference, a final difference enhancement is performed to obtain refined second-layer image features:

[0090]

[0091]

[0092] in, To refine the features of the second layer of infrared image, To refine the features of the second layer of visible light image.

[0093] S4: Obtain the third-layer image features and the fourth-layer image features. Perform cross-channel attention and cross-space attention on the third-layer image features and the fourth-layer image features respectively to obtain the refined third-layer image features and the refined fourth-layer image features respectively.

[0094] Furthermore, the objective of this stage is to sequentially perform cross-channel attention and cross-spatial attention on the third-layer and fourth-layer image features, respectively, thereby obtaining refined third-layer and refined fourth-layer image features. Specifically, step S4 further includes:

[0095] S41: Obtain the third-layer image features and the fourth-layer image features, perform pyramid pooling and convolution projection on the third-layer image features and the fourth-layer image features respectively, and obtain the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively, and obtain the intermediate third-layer image features and the intermediate fourth-layer image features based on the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively.

[0096] S42: Perform pyramid pooling and convolution projection on the intermediate third-layer image features and the intermediate fourth-layer image features respectively to obtain third-layer image key-value pairs and fourth-layer image key-value pairs. Based on the third-layer image key-value pairs and the fourth-layer image key-value pairs, obtain refined third-layer image features and refined fourth-layer image features respectively.

[0097] The specific implementation method for the above steps in this embodiment is as follows:

[0098] Since deep features typically contain more abstract and semantic representations and tend to encode shared semantic content between the two modalities, employing an explicit cross-attention mechanism for modeling in this step is more conducive to modality fusion. Specifically, cross-channel attention helps capture global semantic redundancy and complementary relationships between modalities, while cross-spatial attention focuses more on local structural differences, thereby enhancing structural consistency. Specifically, pyramid pooling and convolutional projection are performed on the third and fourth layer image features, respectively, to obtain the intermediate image key-value pairs for the third and fourth layers, respectively.

[0099] Specifically, pyramid pooling and convolutional projection are first applied sequentially to the third and fourth layer image features, respectively. During pyramid pooling, spatial pyramid pooling is used for the third layer features, while pyramid pooling is used for the fourth layer features. By adjusting the parameters during pyramid pooling and convolutional projection, the intermediate infrared bonds of the i-th layer can be obtained. The visible optical bonds in the middle of the i-th layer Infrared query of the i-th layer Querying visible light in the middle of the i-th layer Infrared values ​​of the i-th layer and the visible light value in the middle of the i-th layer In step S4, the value of i is either 3 or 4. When i=3, the intermediate key, intermediate query, and intermediate value of the i-th layer are obtained by sequentially performing pyramid pooling and convolutional projection on the image features of the third layer, and the resulting intermediate key, intermediate query, and intermediate value constitute the key-value pair of the third layer intermediate image. When i=4, the intermediate key, intermediate query, and intermediate value of the i-th layer are obtained by sequentially performing pyramid pooling and convolutional projection on the image features of the fourth layer, and the key, query, and value at this time constitute the key-value pair of the fourth layer intermediate image. The parameter settings of the convolutional projection function and pyramid pooling are different when obtaining the key, query, and value.

[0100] Then, based on the key-value pairs of the third and fourth intermediate image layers, the features of the third and fourth intermediate image layers are obtained respectively:

[0101]

[0102]

[0103] Where i=3, This refers to the visible light image features of the third intermediate layer. For the third layer of infrared image features, the key-value pairs of the third layer image and the third layer image features are used. When i=4, This represents the visible light image features of the fourth layer in the middle. The fourth layer of infrared image features are obtained by using the key-value pairs of the fourth layer image and the features of the fourth layer image, thus obtaining the third layer image features and the fourth layer image features.

[0104] Then, using the method described above, pyramid pooling and convolutional projection are performed sequentially on the image features of the third and fourth intermediate layers, respectively, to obtain the infrared bonds of the i-th layer. The i-th layer of visible light bonds Infrared query of the i-th layer Visible light query of layer i Infrared values ​​of the i-th layer and the visible light value of the i-th layer Here, the value of i is either 3 or 4. When i=3, the key, query, and value of the i-th layer are obtained by sequentially performing pyramid pooling and convolutional projection on the features of the intermediate third layer image, and the resulting key, query, and value constitute the key-value pairs of the third layer image. When i=4, the key, query, and value of the i-th layer are obtained by sequentially performing pyramid pooling and convolutional projection on the features of the intermediate fourth layer image, and the resulting key, query, and value constitute the key-value pairs of the fourth layer image. The parameter settings of the convolutional projection function and pyramid pooling are different when obtaining the key, query, and value.

[0105] Finally, we obtain the refined third-layer image features and the refined fourth-layer image features:

[0106]

[0107]

[0108] Where i=3, To refine the features of the third layer of visible light image, To refine the features of the third layer of infrared image, the third layer image key-value pairs and intermediate third layer image features are used. When i=4, To refine the features of the fourth layer of visible light image, To refine the features of the fourth layer of infrared image, the key-value pairs of the fourth layer image and the intermediate features of the fourth layer image are used, thus obtaining the refined features of the third layer image and the refined features of the fourth layer image.

[0109] S5: Perform cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. Reconstruct the reconstructed features to obtain a complementary and fused image.

[0110] Furthermore, the objective of this stage is to perform cross-layer upsampling and residual feature distillation to obtain reconstructed features, and then reconstruct the reconstructed features to obtain an information-complementary fused image. Specifically, in step S5, deformable convolution is performed on the refined fourth-layer image features and the refined third-layer image features to obtain reconstructed fourth-layer image features. Residual feature distillation is then performed on the reconstructed fourth-layer image features and the refined third-layer image features to obtain intermediate reconstructed third-layer image features.

[0111] Deformable convolution is performed on the intermediate reconstructed third-layer image features and the refined second-layer image features to obtain the reconstructed third-layer image features. Residual feature distillation is then performed on the reconstructed third-layer image features and the refined second-layer image features to obtain the intermediate reconstructed second-layer image features.

[0112] Deformable convolution is performed on the intermediate reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed second-layer image features. Residual feature distillation is then performed on the reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed features.

[0113] The specific implementation method for the above steps in this embodiment is as follows:

[0114] Here, deformable convolution is used to achieve cross-layer upsampling. First, deformable convolution is performed on the thinned fourth layer image features and the thinned third layer image features to obtain the reconstructed fourth layer image features:

[0115]

[0116] in, To reconstruct the features of the fourth layer of infrared image, To reconstruct the features of the fourth layer of visible light image, To perform infrared fourth-layer deformable convolution on the content within parentheses, To perform a visible light fourth-layer deformable convolution on the content within the brackets.

[0117] By reconstructing the fourth layer of image features and refining the third layer of image features, residual feature distillation is performed to obtain the intermediate reconstructed third layer of image features:

[0118]

[0119] in, To perform infrared third-layer residual feature distillation on the content within parentheses, To perform visible light third-layer residual characteristic distillation on the content within parentheses, To reconstruct the third layer of image features in the infrared intermediate layer, The third layer of image features is reconstructed in the middle of the visible light image.

[0120] Deformable convolution is performed on the reconstructed third-layer image features and the refined second-layer image features to obtain the reconstructed third-layer image features:

[0121]

[0122] in, To perform infrared third-layer deformable convolution on the content within parentheses, To perform a visible light third-layer deformable convolution on the content within the brackets, To reconstruct the third layer of image features using infrared technology, Reconstruct the third layer of image features using visible light.

[0123] By reconstructing the third-layer image features and refining the second-layer image features, residual feature distillation is performed to obtain the intermediate reconstructed second-layer image features:

[0124]

[0125] in, To perform infrared second-layer residual feature distillation on the content within parentheses, To perform visible light second-layer residual feature distillation on the content within parentheses, To reconstruct the second layer of image features from the infrared intermediate layer, Reconstruct the second layer of image features in the visible light intermediate layer.

[0126] Deformable convolution is performed on the reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed second-layer image features:

[0127]

[0128] in, To perform infrared second-layer deformable convolution on the content within parentheses, To perform a second-layer deformable convolution on the content within the brackets using visible light, To reconstruct the second layer of image features using infrared technology, The second layer of image features is reconstructed using visible light.

[0129] Finally, residual feature distillation is performed by reconstructing the second layer of image features and refining the first layer of image features to obtain the reconstructed features:

[0130]

[0131] in, To perform infrared first-layer residual feature distillation on the content within parentheses, To perform visible light first-layer residual feature distillation on the content within parentheses, For infrared reconstruction features, This is for visible light reconstruction features. The parameters of residual feature distillation and deformable convolution for each of the above layers, as well as the parameters of residual feature distillation and deformable convolution for infrared and visible light, all need to be set according to actual needs. Cross-layer upsampling is achieved through deformable convolution.

[0132] Finally, the reconstructed features are rebuilt to obtain a complementary and fused image. :

[0133]

[0134] Here, Sigmoid() inputs the content within the parentheses into the Sigmoid function, and Conv() performs convolution on the content within the parentheses.

[0135] The effectiveness of the infrared image fusion method based on multi-scale feature extraction provided by this invention is described below. First, images from the publicly available TNO dataset are used for testing. DeFusion, ITFuse, YDTR, TUFusion, LRRNet, RFN-Nest, and SpTFuse are all existing infrared image fusion methods. Table 1 compares the fusion results of this method with other methods in the TNO dataset.

[0136] Table 1. Comparison of evaluation results of our method with other fusion methods in the TNO dataset:

[0137]

[0138] Wherein, MI stands for Mutual Information, which measures the amount of information shared between images; a larger value indicates more complete information sharing. SF stands for Spatial Frequency, which measures the overall drastic changes in an image; a larger value indicates a higher degree of detail enhancement. AG stands for Average Gradient, which reflects image sharpness; a larger value indicates a sharper image. VIF stands for Visual Information Fidelity, and Q... abfThe objective non-reference quality evaluation index for image fusion is denoted by , where a higher value indicates richer preservation of important edge information in the image. SSIM (Structural Similarity Index) represents the higher similarity between the fused and unfused images; bolded numbers indicate that this value is the optimal value among all methods. On the TNO dataset, this method demonstrates significantly better infrared target contrast compared to other methods, while also preserving visible light texture details well, exhibiting low sky noise and high clarity, achieving the best values ​​across all metrics.

[0139] Subsequently, tests were also conducted on images on the publicly available M3FD dataset. Table 2 shows a comparison of the fusion results of this method with other methods on the MFID dataset:

[0140] Table 2. Comparison of fusion results between our method and other methods on the MFID dataset.

[0141]

[0142] The bolded numbers indicate that this value is the best among all methods. On the M3FD dataset, this method preserves the texture details of trees and mountains more clearly and the target information of people more prominently compared with other methods, achieving the best value in all metrics.

[0143] Finally, images from the publicly available RoadScene dataset were used for testing, and the fusion results are as follows: Figure 2 As shown in Table 3, this table compares the evaluation results of the fusion of our method with other methods in the RoadScene dataset:

[0144] Table 3. Comparison of fusion results between this method and other methods in the RoadScene dataset.

[0145]

[0146] The bolded numbers indicate that this value is the best among all methods. As can be seen from the comparison results on the RoadScene dataset, this method has moderate contrast, significant infrared target information, and preserves ground texture details while minimizing sky noise, achieving the best value in all metrics.

[0147] Figure 3 A schematic diagram of an infrared image fusion system based on multi-scale feature extraction is shown, such as... Figure 3 As shown, the infrared image fusion method based on multi-scale feature extraction described above includes:

[0148] Layer image feature module 100: used to acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images.

[0149] The first-layer image feature refinement module 200 is used to obtain the first-layer image features from the layer image features. The first-layer image features are first subjected to cross-modal interactive collaboration and cross-injection, and then differential feature calculation and cross-injection are performed to obtain the refined first-layer image features.

[0150] The second-layer image feature refinement module 300 is used to obtain the second-layer image features. The second-layer image features are first subjected to feature fusion and channel attention extraction, followed by channel attention re-extraction and final difference enhancement to obtain the refined second-layer image features.

[0151] The module 400 for refining the third and fourth layer image features is used to obtain the third layer image features and the fourth layer image features. Cross-channel attention and cross-spatial attention are performed on the third layer image features and the fourth layer image features respectively to obtain the refined third layer image features and the refined fourth layer image features respectively.

[0152] Image reconstruction module 500: It is used to perform cross-layer upsampling and residual feature distillation on the refined fourth layer image features, refined third layer image features, refined second layer image features and refined first layer image features in sequence to obtain reconstructed features, and reconstruct the reconstructed features to obtain a complementary and fused image.

[0153] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call a computer program in the memory 830 to execute an infrared image fusion method based on multi-scale feature extraction, the method including:

[0154] S1: Acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images;

[0155] S2: Obtain the first layer image features from the layer image features. First, perform cross-modal interactive collaboration and cross-injection on the first layer image features, and then perform differential feature calculation and cross-injection to obtain the refined first layer image features.

[0156] S3: Obtain the second layer of image features. First, perform feature fusion and channel attention extraction on the second layer of image features. Then, perform channel attention re-extraction and final difference enhancement to obtain refined second layer of image features.

[0157] S4: Obtain the third-layer image features and the fourth-layer image features. Perform cross-channel attention and cross-space attention on the third-layer image features and the fourth-layer image features respectively to obtain the refined third-layer image features and the refined fourth-layer image features respectively.

[0158] S5: Perform cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. Reconstruct the reconstructed features to obtain a complementary and fused image.

[0159] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An infrared image fusion method based on multi-scale feature extraction, characterized in that, include: S1: Acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images; Step S1 further includes: S11: Acquire infrared and visible light images respectively, establish a shallow feature extraction module, a medium feature extraction module and a deep feature extraction module, and establish the multi-scale pyramid through the shallow feature extraction module, the medium feature extraction module and the deep feature extraction module; S12: Input the infrared image and the visible light image into the multi-scale pyramid and extract the features of each layer of the infrared image and the visible light image layer by layer; the formula for the multi-scale pyramid is: in, The first layer of infrared features, The first layer of visible light characteristics, For the second layer of infrared features, This represents the second layer of visible light characteristics. The third layer of infrared features, This represents the third layer of visible light characteristics. This is the fourth layer of infrared features. This represents the fourth layer of visible light characteristics. This is a shallow infrared feature extraction module. This is a shallow visible light feature extraction module. This is a mid-layer infrared feature extraction module. This is a mid-layer visible light feature extraction module. This is the first deep infrared feature extraction module. This is the first deep visible light feature extraction module. This is the second deep infrared feature extraction module. This is the second deep visible light feature extraction module; S2: Obtain the first layer image features from the layer image features. First, perform cross-modal interactive collaboration and cross-injection on the first layer image features, and then perform differential feature calculation and cross-injection to obtain the refined first layer image features. S3: Obtain the second layer of image features. First, perform feature fusion and channel attention extraction on the second layer of image features, then perform channel attention re-extraction and final difference enhancement to obtain refined second layer of image features. S4: Obtain the third-layer image features and the fourth-layer image features. Perform cross-channel attention and cross-space attention on the third-layer image features and the fourth-layer image features respectively to obtain the refined third-layer image features and the refined fourth-layer image features respectively. Step S4 further includes: S41: Obtain the third-layer image features and the fourth-layer image features, perform pyramid pooling and convolution projection on the third-layer image features and the fourth-layer image features respectively, and obtain the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively, and obtain the intermediate third-layer image features and the intermediate fourth-layer image features based on the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively. S42: Perform pyramid pooling and convolution projection on the intermediate third-layer image features and the intermediate fourth-layer image features respectively to obtain third-layer image key-value pairs and fourth-layer image key-value pairs. Based on the third-layer image key-value pairs and fourth-layer image key-value pairs, obtain refined third-layer image features and refined fourth-layer image features respectively. S5: Perform cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. Reconstruct the reconstructed features to obtain a complementary and fused image. In step S5, deformable convolution is performed on the refined fourth layer image features and the refined third layer image features to obtain the reconstructed fourth layer image features. Residual feature distillation is then performed on the reconstructed fourth layer image features and the refined third layer image features to obtain the intermediate reconstructed third layer image features. Deformable convolution is performed on the intermediate reconstructed third-layer image features and the refined second-layer image features to obtain the reconstructed third-layer image features. Residual feature distillation is then performed on the reconstructed third-layer image features and the refined second-layer image features to obtain the intermediate reconstructed second-layer image features. Deformable convolution is performed on the intermediate reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed second-layer image features. Residual feature distillation is then performed on the reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed features.

2. The method for infrared image fusion based on multi-scale feature extraction according to claim 1, characterized in that, In step S2, after obtaining the first layer of image features including the first layer of infrared features and the first layer of visible light features, the first layer of infrared features and the first layer of visible light features are subjected to cross-modal interactive collaboration through cascaded convolution operations to obtain a channel attention map. The channel attention map is then cross-injected with the first layer of image features to obtain the intermediate first layer of image features.

3. The infrared image fusion method based on multi-scale feature extraction according to claim 2, characterized in that, In step S2, the intermediate first layer image features are spliced ​​and the difference features are calculated to obtain a difference attention map. The difference attention map and the intermediate first layer image features are cross-injected to obtain the refined first layer image features.

4. The method for infrared image fusion based on multi-scale feature extraction according to claim 1, characterized in that, In step S3, after obtaining the second layer image features including the second layer infrared features and the second layer visible light features, the second layer infrared features and the second layer visible light features are fused to obtain a feature fusion map. Channel attention extraction is performed on the feature fusion map to obtain a channel feature map. Channel modulation is performed based on the channel feature map to obtain the intermediate second layer image features.

5. The method for infrared image fusion based on multiscale feature extraction according to claim 4, characterized in that, In step S3, channel attention is re-extracted from the intermediate second layer image features to obtain a re-extracted channel feature map. Based on the re-extracted channel feature map, final difference enhancement is performed to obtain the refined second layer image features.

6. An infrared image fusion system based on multi-scale feature extraction for performing the method of infrared image fusion based on multi-scale feature extraction according to any one of claims 1 to 5, characterized in that, include: Layer image feature module: used to acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images; The first-layer image feature refinement module is used to obtain the first-layer image features from the layer image features. The first-layer image features are first subjected to cross-modal interactive collaboration and cross-injection, and then differential feature calculation and cross-injection are performed to obtain the refined first-layer image features. The second-layer image feature refinement module is used to obtain the second-layer image features. The second-layer image features are first fused and channel attention extracted, then channel attention is extracted again and finally difference enhancement is performed to obtain the refined second-layer image features. The module for refining the third and fourth layer image features is used to obtain the third and fourth layer image features. Cross-channel attention and cross-spatial attention are performed on the third and fourth layer image features respectively to obtain the refined third layer image features and the refined fourth layer image features respectively. Image reconstruction module: This module performs cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. The reconstructed features are then used to reconstruct the image to obtain a complementary and fused image.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the infrared image fusion method based on multi-scale feature extraction as described in any one of claims 1 to 5.

Images