Infrared image fusion method, system and equipment based on multi-scale feature extraction

Abstract

The invention relates to the field of image processing, and provides an infrared image fusion method, system and device based on multi-scale feature extraction, and the method comprises the steps: carrying out the extraction through employing a multi-scale pyramid, and obtaining layer image features; first-layer image features are obtained from the layer image features, cross-modal interaction cooperation and cross injection are firstly carried out on the first-layer image features, then difference feature calculation and cross injection are carried out, and refined first-layer image features are obtained; performing feature fusion and channel attention extraction on the second-layer image features, and then performing channel attention re-extraction and final difference enhancement to obtain refined second-layer image features; sequentially performing cross-channel attention and cross-space attention on the third-layer image features and the fourth-layer image features respectively to obtain refined third-layer image features and refined fourth-layer image features respectively; and performing cross-layer up-sampling and residual feature distillation to obtain reconstructed features, and reconstructing the reconstructed features to obtain an information complementary fusion image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an infrared image fusion method, system, and device based on multi-scale feature extraction. Background Technology

[0002] Due to hardware limitations in imaging equipment, sensors of a single type or with a single setup typically cannot fully characterize an imaging scene. For example, visible light images usually contain rich textural details but are susceptible to extreme environments and occlusion, causing them to lose targets within the scene. Conversely, infrared sensors, by capturing thermal radiation information emitted by objects, can effectively highlight prominent targets such as pedestrians and vehicles, but lack detailed descriptions of the scene. Sensors of different types or with different optical setups often contain a wealth of complementary information; therefore, image fusion techniques have emerged.

[0003] Before deep learning swept through the field of computer vision, image fusion had already been extensively studied. Traditional image fusion algorithms typically involve performing activity level measurements in the spatial domain or transform domain (transforming the image to the transform domain through some mathematical transformation) and manually designing fusion rules to achieve image fusion. In recent years, deep learning has dominated the development of computer vision with its powerful feature extraction and representation capabilities. To overcome the shortcomings of traditional algorithms, researchers in the field of image fusion have explored a large number of deep learning-based image fusion algorithms, which has become a hot topic in current image fusion research. However, existing techniques generally suffer from problems such as not considering the differences in features at various scales and the lack of noise suppression processing during the upsampling process, leading to chessboard artifacts. Summary of the Invention

[0004] This invention aims to at least solve one of the technical problems existing in related technologies. To this end, this invention provides an infrared image fusion method, system, and device based on multi-scale feature extraction, which realizes the fusion of infrared images and visible light images.

[0005] This invention provides an infrared image fusion method based on multi-scale feature extraction, comprising: S1: Acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images; S2: Obtain the first layer image features from the layer image features. First, perform cross-modal interactive collaboration and cross-injection on the first layer image features, and then perform differential feature calculation and cross-injection to obtain the refined first layer image features. S3: Obtain the second layer of image features. First, perform feature fusion and channel attention extraction on the second layer of image features. Then, perform channel attention re-extraction and final difference enhancement to obtain refined second layer of image features. S4: Obtain the third-layer image features and the fourth-layer image features. Perform cross-channel attention and cross-space attention on the third-layer image features and the fourth-layer image features respectively to obtain the refined third-layer image features and the refined fourth-layer image features respectively. S5: Perform cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. Reconstruct the reconstructed features to obtain a complementary and fused image.

[0006] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, step S1 further includes: S11: Acquire infrared and visible light images respectively, establish a shallow feature extraction module, a medium feature extraction module and a deep feature extraction module, and establish the multi-scale pyramid through the shallow feature extraction module, the medium feature extraction module and the deep feature extraction module; S12: Input the infrared image and the visible light image into the multi-scale pyramid and extract the infrared image and the visible light image layer by layer to obtain the layer image features.

[0007] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S2, after obtaining the first layer of image features including the first layer of infrared features and the first layer of visible light features, the first layer of infrared features and the first layer of visible light features are subjected to cross-modal interactive collaboration through cascaded convolution operation to obtain a channel attention map. The channel attention map is then cross-injected with the first layer of image features to obtain the intermediate first layer of image features.

[0008] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S2, the intermediate first layer image features are spliced ​​and the difference features are calculated to obtain a difference attention map. The difference attention map and the intermediate first layer image features are cross-injected to obtain the refined first layer image features.

[0009] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S3, after obtaining the second layer image features including the second layer infrared features and the second layer visible light features, the second layer infrared features and the second layer visible light features are fused to obtain a feature fusion map. Channel attention extraction is performed on the feature fusion map to obtain a channel feature map. Channel modulation is performed based on the channel feature map to obtain the intermediate second layer image features.

[0010] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S3, channel attention re-extraction is performed on the intermediate second layer image features to obtain a re-extracted channel feature map, and final difference enhancement is performed based on the re-extracted channel feature map to obtain the refined second layer image features.

[0011] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, step S4 further includes: S41: Obtain the third-layer image features and the fourth-layer image features, perform pyramid pooling and convolution projection on the third-layer image features and the fourth-layer image features respectively, and obtain the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively, and obtain the intermediate third-layer image features and the intermediate fourth-layer image features based on the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively. S42: Perform pyramid pooling and convolution projection on the intermediate third-layer image features and the intermediate fourth-layer image features respectively to obtain third-layer image key-value pairs and fourth-layer image key-value pairs. Based on the third-layer image key-value pairs and the fourth-layer image key-value pairs, obtain refined third-layer image features and refined fourth-layer image features respectively.

[0012] According to the infrared image fusion method based on multi-scale feature extraction provided by the present invention, in step S5, deformable convolution is performed on the refined fourth-layer image features and the refined third-layer image features to obtain reconstructed fourth-layer image features, and residual feature distillation is performed on the reconstructed fourth-layer image features and the refined third-layer image features to obtain intermediate reconstructed third-layer image features. Deformable convolution is performed on the intermediate reconstructed third-layer image features and the refined second-layer image features to obtain the reconstructed third-layer image features. Residual feature distillation is then performed on the reconstructed third-layer image features and the refined second-layer image features to obtain the intermediate reconstructed second-layer image features. Deformable convolution is performed on the intermediate reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed second-layer image features. Residual feature distillation is then performed on the reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed features.

[0013] This invention also provides an infrared image fusion system based on multi-scale feature extraction, comprising: Layer image feature module: used to acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images; The first-layer image feature refinement module is used to obtain the first-layer image features from the layer image features. The first-layer image features are first subjected to cross-modal interactive collaboration and cross-injection, and then differential feature calculation and cross-injection are performed to obtain the refined first-layer image features. The second-layer image feature refinement module is used to obtain the second-layer image features. The second-layer image features are first fused and channel attention extracted, then channel attention is extracted again and finally difference enhancement is performed to obtain the refined second-layer image features. The module for refining the third and fourth layer image features is used to obtain the third and fourth layer image features. Cross-channel attention and cross-spatial attention are performed on the third and fourth layer image features respectively to obtain the refined third layer image features and the refined fourth layer image features respectively. Image reconstruction module: It is used to perform cross-layer upsampling and residual feature distillation on the refined fourth layer image features, refined third layer image features, refined second layer image features and refined first layer image features in sequence to obtain reconstructed features. The reconstructed features are then reconstructed to obtain a complementary and fused image.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the infrared image fusion method based on multi-scale feature extraction as described above.

[0015] The above-described one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: The infrared image fusion method, system, and device based on multi-scale feature extraction provided by this invention designs different feature extraction modules for image features at different scales, thereby fully considering the features of each scale in both spatial and channel dimensions, greatly improving the overall performance of the fusion network framework, effectively suppressing noise in the image, making full use of the information in the image, and preserving the details in the image.

[0016] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the infrared image fusion method based on multi-scale feature extraction provided by the present invention.

[0019] Figure 2This is a comparison image of the fusion results of the infrared image fusion method based on multi-scale feature extraction provided by this invention.

[0020] Figure 3 This is a schematic diagram of the infrared image fusion system based on multi-scale feature extraction provided by the present invention.

[0021] Figure 4 This is a schematic diagram of the structure of the infrared image fusion device based on multi-scale feature extraction provided by the present invention.

[0022] Figure label: 100. Layer image feature module; 200. First layer image feature refinement module; 300. Second layer image feature refinement module; 400. Third and fourth layer image feature refinement modules; 500. Image reconstruction module; 810. Processor; 820. Communication interface; 830. Memory; 840. Communication bus. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The following embodiments are used to illustrate this invention but cannot be used to limit the scope of this invention.

[0024] In the description of the embodiments of the present invention, it should be noted that the terms "first", "second" and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0025] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of the present invention based on the specific circumstances.

[0026] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0027] The following is combined with Figures 1 to 4 Specific embodiments of the present invention are described below. Figure 1 This is a flowchart illustrating the infrared image fusion method based on multi-scale feature extraction provided by the present invention. It includes: S1: Acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images; Furthermore, the objective of this stage is to extract layer image features from infrared and visible light images using a multi-scale pyramid. Specifically, step S1 further includes: S11: Acquire infrared and visible light images respectively, establish a shallow feature extraction module, a medium feature extraction module and a deep feature extraction module, and establish the multi-scale pyramid through the shallow feature extraction module, the medium feature extraction module and the deep feature extraction module; S12: Input the infrared image and the visible light image into the multi-scale pyramid and extract the infrared image and the visible light image layer by layer to obtain the layer image features.

[0028] The specific implementation method for the above steps in this embodiment is as follows: First, it is necessary to acquire infrared images of the same target separately. and visible light images Next, shallow feature extraction, mid-level feature extraction, and deep feature extraction modules need to be established and stacked layer by layer to obtain a multi-scale pyramid. Finally, the infrared and visible light images are input into the multi-scale pyramid, which can then extract features layer by layer from the infrared and visible light images respectively. in, The first layer of infrared features, The first layer of visible light characteristics, This is the second layer of infrared features. This represents the second layer of visible light characteristics. The third layer of infrared features, This represents the third layer of visible light characteristics. This is the fourth layer of infrared features. This represents the fourth layer of visible light characteristics. This is a shallow infrared feature extraction module. This is a shallow visible light feature extraction module. This is a mid-layer infrared feature extraction module. This is a mid-layer visible light feature extraction module. This is the first deep infrared feature extraction module. This is the first deep visible light feature extraction module. This is the second deep infrared feature extraction module. This is the second deep visible light feature extraction module.

[0029] Each layer of the multi-scale pyramid is configured with different parameters, allowing image features to be extracted layer by layer from shallow to deep. This enables the low-level structural information of infrared and visible light images to establish a cross-scale feature synergy with the high-level semantic representation, thereby enhancing the ability of the fused image to retain detailed information and perceive salient targets.

[0030] S2: Obtain the first layer image features from the layer image features. First, perform cross-modal interactive collaboration and cross-injection on the first layer image features, and then perform differential feature calculation and cross-injection to obtain the refined first layer image features. Furthermore, the objective of this stage is to refine the first-layer image features to obtain refined first-layer image features. Specifically, in step S2, after obtaining the first-layer image features including first-layer infrared features and first-layer visible light features, the first-layer infrared features and first-layer visible light features are subjected to cross-modal interactive collaboration through cascaded convolution operations to obtain a channel attention map. The channel attention map is then cross-injected with the first-layer image features to obtain intermediate first-layer image features.

[0031] In step S2, the intermediate first layer image features are spliced ​​and the difference features are calculated to obtain a difference attention map. The difference attention map and the intermediate first layer image features are cross-injected to obtain the refined first layer image features.

[0032] The specific implementation method for the above steps in this embodiment is as follows: First, after obtaining the first layer of image features, which includes the first layer of infrared features and the first layer of visible light features, it is necessary to perform cross-modal interactive collaboration on the first layer of infrared features and the first layer of visible light features through cascaded convolution operations to obtain a channel attention map α that fuses multi-source information: The `Concat()` function concatenates the contents within the parentheses. This represents a 3-level convolution operation. Note the diagram indicating the middle passage. As the activation function, GAP() indicates that the content within the parentheses is subjected to global average pooling.

[0033] Then, cross-injection is performed to obtain the first layer of image features in the middle, in order to further promote cross-modal interaction and collaborative representation: in, This represents the first visible light image feature in the middle layer. The first layer of infrared image features is the middle layer. This is an element-wise multiplication operation.

[0034] Subsequently, the image features of the first intermediate layer are stitched together and difference features are calculated to obtain the difference attention map. This allows for the extraction of differential features and explicit modeling of differences between modalities, capturing highly complementary differential information. in, Note the first difference in the diagram. Note the second difference in the diagram. This is an element-by-element subtraction operation. This is a two-stage cascaded convolution operation. Cross-injection is then performed to supplement missing semantic information related to modality differences, resulting in refined first-layer image features: in, This is an element-wise addition operation. To refine the features of the first layer of infrared image, To refine the features of the first layer of visible light image.

[0035] S3: Obtain the second layer of image features. First, perform feature fusion and channel attention extraction on the second layer of image features. Then, perform channel attention re-extraction and final difference enhancement to obtain refined second layer of image features. Furthermore, the objective of this stage is to obtain refined second-layer image features. Specifically, in step S3, after obtaining the second-layer image features including second-layer infrared features and second-layer visible light features, feature fusion is performed on the second-layer infrared features and the second-layer visible light features to obtain a feature fusion map. Channel attention extraction is then performed on the feature fusion map to obtain a channel feature map. Based on the channel feature map, channel modulation is performed to obtain the intermediate second-layer image features.

[0036] In step S3, channel attention is re-extracted from the intermediate second layer image features to obtain a re-extracted channel feature map. Based on the re-extracted channel feature map, final difference enhancement is performed to obtain the refined second layer image features.

[0037] The specific implementation method for the above steps in this embodiment is as follows: First, to enhance channel selectivity, a second layer of image features, including a second layer of infrared features and a second layer of visible light features, is obtained. Then, the second layer of infrared features and the second layer of visible light features are fused to obtain a feature fusion map. This allows for the simultaneous characterization of the overall activation intensity and activation fluctuations of each channel. Channel attention extraction is then performed to obtain channel feature map A. in, For the first channel, pay attention to extracting the weight coefficients. For the second channel, weighting coefficients are extracted, and SD represents the standard deviation within parentheses. Subsequently, to preserve basic structural information, channel modulation is performed to obtain the intermediate second-layer image features. : Next, to further construct explicit intermodal differences, a parallel channel attention mechanism is introduced in the channel dimension to re-extract the image features of the intermediate second layer, resulting in a re-extracted channel feature map. : in, For the first channel, pay attention to extracting coefficients again. To further extract coefficients for the second channel, and finally to effectively suppress redundant information accumulation and improve the discriminative power of the fusion result, highlighting areas of significant difference, a final difference enhancement is performed to obtain refined second-layer image features: in, To refine the features of the second layer of infrared image, To refine the features of the second layer of visible light image.

[0038] S4: Obtain the third-layer image features and the fourth-layer image features. Perform cross-channel attention and cross-space attention on the third-layer image features and the fourth-layer image features respectively to obtain the refined third-layer image features and the refined fourth-layer image features respectively. Furthermore, the objective of this stage is to sequentially perform cross-channel attention and cross-spatial attention on the third-layer and fourth-layer image features, respectively, thereby obtaining refined third-layer and refined fourth-layer image features. Specifically, step S4 further includes: S41: Obtain the third-layer image features and the fourth-layer image features, perform pyramid pooling and convolution projection on the third-layer image features and the fourth-layer image features respectively, and obtain the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively, and obtain the intermediate third-layer image features and the intermediate fourth-layer image features based on the third-layer intermediate image key-value pairs and the fourth-layer intermediate image key-value pairs respectively. S42: Perform pyramid pooling and convolution projection on the intermediate third-layer image features and the intermediate fourth-layer image features respectively to obtain third-layer image key-value pairs and fourth-layer image key-value pairs. Based on the third-layer image key-value pairs and the fourth-layer image key-value pairs, obtain refined third-layer image features and refined fourth-layer image features respectively.

[0039] The specific implementation method for the above steps in this embodiment is as follows: Since deep features typically contain more abstract and semantic representations and tend to encode shared semantic content between the two modalities, employing an explicit cross-attention mechanism for modeling in this step is more conducive to modality fusion. Specifically, cross-channel attention helps capture global semantic redundancy and complementary relationships between modalities, while cross-spatial attention focuses more on local structural differences, thereby enhancing structural consistency. Specifically, pyramid pooling and convolutional projection are performed on the third and fourth layer image features, respectively, to obtain the intermediate image key-value pairs for the third and fourth layers, respectively.

[0040] Specifically, pyramid pooling and convolutional projection are first applied sequentially to the third and fourth layer image features, respectively. During pyramid pooling, spatial pyramid pooling is used for the third layer features, while pyramid pooling is used for the fourth layer features. By adjusting the parameters during pyramid pooling and convolutional projection, the intermediate infrared bonds of the i-th layer can be obtained. Visible optical bonds in the middle of the i-th layer Infrared query of the i-th layer Querying visible light in the middle of the i-th layer Infrared values ​​of the i-th layer and the visible light value in the middle of the i-th layer In step S4, the value of i is either 3 or 4. When i=3, the intermediate key, intermediate query, and intermediate value of the i-th layer are obtained by sequentially performing pyramid pooling and convolutional projection on the image features of the third layer, and the resulting intermediate key, intermediate query, and intermediate value constitute the key-value pair of the third layer intermediate image. When i=4, the intermediate key, intermediate query, and intermediate value of the i-th layer are obtained by sequentially performing pyramid pooling and convolutional projection on the image features of the fourth layer, and the key, query, and value at this time constitute the key-value pair of the fourth layer intermediate image. The parameter settings of the convolutional projection function and pyramid pooling are different when obtaining the key, query, and value.

[0041] Then, based on the key-value pairs of the third and fourth intermediate image layers, the features of the third and fourth intermediate image layers are obtained respectively: Where i=3, This refers to the visible light image features of the third intermediate layer. For the third layer of infrared image features, the key-value pairs of the third layer image and the third layer image features are used. When i=4, This represents the visible light image features of the fourth layer in the middle. The fourth layer of infrared image features are obtained by using the key-value pairs of the fourth layer image and the features of the fourth layer image, thus obtaining the third layer image features and the fourth layer image features.

[0042] Then, using the method described above, pyramid pooling and convolutional projection are performed sequentially on the image features of the third and fourth intermediate layers, respectively, to obtain the infrared bonds of the i-th layer. The i-th layer of visible light bonds Infrared query of the i-th layer Visible light query of layer i Infrared values ​​of the i-th layer and the visible light value of the i-th layer Here, the value of i is either 3 or 4. When i=3, the key, query, and value of the i-th layer are obtained by sequentially performing pyramid pooling and convolutional projection on the features of the intermediate third layer image, and the resulting key, query, and value constitute the key-value pairs of the third layer image. When i=4, the key, query, and value of the i-th layer are obtained by sequentially performing pyramid pooling and convolutional projection on the features of the intermediate fourth layer image, and the resulting key, query, and value constitute the key-value pairs of the fourth layer image. The parameter settings of the convolutional projection function and pyramid pooling are different when obtaining the key, query, and value.

[0043] Finally, we obtain the refined third-layer image features and the refined fourth-layer image features: Where i=3, To refine the features of the third layer of visible light image, To refine the features of the third layer of infrared image, the third layer image key-value pairs and intermediate third layer image features are used. When i=4, To refine the features of the fourth layer of visible light image, To refine the features of the fourth layer of infrared image, the key-value pairs of the fourth layer image and the intermediate features of the fourth layer image are used, thus obtaining the refined features of the third layer image and the refined features of the fourth layer image.

[0044] S5: Perform cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. Reconstruct the reconstructed features to obtain a complementary and fused image.

[0045] Furthermore, the objective of this stage is to perform cross-layer upsampling and residual feature distillation to obtain reconstructed features, and then reconstruct the reconstructed features to obtain an information-complementary fused image. Specifically, in step S5, deformable convolution is performed on the refined fourth-layer image features and the refined third-layer image features to obtain reconstructed fourth-layer image features. Residual feature distillation is then performed on the reconstructed fourth-layer image features and the refined third-layer image features to obtain intermediate reconstructed third-layer image features. Deformable convolution is performed on the intermediate reconstructed third-layer image features and the refined second-layer image features to obtain the reconstructed third-layer image features. Residual feature distillation is then performed on the reconstructed third-layer image features and the refined second-layer image features to obtain the intermediate reconstructed second-layer image features. Deformable convolution is performed on the intermediate reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed second-layer image features. Residual feature distillation is then performed on the reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed features.

[0046] The specific implementation method for the above steps in this embodiment is as follows: Here, deformable convolution is used to achieve cross-layer upsampling. First, deformable convolution is performed on the thinned fourth layer image features and the thinned third layer image features to obtain the reconstructed fourth layer image features: in, To reconstruct the features of the fourth layer of infrared image, To reconstruct the features of the fourth layer of visible light image, To perform infrared fourth-layer deformable convolution on the content within parentheses, To perform a visible light fourth-layer deformable convolution on the content within the brackets.

[0047] By reconstructing the fourth layer of image features and refining the third layer of image features, residual feature distillation is performed to obtain the intermediate reconstructed third layer of image features: in, To perform infrared third-layer residual feature distillation on the content within parentheses, To perform visible light third-layer residual characteristic distillation on the content within parentheses, To reconstruct the third layer of image features in the infrared intermediate layer, The third layer of image features is reconstructed in the middle of the visible light image.

[0048] Deformable convolution is performed on the reconstructed third-layer image features and the refined second-layer image features to obtain the reconstructed third-layer image features: in, To perform infrared third-layer deformable convolution on the content within parentheses, To perform a visible light third-layer deformable convolution on the content within the brackets, To reconstruct the third layer of image features using infrared technology, Reconstruct the third layer of image features using visible light.

[0049] By reconstructing the third-layer image features and refining the second-layer image features, residual feature distillation is performed to obtain the intermediate reconstructed second-layer image features: in, To perform infrared second-layer residual feature distillation on the content within parentheses, To perform visible light second-layer residual feature distillation on the content within parentheses, To reconstruct the second layer of image features from the infrared intermediate layer, Reconstruct the second layer of image features in the visible light intermediate layer.

[0050] Deformable convolution is performed on the reconstructed second-layer image features and the refined first-layer image features to obtain the reconstructed second-layer image features: in, To perform infrared second-layer deformable convolution on the content within parentheses, To perform a second-layer deformable convolution on the content within the brackets using visible light, To reconstruct the second layer of image features using infrared technology, The second layer of image features is reconstructed using visible light.

[0051] Finally, residual feature distillation is performed by reconstructing the second layer of image features and refining the first layer of image features to obtain the reconstructed features: in, To perform infrared first-layer residual feature distillation on the content within parentheses, To perform visible light first-layer residual feature distillation on the content within parentheses, For infrared reconstruction features, This is for visible light reconstruction features. The parameters of residual feature distillation and deformable convolution for each of the above layers, as well as the parameters of residual feature distillation and deformable convolution for infrared and visible light, all need to be set according to actual needs. Cross-layer upsampling is achieved through deformable convolution.

[0052] Finally, the reconstructed features are rebuilt to obtain a complementary and fused image. : Here, Sigmoid() inputs the content within the parentheses into the Sigmoid function, and Conv() performs convolution on the content within the parentheses.

[0053] The effectiveness of the infrared image fusion method based on multi-scale feature extraction provided by this invention is described below. First, images from the publicly available TNO dataset are used for testing. DeFusion, ITFuse, YDTR, TUFusion, LRRNet, RFN-Nest, and SpTFuse are all existing infrared image fusion methods. Table 1 compares the fusion results of this method with other methods in the TNO dataset. Table 1. Comparison of evaluation results of our method with other fusion methods in the TNO dataset:

[0054] Wherein, MI stands for Mutual Information, which measures the amount of information shared between images; a larger value indicates more complete information sharing. SF stands for Spatial Frequency, which measures the overall drastic changes in an image; a larger value indicates a higher degree of detail enhancement. AG stands for Average Gradient, which reflects image sharpness; a larger value indicates a sharper image. VIF stands for Visual Information Fidelity, and Q... abfThe objective non-reference quality evaluation index for image fusion is denoted by , where a higher value indicates richer preservation of important edge information in the image. SSIM (Structural Similarity Index) represents the higher similarity between the fused and unfused images; bolded numbers indicate that this value is the optimal value among all methods. On the TNO dataset, this method demonstrates significantly better infrared target contrast compared to other methods, while also preserving visible light texture details well, exhibiting low sky noise and high clarity, achieving the best values ​​across all metrics.

[0055] Subsequently, tests were also conducted on images on the publicly available M3FD dataset. Table 2 shows a comparison of the fusion results of this method with other methods on the MFID dataset: Table 2. Comparison of the fusion results of our method with other methods on the MFID dataset.

[0056] The bolded numbers indicate that this value is the best among all methods. On the M3FD dataset, this method preserves the texture details of trees and mountains more clearly and the target information of people more prominently compared with other methods, achieving the best value in all metrics.

[0057] Finally, images from the publicly available RoadScene dataset were used for testing, and the fusion results are as follows: Figure 2 As shown in Table 3, this table compares the evaluation results of the fusion of our method with other methods in the RoadScene dataset: Table 3. Comparison of fusion results between this method and other methods in the RoadScene dataset.

[0058] The bolded numbers indicate that this value is the best among all methods. As can be seen from the comparison results on the RoadScene dataset, this method has moderate contrast, significant infrared target information, and preserves ground texture details while minimizing sky noise, achieving the best value in all metrics.

[0059] Figure 3 A schematic diagram of an infrared image fusion system based on multi-scale feature extraction is shown, as follows: Figure 3 As shown, the infrared image fusion method based on multi-scale feature extraction described above includes: Layer image feature module 100: used to acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images. The first-layer image feature refinement module 200 is used to obtain the first-layer image features from the layer image features. The first-layer image features are first subjected to cross-modal interactive collaboration and cross-injection, and then differential feature calculation and cross-injection are performed to obtain the refined first-layer image features. The second-layer image feature refinement module 300 is used to obtain the second-layer image features. The second-layer image features are first subjected to feature fusion and channel attention extraction, then channel attention is further extracted and finally difference enhancement is performed to obtain the refined second-layer image features. The module 400 for refining the third and fourth layer image features is used to obtain the third layer image features and the fourth layer image features. Cross-channel attention and cross-space attention are performed on the third layer image features and the fourth layer image features respectively to obtain the refined third layer image features and the refined fourth layer image features respectively. Image reconstruction module 500: It is used to perform cross-layer upsampling and residual feature distillation on the refined fourth layer image features, refined third layer image features, refined second layer image features and refined first layer image features in sequence to obtain reconstructed features, and reconstruct the reconstructed features to obtain a complementary and fused image.

[0060] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call a computer program in the memory 830 to execute an infrared image fusion method based on multi-scale feature extraction, the method including: S1: Acquire infrared and visible light images, build a multi-scale pyramid, and use the multi-scale pyramid to extract layer image features from the infrared and visible light images; S2: Obtain the first layer image features from the layer image features. First, perform cross-modal interactive collaboration and cross-injection on the first layer image features, and then perform differential feature calculation and cross-injection to obtain the refined first layer image features. S3: Obtain the second layer of image features. First, perform feature fusion and channel attention extraction on the second layer of image features. Then, perform channel attention re-extraction and final difference enhancement to obtain refined second layer of image features. S4: Obtain the third-layer image features and the fourth-layer image features. Perform cross-channel attention and cross-space attention on the third-layer image features and the fourth-layer image features respectively to obtain the refined third-layer image features and the refined fourth-layer image features respectively. S5: Perform cross-layer upsampling and residual feature distillation on the refined fourth-layer image features, refined third-layer image features, refined second-layer image features, and refined first-layer image features in sequence to obtain reconstructed features. Reconstruct the reconstructed features to obtain a complementary and fused image.

[0061] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0062] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An infrared image fusion method based on multi-scale feature extraction, characterized in that, Comprising: S1: acquiring an infrared image and a visible light image, establishing a multi-scale pyramid, using the multi-scale pyramid to extract the infrared image and the visible light image to obtain layer image features; S2: obtaining first layer image features from the layer image features, first performing cross-modal interactive collaboration and cross injection on the first layer image features, and then performing difference feature calculation and cross injection to obtain refined first layer image features; S3: obtaining second layer image features, first performing feature fusion and channel attention extraction on the second layer image features, and then performing channel attention re-extraction and final difference enhancement to obtain refined second layer image features; S4: obtaining third layer image features and fourth layer image features, respectively performing cross-channel attention and cross-space attention on the third layer image features and the fourth layer image features in turn to obtain refined third layer image features and refined fourth layer image features; S5: performing cross-layer upsampling and residual feature distillation on the refined fourth layer image features, the refined third layer image features, the refined second layer image features and the refined first layer image features in turn to obtain reconstruction features, and performing reconstruction on the reconstruction features to obtain information complementary fusion images.

2. The method for infrared image fusion based on multi-scale feature extraction according to claim 1, characterized in that, Step S1 further comprises: S11: respectively acquiring an infrared image and a visible light image, establishing a shallow feature extraction module, a middle feature extraction module and a deep feature extraction module, and establishing the multi-scale pyramid through the shallow feature extraction module, the middle feature extraction module and the deep feature extraction module; S12: inputting the infrared image and the visible light image into the multi-scale pyramid and performing layer-by-layer extraction on the infrared image and the visible light image to obtain the layer image features. 3.The method of claim 1, wherein, In step S2, after obtaining the first layer image features including the first layer infrared features and the first layer visible light features, cross-modal interactive collaboration is performed on the first layer infrared features and the first layer visible light features through a cascaded convolution operation to obtain a channel attention map, and cross injection is performed on the channel attention map and the first layer image features to obtain an intermediate first layer image feature.

4. The method of claim 3, wherein the method further comprises: In step S2, the intermediate first layer image feature is spliced and difference feature calculation is performed to obtain a difference attention map, and cross injection is performed on the difference attention map and the intermediate first layer image feature to obtain the refined first layer image feature.

5. The method for infrared image fusion based on multiscale feature extraction according to claim 1, characterized in that, In step S3, after obtaining the second layer image features including the second layer infrared features and the second layer visible light features, feature fusion is performed on the second layer infrared features and the second layer visible light features to obtain a feature fusion map, channel attention extraction is performed on the feature fusion map to obtain a channel feature map, and channel modulation is performed based on the channel feature map to obtain an intermediate second layer image feature.

6. The method of claim 5, wherein the method further comprises: In step S3, channel attention re-extraction is performed on the intermediate second layer image feature to obtain a re-extracted channel feature map, and final difference enhancement is performed based on the re-extracted channel feature map to obtain the refined second layer image feature.

7. The method for infrared image fusion based on multiscale feature extraction according to claim 1, characterized in that, Step S4 further comprises: S41: Obtain the third layer image feature and the fourth layer image feature, pyramid pool and convolution projection are performed on the third layer image feature and the fourth layer image feature respectively, and the third layer intermediate image key-value pair and the fourth layer intermediate image key-value pair are obtained respectively, and the intermediate third layer image feature and the intermediate fourth layer image feature are obtained based on the third layer intermediate image key-value pair and the fourth layer intermediate image key-value pair respectively; S42: Pyramid pool and convolution projection are performed on the intermediate third layer image feature and the intermediate fourth layer image feature respectively, and the third layer image key-value pair and the fourth layer image key-value pair are obtained, and the refined third layer image feature and the refined fourth layer image feature are obtained based on the third layer image key-value pair and the fourth layer image key-value pair respectively.

8. The method for infrared image fusion based on multiscale feature extraction according to claim 1, characterized in that, In step S5, deformable convolution is performed on the refined fourth layer image feature and the refined third layer image feature to obtain a reconstructed fourth layer image feature, and residual feature distillation is performed on the reconstructed fourth layer image feature and the refined third layer image feature to obtain an intermediate reconstructed third layer image feature; Deformable convolution is performed on the intermediate reconstructed third layer image feature and the refined second layer image feature to obtain a reconstructed third layer image feature, and residual feature distillation is performed on the reconstructed third layer image feature and the refined second layer image feature to obtain an intermediate reconstructed second layer image feature; Deformable convolution is performed on the intermediate reconstructed second layer image feature and the refined first layer image feature to obtain a reconstructed second layer image feature, and residual feature distillation is performed on the reconstructed second layer image feature and the refined first layer image feature to obtain the reconstructed feature.

9. An infrared image fusion system based on multiscale feature extraction for performing the method of infrared image fusion based on multiscale feature extraction according to any one of claims 1 to 8, characterized in that, Comprise: Layer image feature module: used for acquiring infrared image and visible light image, establishing multi-scale pyramid, using multi-scale pyramid to extract infrared image and visible light image to obtain layer image feature; Refined first layer image feature module: used for obtaining first layer image feature from layer image feature, first cross-modal interaction and cross injection are performed on first layer image feature, and then difference feature calculation and cross injection are performed to obtain refined first layer image feature; Refined second layer image feature module: used for obtaining second layer image feature, first feature fusion and channel attention extraction are performed on second layer image feature, and then channel attention re-extraction and final difference enhancement are performed to obtain refined second layer image feature; Refined third and fourth layer image feature module: used for obtaining third layer image feature and fourth layer image feature, cross-channel attention and cross-space attention are sequentially performed on third layer image feature and fourth layer image feature respectively to obtain refined third layer image feature and refined fourth layer image feature respectively; Image reconstruction module: used for sequentially performing cross-layer upsampling and residual feature distillation on refined fourth layer image feature, refined third layer image feature, refined second layer image feature and refined first layer image feature to obtain reconstructed feature, and reconstructing the reconstructed feature to obtain information complementary fusion image.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the infrared image fusion method based on multi-scale feature extraction as claimed in any one of claims 1 to 8. The processor executes the computer program to realize the steps of the infrared image fusion method based on multi-scale feature extraction as claimed in any one of claims 1 to 8.

Images