An Infrared-Visible Image Fusion Method Based on 3D Convolutional Neural Networks

Abstract

This invention relates to an infrared and visible light image fusion method based on a 3D convolutional neural network, belonging to the field of image enhancement technology. The method includes: acquiring a fully trained image fusion model, which includes an encoder, a feature fusion module, and a decoder; acquiring infrared and visible light images to be fused; extracting high-dimensional semantic features from the infrared and visible light images based on the encoder to obtain multiple visible light feature maps and multiple infrared feature maps; cross-matching and stitching the multiple visible light and infrared feature maps based on the feature fusion module, and weighting the stitched features to obtain a processed feature map; and performing weighted fusion of the processed feature map based on the decoder to obtain the image fusion result. This invention solves the technical problems in existing image fusion technologies, such as brightness neutralization or oversaturation, and information loss, which affect fusion performance.

Description

Technical Field

[0001] This invention relates to the field of image enhancement technology, and in particular to an infrared and visible light image fusion method based on a 3D convolutional neural network. Background Technology

[0002] In recent years, thanks to the powerful feature extraction and image reconstruction capabilities of neural networks, deep learning-based image fusion methods have seen widespread development. Existing deep learning-based image fusion methods can achieve end-to-end image fusion. Specifically, end-to-end infrared and visible light image fusion methods can achieve infrared and visible light image fusion using a single holistic network architecture. This holistic network architecture mainly involves various stages of the fusion algorithm, including feature extraction, feature fusion, and image reconstruction.

[0003] Due to the limitations of simple networks in extracting deep features, novel learning methods such as deep reinforcement learning, Transformer, and Diffusion have been introduced into image fusion tasks to enhance feature extraction performance. On the other hand, deep learning-based methods have introduced attention modules and novel loss functions to enhance the ability to extract and preserve detailed information. For example, by introducing relevant differences and a loss function, GANFuse constructs an adversarial game between the fused image and the source image from the perspective of information integration. MAFusion introduces a multi-scale attention network to highlight thermal targets in infrared images and texture details in visible images. SSAU-Net is an image fusion method based on a spectral-channel attention mechanism, which can enhance the spectral and channel information of images. In summary, deep learning-based fusion methods employ complex network architectures and ingenious loss functions to achieve promising results.

[0004] However, due to the non-interpretability of deep features, feature fusion strategies struggle to assign appropriate weights to these features. Most deep learning-based methods employ manually designed fusion strategies, such as stitching, pixel summation, weighted pixel summation, pixel averaging, and maximum pixel selection. However, these strategies cannot adaptively adjust to the input image, potentially leading to issues like brightness neutralization or oversaturation, and information loss, thus limiting fusion performance. Summary of the Invention

[0005] In view of this, it is necessary to provide an infrared and visible light image fusion method based on 3D convolutional neural networks to solve the technical problem that existing technologies cannot adaptively adjust the fusion strategy according to the input image during image fusion, resulting in brightness neutralization or oversaturation, information loss, etc., which affect the fusion performance.

[0006] To address the aforementioned problems, this invention provides an infrared-visible light image fusion method based on a 3D convolutional neural network, comprising:

[0007] Obtain a fully trained image fusion model, which includes an encoder, a feature fusion module, and a decoder;

[0008] Acquire the infrared and visible light images to be fused;

[0009] Based on the encoder, high-dimensional semantic features are extracted from the infrared and visible light images to be fused to obtain multiple visible light feature maps and multiple infrared feature maps; based on the feature fusion module, the multiple visible light feature maps and multiple infrared feature maps are cross-matched and stitched together, and the stitched features are weighted to obtain a processed feature map; based on the decoder, the processed feature map is weighted and fused to obtain the image fusion result.

[0010] In one possible implementation, the encoder includes a convolution module and multiple densely connected modules;

[0011] The convolution module is used to perform convolution operations on the input infrared image and visible light image to be fused, to obtain multiple infrared local features and multiple visible light local features;

[0012] The dense connection module includes a convolutional layer, a batch normalization layer, an activation function layer, a merging operation layer, and a feature skip connection layer. Each current dense connection module receives multiple infrared local features, multiple visible light local features, and the outputs of all previous dense connection modules as inputs based on the feature skip connection layer. The merging operation layer then merges all inputs and inputs them into the convolutional layer, the batch normalization layer, and the activation function layer to obtain multiple visible light feature maps and multiple infrared feature maps.

[0013] In one possible implementation, the feature fusion module includes a cross-matching splicing operation layer and a channel attention mechanism;

[0014] The cross-matching splicing operation layer is used to cross-splice the visible light feature map and the infrared feature map obtained by the same convolutional kernel according to a preset rule to obtain a cross-matching feature map.

[0015] The channel attention mechanism is used to perform global average pooling on the cross-matching feature map, and then weight the pooled feature map to obtain the processed feature map.

[0016] In one possible implementation, the step of cross-stitching the visible light feature map and the infrared feature map obtained from the same convolutional kernel according to a preset rule includes:

[0017] The channels of the visible light feature map and the infrared feature map are arranged in a cross-order.

[0018] In one possible implementation, the channel attention mechanism includes a global pooling layer, multiple convolutional layers, a feature addition layer, an activation layer, and a feature multiplication layer;

[0019] The global pooling layer is used to perform global average pooling on the cross-matching feature map to obtain the global average pooling result.

[0020] The multiple convolutional layers are used to extract multi-scale features from the global average pooling result, resulting in multiple convolutional feature maps.

[0021] The feature addition layer is used to add multiple convolutional feature maps to obtain a first addition result;

[0022] The activation layer is used to assign weights to the channels of the first addition result to obtain a weighted feature map;

[0023] The feature multiplication layer is used to multiply the weighted feature map and the cross-matching feature map to obtain the processed feature map.

[0024] In one possible implementation, the loss function of the image fusion model includes a thermal radiation feature loss function and an edge feature loss function, expressed by the following formula:

[0025] ,

[0026] Where γ is a weight used to control the loss of thermal radiation characteristics ( ) and edge feature loss ( The weights between )

[0027] The characteristic loss of thermal radiation is defined as follows:

[0028]

[0029] The fused image, visible light image, and infrared image are respectively used The image is represented by h and w, which represent the height and width of the image, respectively. Denotes the Frobenius norm of a matrix; Used to calculate the average pixel value of the matrix; Used to balance content loss between visible light and infrared images;

[0030] The edge feature loss is defined as follows:

[0031]

[0032] in, The operation can obtain gradient information from the matrix. Used to obtain maximum gradient information from visible light and infrared images.

[0033] In one possible implementation, obtaining the fully trained image fusion model includes:

[0034] Obtain the initial image fusion model;

[0035] Obtain an image training sample set containing annotation information, wherein the annotation information includes the actual fusion result of each sample image pixel in the image training sample set;

[0036] The value of the loss function of the initial image fusion model is determined based on the actual fusion result;

[0037] Adjust the parameters of the initial image fusion model according to the value of the loss function until the convergence condition is met, and determine the fully trained image fusion model.

[0038] Secondly, the present invention also provides an infrared-visible light image fusion device based on a 3D convolutional neural network, comprising:

[0039] The first acquisition module is used to acquire a fully trained image fusion model, which includes an encoder, a feature fusion module, and a decoder.

[0040] The second acquisition module is used to acquire the infrared image and the visible light image to be fused;

[0041] The image fusion module is used to extract high-dimensional semantic features from the infrared and visible light images to be fused based on the encoder, to obtain multiple visible light feature maps and multiple infrared feature maps; the feature fusion module performs cross-matching and stitching of the multiple visible light feature maps and multiple infrared feature maps, and weights the stitched features to obtain a processed feature map; the decoder performs weighted fusion of the processed feature map to obtain the image fusion result.

[0042] Thirdly, the present invention also provides an electronic device, comprising: a processor and a memory;

[0043] The memory stores a computer-readable program that can be executed by the processor;

[0044] When the processor executes the computer-readable program, it implements the steps in the infrared-visible light image fusion method based on 3D convolutional neural networks as described above.

[0045] Fourthly, the present invention also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the infrared-visible light image fusion method based on 3D convolutional neural networks as described above.

[0046] The beneficial effects of this invention are as follows: The provided image fusion model includes an encoder, a feature fusion module, and a decoder. The encoder extracts high-dimensional semantic features from the infrared and visible light images to be fused, obtaining multiple visible light feature maps and multiple infrared feature maps. Based on the feature fusion module, the multiple visible light feature maps and multiple infrared feature maps are cross-matched and stitched together, and the stitched features are weighted to obtain a processed feature map. This enhances the correlation between feature maps and enriches the information content of the feature maps, while avoiding the loss of effective feature information. It can adaptively retain significant thermal targets and rich texture structures. Finally, the decoder based on channel convolution performs weighted fusion on the processed feature maps to obtain the image fusion result, which can retain more effective information in the source image. Attached Figure Description

[0047] Figure 1 A flowchart illustrating an embodiment of the infrared and visible light image fusion method based on a 3D convolutional neural network provided by the present invention.

[0048] Figure 2 A schematic diagram of the structure of an embodiment of the image fusion model in the infrared and visible light image fusion method based on 3D convolutional neural network provided by the present invention;

[0049] Figure 3 A schematic diagram of the encoder structure in an embodiment of the infrared and visible light image fusion method based on 3D convolutional neural network provided by the present invention;

[0050] Figure 4 A schematic diagram of an embodiment of the cross-matching stitching operation in the infrared and visible light image fusion method based on 3D convolutional neural network provided by the present invention;

[0051] Figure 5 A schematic diagram of the structure of an embodiment of the channel attention mechanism in the infrared and visible light image fusion method based on 3D convolutional neural network provided by the present invention;

[0052] Figure 6 A schematic diagram of the decoder structure in an embodiment of the infrared and visible light image fusion method based on 3D convolutional neural networks provided by the present invention.

[0053] Figure 7 This is a schematic diagram of an embodiment of the infrared and visible light image fusion device based on a 3D convolutional neural network provided by the present invention;

[0054] Figure 8 This is a schematic diagram of the operating environment of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0055] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0056] A specific embodiment of the present invention discloses an infrared-visible light image fusion method based on a 3D convolutional neural network. Please refer to [link to relevant documentation]. Figure 1 ,include:

[0057] S101. Obtain a fully trained image fusion model, which includes an encoder, a feature fusion module, and a decoder;

[0058] S102. Acquire the infrared image and visible light image to be fused;

[0059] S103. Based on the encoder, high-dimensional semantic features are extracted from the infrared image and visible light image to be fused to obtain multiple visible light feature maps and multiple infrared feature maps; based on the feature fusion module, the multiple visible light feature maps and multiple infrared feature maps are cross-matched and stitched together, and the stitched features are weighted to obtain a processed feature map; based on the decoder, the processed feature map is weighted and fused to obtain the image fusion result.

[0060] In this embodiment, the provided image fusion model includes an encoder, a feature fusion module, and a decoder. The encoder extracts high-dimensional semantic features from the infrared and visible light images to be fused, obtaining multiple visible light feature maps and multiple infrared feature maps. The feature fusion module cross-matches and stitches the multiple visible light and infrared feature maps, and weights the stitched features to obtain a processed feature map. This enhances the correlation between feature maps and enriches their information content. The decoder then performs weighted fusion on the processed feature maps to obtain the image fusion result. Therefore, the infrared-visible light fused image enhanced by this invention can avoid the loss of effective feature information and adaptively retain significant thermal targets and rich texture structures.

[0061] It should be understood that the graph fusion model can be any of the following neural network models: R-CNN (Region-CNN), SPP-NET (Spatial Pyramid Pooling in Deep Convolutional Networks), Fast R-CNN, Faster R-CNN, R-FCN (Region-based fully convolutional network), YOLO, SSD (Single Shot MultiBoxDetector), EfficientDet, etc.

[0062] It should be noted that a well-trained image fusion model refers to an image fusion model whose mean average precision (mAP) of the fusion results on the image fusion test set is greater than or equal to 95%.

[0063] Using mAP, an object detection evaluation metric, can ensure the fusion effect of the image fusion model.

[0064] It should also be noted that when the mean average precision (mAP) of the detection results on the fused image test set is less than 95%, the image sample set to be fused is obtained, and the image fusion model is retrained using the image sample set to be fused.

[0065] It should be noted that, as Figure 2 As shown, the image fusion model includes an encoder, a fusion strategy, and a decoder to achieve feature extraction, feature fusion, and image reconstruction. The overall network framework is illustrated in the figure. Specifically, the encoder is used to acquire high-dimensional semantic feature information of the image. In the fusion strategy, a cross-matching concatenation operation is designed, which concatenates feature maps extracted from the same convolutional kernel in the source image. Subsequently, an optimized multi-scale channel attention module is used to weight the concatenated deep features. This fusion strategy enhances the correlation between feature maps and uses a multi-scale convolutional channel attention mechanism to strengthen the salient information of the feature maps, thereby further enriching the information content of the feature maps. Combining this fusion strategy, the decoder uses a channel convolutional neural network to fuse the processed feature maps to obtain the final fused image. Compared with the average fusion operation in traditional convolutional neural networks, the channel convolutional neural network can provide appropriate weights for the fusion of feature maps of different channel dimensions, thus retaining more effective information in the source image.

[0066] In some embodiments, please refer to Figure 3 The encoder includes a convolution module and multiple densely connected modules;

[0067] The convolution module is used to perform convolution operations on the input infrared image and visible light image to be fused, to obtain multiple infrared local features and multiple visible light local features;

[0068] The dense connection module includes a convolutional layer, a batch normalization layer, an activation function layer, a merging operation layer, and a feature skip connection layer. Each current dense connection module receives multiple infrared local features, multiple visible light local features, and the outputs of all previous dense connection modules as inputs based on the feature skip connection layer. The merging operation layer then merges all inputs and inputs them into the convolutional layer, the batch normalization layer, and the activation function layer to obtain multiple visible light feature maps and multiple infrared feature maps.

[0069] In this embodiment, the number of densely connected modules is not limited and can be adjusted according to factors such as the size of the fused image. In this embodiment, there are four densely connected modules. The dense modules are connected sequentially, and the input channels of each dense module gradually increase.

[0070] In one specific embodiment, the kernel size and stride of all modules in the encoder are set to 3×3 and 1, respectively. Batch normalization and ReLU activation functions are used in each module. The number "1×48" below the convolutional module indicates that the input channels and output channels are 1 and 48, respectively. The densely connected module uses a feature skip connection approach to obtain deep features, which helps to alleviate the gradient vanishing problem, enhances feature propagation, and emphasizes the reuse of features.

[0071] In some embodiments, the feature fusion module includes a cross-matching splicing operation layer and a channel attention mechanism;

[0072] The cross-matching splicing operation layer is used to cross-splice the visible light feature map and the infrared feature map obtained by the same convolutional kernel according to a preset rule to obtain a cross-matching feature map.

[0073] The channel attention mechanism is used to perform global average pooling on the cross-matching feature map, and then weight the pooled feature map to obtain the processed feature map.

[0074] In this embodiment, the cross-feature matching operation stacks feature maps obtained from the same convolutional kernel. The detailed process of the cross-matching concatenation operation is as follows: Figure 4 As shown in the figure, feature maps with the same label represent features obtained from the same convolutional kernel. Cross-feature matching enhances the correlation between the processed feature maps.

[0075] In one specific embodiment, the infrared features include feature 1, feature 2, and feature 3, and the visible light features include feature 1, feature 2, and feature 3. The cross-matching feature map is obtained by stitching the infrared feature 1, visible light feature 1, infrared feature 2, visible light feature 2, infrared feature 3, and visible light feature 3 in that order.

[0076] The channel attention mechanism integrates a novel channel attention mechanism to flexibly adjust the weights of feature channels to amplify or suppress different channels, thereby enabling the subsequent decoder to better fuse infrared and visible light features. The channel attention mechanism includes a global pooling layer, multiple convolutional layers, a feature addition layer, an activation layer, and a feature multiplication layer. The global pooling layer performs global average pooling on the cross-matching feature map to obtain a global average pooling result. The multiple convolutional layers extract multi-scale features from the global average pooling result to obtain multiple convolutional feature maps. The feature addition layer adds the multiple convolutional feature maps to obtain a first addition result. The activation layer assigns weights to the channels of the first addition result to obtain a weighted feature map. The feature multiplication layer multiplies the weighted feature map and the cross-matching feature map to obtain a processed feature map.

[0077] The channel attention mechanism designed in this embodiment of the invention is as follows: Figure 5 As shown, a multi-scale convolution is performed on the one-dimensional tensor obtained by global average pooling. Next, the outputs of the convolution operations are integrated. Then, channel weights are obtained through an activation module. Finally, the channel weights are multiplied by the feature map of the original image to obtain the final result. The structure of the channel attention mechanism is shown in the figure below.

[0078] Furthermore, the decoder involves a novel multi-scale 3D CNN module capable of extracting effective features with different strides. Using a smaller stride for feature fusion yields rich information but may introduce redundancy. Conversely, using a larger stride reduces redundancy but may also diminish feature detail. Therefore, this embodiment combines these two approaches to achieve better fusion results. The first module of the decoder is as follows: Figure 6 As shown. Subsequent modules apply convolutional operations to reduce the dimensionality of the fused feature maps and obtain the final fused image.

[0079] Furthermore, the loss functions designed in this embodiment of the invention include a thermal radiation feature loss function and an edge feature loss function, the mathematical definitions of which are as follows:

[0080] ,

[0081] Where γ is a weight used to control the loss of thermal radiation characteristics ( ) and edge feature loss ( The weights between these weights are as follows. The thermal radiation characteristic loss is defined as follows:

[0082]

[0083] The fused image, visible light image, and infrared image are respectively used The symbols h and w represent the height and width of the image, respectively. This represents the Frobenius norm of a matrix, used to compare the similarity between the true matrix and the estimated matrix. Used to calculate the average pixel value of a matrix. Used to balance content loss between visible light and infrared images.

[0084] The edge feature loss is defined as follows:

[0085]

[0086] in, The operation can extract gradient information from the matrix, thereby revealing the texture details of the image. Used to obtain maximum gradient information from visible light and infrared images.

[0087] Based on the above-described infrared-visible image fusion method based on 3D convolutional neural networks, this invention also provides an infrared-visible image fusion device based on 3D convolutional neural networks. Please refer to [link to relevant documentation]. Figure 7 ,include:

[0088] The first acquisition module 710 is used to acquire a fully trained image fusion model, which includes an encoder, a feature fusion module, and a decoder.

[0089] The second acquisition module 720 is used to acquire the infrared image and the visible light image to be fused;

[0090] The image fusion module 730 is used to extract high-dimensional semantic features from the infrared image and visible light image to be fused based on the encoder, to obtain multiple visible light feature maps and multiple infrared feature maps; based on the feature fusion module, the multiple visible light feature maps and multiple infrared feature maps are cross-matched and stitched together, and the stitched features are weighted to obtain a processed feature map; based on the decoder, the processed feature map is weighted and fused to obtain the image fusion result.

[0091] like Figure 8As shown, based on the above-described infrared and visible light image fusion method based on 3D convolutional neural networks, the present invention also provides an electronic device, which can be a mobile terminal, desktop computer, laptop, handheld computer, server, or other computing electronic device. The electronic device includes a processor 810, a memory 820, and a display 830. Figure 8 Only some components of the electronic device are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0092] In some embodiments, memory 820 may be an internal storage unit of the electronic device, such as a hard disk or memory. In other embodiments, memory 820 may be an external storage device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Furthermore, memory 820 may include both internal and external storage devices. Memory 820 is used to store application software and various types of data installed on the electronic device, such as program code installed on the electronic device. Memory 820 can also be used to temporarily store data that has been output or will be output. In one embodiment, memory 820 stores an infrared-visible light image fusion program 840 based on a 3D convolutional neural network. This infrared-visible light image fusion program 840 based on a 3D convolutional neural network can be executed by processor 810 to implement the infrared-visible light image fusion method based on a 3D convolutional neural network according to the embodiments of this application.

[0093] In some embodiments, processor 810 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 820 or process data, such as executing an infrared and visible light image fusion method based on a 3D convolutional neural network.

[0094] In some embodiments, display 830 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 830 is used to display information from the infrared-visible light image fusion electronic device based on a 3D convolutional neural network and to display a visual user interface. Components 810-830 of the electronic device communicate with each other via a system bus.

[0095] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0096] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for infrared and visible light image fusion based on a 3D convolutional neural network, characterized in that, include: A fully trained image fusion model is obtained. The image fusion model includes an encoder, a feature fusion module, and a decoder. The feature fusion module includes a cross-matching stitching operation layer and a channel attention mechanism. The cross-matching stitching operation layer is used to cross-stitch the visible light feature map and the infrared feature map obtained by the same convolutional kernel according to a preset rule to obtain a cross-matching feature map. The cross-stitching of the visible light feature map and the infrared feature map obtained by the same convolutional kernel according to the preset rule includes: arranging the channels of the visible light feature map and the infrared feature map in a certain order. The channel attention mechanism is used to perform a global average pooling operation on the cross-matching feature map and weight the feature map after the pooling operation to obtain a processed feature map. The channel attention mechanism includes a global pooling layer, multiple convolutional layers, a feature addition layer, an activation layer, and a feature multiplication layer. The global pooling layer performs global average pooling on the cross-matching feature map to obtain a global average pooling result. The multiple convolutional layers extract multi-scale features from the global average pooling result to obtain multiple convolutional feature maps. The feature addition layer adds the multiple convolutional feature maps to obtain a first addition result. The activation layer assigns weights to the channels of the first addition result to obtain a weighted feature map. The feature multiplication layer multiplies the weighted feature map and the cross-matching feature map to obtain a processed feature map. Acquire the infrared and visible light images to be fused; Based on the encoder, high-dimensional semantic features are extracted from the infrared and visible light images to be fused to obtain multiple visible light feature maps and multiple infrared feature maps; based on the feature fusion module, the multiple visible light feature maps and multiple infrared feature maps are cross-matched and stitched together, and the stitched features are weighted to obtain a processed feature map; based on the decoder, the processed feature map is weighted and fused to obtain the image fusion result.

2. The infrared-visible light image fusion method based on 3D convolutional neural network according to claim 1, characterized in that, The encoder includes a convolution module and multiple densely connected modules; The convolution module is used to perform convolution operations on the input infrared image and visible light image to be fused, to obtain multiple infrared local features and multiple visible light local features; The dense connection module includes a convolutional layer, a batch normalization layer, an activation function layer, a merging operation layer, and a feature skip connection layer. Each current dense connection module receives multiple infrared local features, multiple visible light local features, and the outputs of all previous dense connection modules as inputs based on the feature skip connection layer. Based on the merging operation layer, all inputs are merged and then input to the convolutional layer, the batch normalization layer, and the activation function layer to obtain multiple visible light feature maps and multiple infrared feature maps.

3. The infrared and visible light image fusion method based on 3D convolutional neural networks according to claim 1, characterized in that, The loss function of the image fusion model includes a thermal radiation feature loss function and an edge feature loss function, which are expressed by the following formula: , Where γ is a weight used to control the thermal radiation characteristic loss. and edge feature loss Weights between them; The characteristic loss of thermal radiation is defined as follows: The fused image, visible light image, and infrared image are respectively used The image is represented by h and w, which represent the height and width of the image, respectively. Denotes the Frobenius norm of a matrix; Used to calculate the average pixel value of the matrix; Used to balance content loss between visible light and infrared images; The edge feature loss is defined as follows: in, The operation can obtain gradient information from the matrix. Used to obtain maximum gradient information from visible light and infrared images.

4. The infrared-visible light image fusion method based on 3D convolutional neural network according to claim 1, characterized in that, The process of obtaining a fully trained image fusion model includes: Obtain the initial image fusion model; Obtain an image training sample set containing annotation information, wherein the annotation information includes the actual fusion result of each sample image pixel in the image training sample set; The value of the loss function of the initial image fusion model is determined based on the actual fusion result; Adjust the parameters of the initial image fusion model according to the value of the loss function until the convergence condition is met, and determine the fully trained image fusion model.

5. An infrared-visible light image fusion device based on a 3D convolutional neural network, characterized in that, include: The first acquisition module is used to acquire a fully trained image fusion model. The image fusion model includes an encoder, a feature fusion module, and a decoder. The feature fusion module includes a cross-matching concatenation layer and a channel attention mechanism. The cross-matching concatenation layer is used to cross-concatenate the visible light feature map and the infrared feature map obtained from the same convolutional kernel according to a preset rule, obtaining a cross-matching feature map. Cross-concatenating the visible light feature map and the infrared feature map according to the preset rule includes: arranging the channels of the visible light feature map and the infrared feature map in a certain order. The channel attention mechanism is used to perform a global average pooling operation on the cross-matching feature map and to perform pooling operations... The processed feature maps are weighted to obtain the processed feature maps. The channel attention mechanism includes a global pooling layer, multiple convolutional layers, a feature addition layer, an activation layer, and a feature multiplication layer. The global pooling layer performs global average pooling on the cross-matching feature maps to obtain a global average pooling result. The multiple convolutional layers extract multi-scale features from the global average pooling result to obtain multiple convolutional feature maps. The feature addition layer adds the multiple convolutional feature maps to obtain a first addition result. The activation layer assigns weights to the channels of the first addition result to obtain a weighted feature map. The feature multiplication layer multiplies the weighted feature map and the cross-matching feature map to obtain the processed feature map. The second acquisition module is used to acquire the infrared image and the visible light image to be fused; The image fusion module is used to extract high-dimensional semantic features from the infrared and visible light images to be fused based on the encoder, to obtain multiple visible light feature maps and multiple infrared feature maps; the feature fusion module performs cross-matching and stitching of the multiple visible light feature maps and multiple infrared feature maps, and weights the stitched features to obtain a processed feature map; the decoder performs weighted fusion of the processed feature map to obtain the image fusion result.

6. An electronic device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in the infrared-visible light image fusion method based on a 3D convolutional neural network as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps in the infrared-visible light image fusion method based on a 3D convolutional neural network as described in any one of claims 1-4.

Images