Method, system and electronic device for adaptive fusion of infrared and visible light images

Abstract

The present invention relates to the technical field of image fusion, and in particular to a method, system, and electronic device for adaptive fusion of infrared and visible light images. The method comprises: constructing an encoder network, a cross-scale feature interaction network, an infrared and visible light adaptive fusion network, and a multi-scale decoding network, and training them; inputting the infrared and visible light images to be fused into the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network, and the currently trained infrared and visible light adaptive fusion network to obtain multi-scale fusion features corresponding to the infrared and visible light images to be fused; inputting the multi-scale fusion features corresponding to the infrared and visible light images to be fused into the currently trained multi-scale decoding network to obtain a fused image corresponding to the infrared and visible light images to be fused, thereby greatly enhancing the degree of communication between features of different scales and enabling the model to adaptively adjust the fusion weights.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a method, system and electronic equipment for adaptively fusing infrared and visible light images. Background Art

[0002] Infrared and visible light image fusion technology can provide high-quality input data for advanced vision tasks such as image classification and target detection. While the salient target information in infrared images is very stable and less susceptible to external environmental influences, the amount of texture and scene information is relatively low. Visible light images contain rich texture details and are more consistent with human visual perception, but they are susceptible to external influences such as lighting and weather. Fusion technology allows the fused image to combine the advantages of both. This technology is widely used in target detection, target tracking, autonomous driving, and other fields.

[0003] Existing deep learning-based infrared and visible image fusion methods first extract their respective features, then fuse them together through a fusion module to reconstruct a fused image. Sufficient information exchange plays a key role in the feature extraction and image reconstruction process. Recent convolution-based infrared and visible image methods use residual networks or densely connected networks to achieve information exchange, but this unfiltered feature reuse leads to a large number of redundant features, and the intermediate features used for information exchange can cause excessive memory usage. Transformer-based infrared and visible image fusion methods can efficiently extract the required target features through an attention mechanism, but existing methods ignore the interaction between features at different scales, reducing the utilization of cross-scale information. In the fusion module, existing methods perform simple addition or channel-wise concatenation for fusion. Such simple fusion strategies cannot adaptively adjust weights based on the input data, resulting in unstable fusion results. Therefore, sufficient feature interaction and adaptive adjustment of fusion weights are urgent issues for those skilled in the art. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the existing technology and provide a method, system and electronic device for adaptive fusion of infrared and visible light images.

[0005] The technical solution of the adaptive fusion method of infrared and visible light images of the present invention is as follows:

[0006] Construct an encoder network, a cross-scale feature interaction network, an infrared and visible light adaptive fusion network, and a multi-scale decoding network, and train them;

[0007] After inputting the infrared and visible light images to be fused into the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network and the currently trained infrared and visible light adaptive fusion network, the multi-scale fusion features corresponding to the infrared and visible light images to be fused are obtained. The multi-scale fusion features corresponding to the infrared and visible light images to be fused are input into the currently trained multi-scale decoding network to obtain a fused image corresponding to the infrared and visible light images to be fused.

[0008] The technical solution of the adaptive fusion system of infrared and visible light images of the present invention is as follows:

[0009] Including building training modules and fusion modules;

[0010] The training module is used to construct an encoder network, a cross-scale feature interaction network, an infrared and visible light adaptive fusion network, and a multi-scale decoding network, and perform training;

[0011] The fusion module is used to: input the infrared and visible light images to be fused into the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network and the currently trained infrared and visible light adaptive fusion network to obtain multi-scale fusion features corresponding to the infrared and visible light images to be fused; input the multi-scale fusion features corresponding to the infrared and visible light images to be fused into the currently trained multi-scale decoding network to obtain a fused image corresponding to the infrared and visible light images to be fused.

[0012] A storage medium of the present invention stores instructions. When a computer reads the instructions, the computer executes the above-mentioned method for adaptively fusing infrared and visible light images.

[0013] An electronic device of the present invention includes a processor and the above-mentioned storage medium, wherein the processor executes instructions in the storage medium.

[0014] The beneficial effects of the present invention are as follows:

[0015] On the one hand, the cross-scale feature interaction method enables sufficient information exchange between low-level features such as shallow textures and deep semantic features, greatly strengthening the communication between features at different scales. On the other hand, the combined features of infrared and visible light features are used as a guide to achieve adaptive adjustment of the model's fusion weights. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0017] Figure 1Schematic diagram of a flow chart of a method for adaptive fusion of infrared and visible light images according to an embodiment of the present invention;

[0018] Figure 2 This is a model structure diagram corresponding to the adaptive fusion method of infrared and visible light images of the present invention;

[0019] Figure 3 is an infrared image;

[0020] Figure 4 is a visible light image;

[0021] Figure 5 It is a fused image reconstructed by directly adding the infrared and visible light feature maps;

[0022] Figure 6 A fused image reconstructed using an adaptive fusion method of infrared and visible light images of the present invention;

[0023] Figure 7 The figure is a schematic structural diagram of an infrared and visible light image adaptive fusion system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] like Figure 1 As shown, an adaptive fusion method of infrared and visible light images according to an embodiment of the present invention includes the following steps:

[0025] S1. Construct an encoder network, a cross-scale feature interaction network, an infrared and visible light adaptive fusion network, and a multi-scale decoding network, and train them.

[0026] S2. Input the infrared and visible light images to be fused into the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network and the currently trained infrared and visible light adaptive fusion network to obtain the multi-scale fusion features corresponding to the infrared and visible light images to be fused. Input the multi-scale fusion features corresponding to the infrared and visible light images to be fused into the currently trained multi-scale decoding network to obtain the fused image corresponding to the infrared and visible light images to be fused.

[0027] Optionally, in the above technical solution, in S1, the process of training the encoder network, the cross-scale feature interaction network, the infrared and visible light adaptive fusion network, and the multi-scale decoding network includes:

[0028] S10. Based on a dataset including a plurality of infrared and visible light source image samples, train the encoder network, the cross-scale feature interaction network, and the multi-scale decoding network under preset constraints, wherein the preset constraints are that the infrared and visible light source image samples input to the encoder network are the same as the output results of the multi-scale decoding network;

[0029] The encoder network is used to extract multiple scale features of each infrared and visible light source image sample;

[0030] The cross-scale feature interaction network is used to obtain cross-scale information interaction features corresponding to each scale feature, including infrared features and visible light features;

[0031] The multi-scale decoding network is used to reconstruct the multiple cross-scale information interaction features corresponding to each infrared and visible light source image sample into the infrared and visible light source image, i.e., the output result of the multi-scale decoding network;

[0032] The cross-scale feature interaction network is specifically used for:

[0033] Each scale feature is position-encoded to obtain the scale feature after each position encoding, and feature extraction is performed on each position-encoded scale feature to obtain the infrared feature and visible light feature corresponding to each scale feature. Information interaction is performed between every two scale features of different scales after position encoding to obtain multiple information interaction features. The infrared feature and visible light feature and the corresponding information interaction features corresponding to each scale feature are aggregated to obtain the cross-scale information interaction feature corresponding to each scale feature including the infrared feature and the visible light feature.

[0034] That is to say, the output of the encoder network is the input of the cross-scale feature interaction network, and the output of the cross-scale feature interaction network is the input of the multi-scale decoding network.

[0035] S11, freezing the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network, and the currently trained multi-scale decoding network, and training the infrared and visible light adaptive fusion network;

[0036] Among them, the infrared and visible light adaptive fusion network is used for:

[0037] The sum of the infrared features and visible light features in each cross-scale information interaction feature is used as the guiding feature. The adaptive weight corresponding to each cross-scale information interaction feature is extracted and calculated based on the global average pooling feature and the global maximum pooling feature of each guiding feature. According to the adaptive weight corresponding to each cross-scale information interaction feature, the weights of the infrared features and the visible light features in the infrared and visible light adaptive fusion network are adaptively recalibrated. The calibrated infrared features and visible light features are added together to obtain the multi-scale fusion feature.

[0038] Among them, one infrared and visible light source image sample corresponds to multiple scale features, one scale feature corresponds to a cross-scale information interaction feature, and one cross-scale information interaction feature corresponds to a multi-scale fusion feature.

[0039] The encoder network consists of four sequentially arranged combined convolution blocks, each of which includes two sequentially arranged convolution layers with 3×3 convolution kernels. The first convolution layer in each combined convolution block has a step size of 1 and the second convolution layer has a step size of 2. After the infrared and visible light source image samples are processed by the four sequentially arranged combined convolution blocks, the four scale features φ of the infrared and visible light source images are obtained. m , in, Represents the scale features of the infrared and visible light source images obtained after the first combined convolution block operation, express The scale feature obtained after the second combined convolution block operation, express The scale feature obtained after the third combined convolution block operation, express The scale features obtained after the fourth combined convolution block operation.

[0040] The cross-scale feature interaction network includes a convolution-based position encoding module and a channel information interaction module, as well as a transformer-based self-attention information interaction module and a spatial information interaction module. The self-attention feature module extracts feature information at each scale, while the channel information interaction module and the spatial information interaction module perform information interaction between features at each scale. The specific process is as follows:

[0041] S21, the position encoding module for each scale feature φ m Use depthwise separable convolution for position encoding, with a convolution kernel of 3×3 and a padding of 1, and add the features of each scale before position encoding to the features of the same scale after position encoding through the residual structure to obtain the position encoding ψ m , in, Represents the first scale feature The position encoding feature obtained after position encoding, Represents the second scale feature The position encoding feature obtained after position encoding, Represents the third scale feature The position encoding feature obtained after position encoding, Represents the fourth scale feature The position encoding feature obtained after position encoding.

[0042] S22, self-attention feature module encodes the position of S21 ψ mSerialize and downsample using convolution with a step size of 2. Serialize the features of the two scales, un-downsampled and downsampled, and obtain the un-downsampled sequence vector Q and the downsampled sequence vectors K and V respectively. Use the transformer architecture to extract attention features through MSA, and use the MLP interaction channel information to finally convert the obtained feature sequence into self-attention features. in, Represents the first position encoding feature in S21 The self-attention features obtained by the self-attention module, Represents the second position encoding feature in S21 The self-attention features obtained by the self-attention module, Represents the third position encoding feature in S21 The self-attention features obtained by the self-attention module, Represents the third position encoding feature in S21 Self-attention features obtained by the self-attention module;

[0043] S23, channel information interaction module encodes the position of S21 ψ m Use global average pooling to extract the spatial information of the shallow feature map, and use 1×1 convolution to increase the number of channels to the feature map. The number of channels, using the channel information to reweight and adjust the features Complete the information interaction between shallow features and deep features, and transform the features Use convolution to downsample to features The scale of the branch and the feature Add, where i and j represent the position code ψ m The index of each scale feature, and j>i, finally the channel information interaction feature is obtained in, Represents the first position encoding feature in S21 The channel information interaction features obtained by the channel information interaction module, Represents the second position encoding feature in S21 The channel information interaction features obtained by the channel information interaction module, Represents the third position encoding feature in S21 The channel information interaction features obtained by the channel information interaction module, Represents the third position encoding feature in S21 Channel information interaction features obtained by the channel information interaction module;

[0044] S24, spatial information interaction module encodes the position of S21 Shallow features in and deep features Serialization is performed, where i and j represent the position code ψ m The index of each scale feature, and j>i, respectively obtain the deep sequence vector Q and the shallow sequence vector K and V, use the same transformer architecture as step S22 to perform information interaction on the deep feature map and the shallow feature map in space, and convert the obtained feature sequence into a spatial information interaction feature in, Represents the first position encoding feature in S21 The spatial information interaction features obtained by the spatial information interaction module, Represents the second position encoding feature in S21 The spatial information interaction features obtained by the spatial information interaction module, Represents the third position encoding feature in S21 The spatial information interaction features obtained by the spatial information interaction module, Represents the fourth position encoding feature in S21 Spatial information interaction features obtained by the spatial information interaction module;

[0045] S25, combine the features ψ obtained by S21, S22, S23 and S24 m 、 and Merge on the channel, and then perform channel information interaction through a convolution operation with a convolution kernel of 1×1 to obtain the cross-scale information interaction feature α m , in, Representation characteristics and The first cross-scale information interaction feature after merging on the channel, Representation characteristics and The second cross-scale information interaction feature after merging on the channel, Representation characteristics and The third cross-scale information interaction feature after merging on the channel, Representation characteristics and The fourth cross-scale information interaction feature after merging on the channel.

[0046] The infrared and visible light adaptive fusion network consists of a global maximum pooling layer, a global average pooling layer, and two fully connected layers. It is used to adaptively fuse the cross-scale information interaction features of infrared and visible light of the same size. The specific process is as follows:

[0047] S31, infrared features of cross-scale information interaction obtained from S25 and visible light characteristics Add and merge as guiding features Where i represents the index of multiple scale features;

[0048] S32, extract S31 guidance features The global average pooling feature S ap And the global maximum pooling feature S mp , where the c-th layer feature is calculated as:

[0049]

[0050] S33. Add and merge the maximum and average pooling features of the infrared features and the visible light features, and calculate the channel-based adaptive weights of the infrared and visible light features:

[0051]

[0052] in, They represent the global average pooling feature and global maximum pooling feature of infrared features at layer c, respectively. c Represents the adaptive weight of infrared features in layer c, They represent the global average pooling feature and global maximum pooling feature of the visible light feature at the cth layer, P c represents the adaptive weight of the visible light feature at layer c, and p c +q c =1;

[0053] S34, using S33 adaptive weights to cross-scale information interaction features of S25 Perform weighted adjustment to obtain the infrared features after adaptive recalibration and visible light characteristics Specifically:

[0054]

[0055] Among them, the multi-scale decoder network consists of 4 combined convolution blocks, each of which includes two convolution operations. The first convolution kernel size is 3×3 and the step size is 1, which is used to reconstruct the spatial information of the feature. The second convolution kernel size is 1×1 and the step size is 1, which is used to communicate information between channels. After the densely connected combined convolution operation, the output is an image with the same size as the source input image. Depending on the training stage, the input is the S34 mid-infrared feature. Visible light characteristics Or the S34 mid-infrared feature Visible light characteristics The process of obtaining the fused image by pixel-by-pixel sum is as follows:

[0056] S51, the infrared interactive feature of S25 Interaction characteristics with visible light Image reconstruction is performed through a multi-scale decoding network. Under the first constraint, a structural similarity loss function is used. and pixel reconstruction loss function Constraints are imposed on image reconstruction, and the calculation formula is:

[0057]

[0058] Among them, Input m Output is the infrared and visible light source image input in the image reconstruction process. m The total loss function of the image reconstruction stage is the output infrared and visible light reconstructed images. ω1 and ω2 are Loss and The trade-off parameter between the losses, ‖·‖ F represents the Frobenius norm;

[0059] S52, freeze the encoder network, feature information interaction network and multi-scale decoding network in the reconstruction process of S51, and convert the infrared feature network after weight adaptive adjustment in S34 into Visible light characteristics Pixel-by-pixel addition and merging are performed, and image fusion is performed through an adaptive fusion network. Under the second constraint, the loss function of the image fusion process is divided into three parts: perceptual loss Constrain the fused image and intermediate features, structural similarity function Constrain the local features of the fused image, L1 loss function To supplement the visible light detail features in the fused image, the calculation formula is:

[0060]

[0061] Among them, Input m is the infrared and visible light source image input during the image fusion process, and Output is the fused image output during the image fusion process. Represents the infrared features of the i-th layer Visible light characteristics After fusion of features, the total loss function of the image fusion stage is ω3, ω4, and ω5 are the trade-off parameters between the three losses.

[0062] In another embodiment, there are four steps from S100 to S400. Figure 2This is the model structure diagram corresponding to the present invention, which is mainly divided into four parts: encoder (Encoder), i.e. encoder network, feature interaction module (FeatureInteration), i.e. cross-scale feature interaction network, feature fusion module (Feature Fusion), i.e. infrared and visible light adaptive fusion network, and multi-scale decoder (Decoder), i.e. multi-scale decoding network, where S represents step size, P represents padding, and C represents the number of channels. Figure 3 、 Figure 4 、 Figure 5 and Figure 6 It is the input image and the fusion effect image of the present invention.

[0063] S100, build the encoder network. Figure 2 The encoder part consists of four combined convolution modules. Each convolution block uses a 3×3 convolution kernel and is padded with 0. Each convolution block is composed of two convolution sequences, which are convolution layer, batch normalization layer and activation layer in order. The convolution layer in the first convolution sequence has a step size of 1, and the second convolution sequence has a step size of 2. The activation layer uses LeakyReLU. The number of input channels of the four combined convolution modules is 1, 16, 32, and 64, and the number of output channels varies to 16, 32, 64, and 128. The input Figure 3 Infrared images and Figure 4 The visible light image has a size of 224×224, and the scales of the four feature maps obtained are 112×112, 56×56, 28×28, and 14×14 respectively.

[0064] S200, perform cross-scale and cross-space information interaction on the features of the four scales outputted in step S100, such as Figure 2 As shown, it includes the following steps:

[0065] S201. Position encoding is performed on features of each scale before information exchange. A convolution operation with a convolution kernel of 3×3 and padding of 1 is used, and then the features before and after convolution are added using a residual structure.

[0066] S202. Use the transformer architecture in VIT to extract the features of the current feature layer. To reduce the amount of computation, the features encoded in S201 are downsampled through a convolution with a kernel of 3×3, a step size of 2, and a padding of 1. These are used as the input sequence vectors K and V in MSA (Multi-head self-attention). The sequence vector Q is not downsampled.

[0067] S203. In order to realize the information interaction between shallow features and deep features after position encoding in step S21, global average pooling is used to extract the spatial information of shallow features, and the channel dimension is increased by 1×1 convolution to make the number of channels consistent with the deep features. The deep features are re-weighted and adjusted with this feature. At the same time, the shallow features are downsampled to the scale of the deep features using a convolution operation with a step size greater than 1. The shallow features are added to the weighted adjusted features as branches to output the interactive features of the shallow features and the deep features, with scales of 112×112, 56×56, and 28×28.

[0068] S204: To realize the information interaction between the deep features and the shallow features after position encoding in step S201, the transformer architecture in VIT is used to realize the information interaction, wherein the deep features are used as the input sequence vector Q in MSA, and the shallow features are used as the input sequence vectors K and V. This structure is exactly the same as step S202, completing the interactive features between the deep features and the shallow features, and the output scales are 56×56, 28×28, and 14×14.

[0069] S25. The feature maps obtained in steps S100, S202, S203, and S204 are merged on the channel, and then a convolution operation with a convolution kernel of 1×1 is performed to realize the information transmission of each feature map on the channel. At the same time, the number of channels is restored to the same as that in step S1. The number of channels of the output feature maps are 16, 32, 64, and 128, respectively, and the corresponding feature scales are 112×112, 56×56, 28×28, and 14×14, respectively.

[0070] S3, such as Figure 5 As shown in , the fused image reconstructed after directly adding the infrared and visible light feature maps already has a visible fusion effect to the human eye. Therefore, the features of the infrared feature map and the visible light feature map of the same scale output in step S200 are used as a guide. Global average pooling and global maximum pooling are used respectively. The obtained global average pooling vector and global maximum pooling vector are added together. The channel-based adaptive weights of the infrared feature map and the visible light feature map are calculated by softmax. The infrared feature map and the visible light feature map output by S200 are weighted and adjusted with the weights. The multi-scale fused feature map is added together again, as shown in Figure 2 The fusion module in ,the output channels are 16, 32, 64, and 128, and the corresponding feature scales are 112×112, 56×56, 28×28, and 14×14, respectively.

[0071] S400: Constructing a multi-scale decoder network, including the following steps:

[0072] S401, such as Figure 2The Decoder part consists of four combined convolution modules. Each convolution block uses two convolution sequences and an upsampling layer. The convolution sequence is convolution layer, batch normalization layer and activation layer in order. The convolution kernel size of the convolution layer of the first convolution sequence is 3×3, the step size is 1, and the padding is 1. The convolution kernel size of the convolution layer of the second convolution sequence is 1×1, the step size is 1, and the padding is 0. The activation layer uses LeakyReLU, the sampling factor of the upsampling layer is 2, and the entire structure of the Decoder uses a dense connection method.

[0073] S402 , reconstructing the infrared features and visible light features outputted in step S200 through the decoder network in S401 , for training the encoder network, the cross-scale information interaction network, and the decoder network.

[0074] S403, freeze the encoder network, cross-scale information interaction network and decoder network in step S402, use the infrared features and visible light features output in step S200 as input to train the fusion network in step S300, and finally reconstruct the fusion image through the decoder network in step S401. Figure 6 , compared to simple additive fusion Figure 5 , Figure 6 It retains more significant contrast information of pedestrians while taking into account the texture details of the trees on the left. The contrast of the entire image is also improved, which is consistent with the visual perception of the human eye.

[0075] The beneficial effects of the present invention are as follows:

[0076] On the one hand, the cross-scale feature interaction method enables sufficient information exchange between low-level features such as shallow textures and deep semantic features, greatly strengthening the communication between features at different scales. On the other hand, the combined features of infrared and visible light features are used as a guide to achieve adaptive adjustment of the model's fusion weights.

[0077] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given in this application. Those skilled in the art can adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the scope of protection of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0078] like Figure 7 As shown, an infrared and visible light image adaptive fusion system 200 according to an embodiment of the present invention includes a training module 210 and a fusion module 220;

[0079] The training module 210 is used to construct an encoder network, a cross-scale feature interaction network, an infrared and visible light adaptive fusion network, and a multi-scale decoding network, and perform training;

[0080] The fusion module 220 is used to: input the infrared and visible light images to be fused into the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network and the currently trained infrared and visible light adaptive fusion network to obtain multi-scale fusion features corresponding to the infrared and visible light images to be fused; input the multi-scale fusion features corresponding to the infrared and visible light images to be fused into the currently trained multi-scale decoding network to obtain a fused image corresponding to the infrared and visible light images to be fused.

[0081] Optionally, in the above technical solution, the process of constructing the training module 210 to train the encoder network, the cross-scale feature interaction network, the infrared and visible light adaptive fusion network, and the multi-scale decoding network includes:

[0082] Based on a dataset of multiple infrared and visible light image samples, the encoder network, cross-scale feature interaction network, and multi-scale decoding network are trained under the preset constraints. The preset constraints are that the infrared and visible light image samples input to the encoder network are the same as the output of the multi-scale decoding network.

[0083] The encoder network is used to extract multiple scale features of each infrared and visible light source image sample;

[0084] The cross-scale feature interaction network is used to obtain cross-scale information interaction features corresponding to each scale feature, including infrared features and visible light features;

[0085] The multi-scale decoding network is used to reconstruct the multiple cross-scale information interaction features corresponding to each infrared and visible light source image sample into infrared and visible light source images;

[0086] Freeze the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network, and the currently trained multi-scale decoding network, and train the infrared and visible light adaptive fusion network;

[0087] Among them, the infrared and visible light adaptive fusion network is used for:

[0088] The sum of the infrared features and visible light features in each cross-scale information interaction feature is used as the guiding feature. The adaptive weight corresponding to each cross-scale information interaction feature is extracted and calculated based on the global average pooling feature and the global maximum pooling feature of each guiding feature. According to the adaptive weight corresponding to each cross-scale information interaction feature, the weights of the infrared features and the visible light features in the infrared and visible light adaptive fusion network are adaptively recalibrated. The calibrated infrared features and visible light features are added together to obtain the multi-scale fusion feature.

[0089] Optionally, in the above technical solution, the encoder network includes four sequentially arranged combined convolution blocks, each combined convolution block includes two sequentially arranged convolution layers with convolution kernels of 3×3, the first convolution layer in each combined convolution block has a step size of 1, and the second convolution layer has a step size of 2. After the infrared and visible light source image samples are operated by the four sequentially arranged combined convolution blocks, the four scale features φ of the infrared and visible light source images are obtained. m , in, Represents the scale features of the infrared and visible light source images obtained after the first combined convolution block operation, express The scale feature obtained after the second combined convolution block operation, express The scale feature obtained after the third combined convolution block operation, express The scale features obtained after the fourth combined convolution block operation.

[0090] Optionally, in the above technical solution, the cross-scale feature interaction network is specifically used to:

[0091] Each scale feature is position-encoded to obtain the scale feature after each position encoding, and feature extraction is performed on each position-encoded scale feature to obtain the infrared feature and visible light feature corresponding to each scale feature. Information interaction is performed between every two scale features of different scales after position encoding to obtain multiple information interaction features. The infrared feature and visible light feature and the corresponding information interaction features corresponding to each scale feature are aggregated to obtain the cross-scale information interaction feature corresponding to each scale feature including the infrared feature and the visible light feature.

[0092] The above-mentioned parameters and steps for each unit module to implement corresponding functions in the infrared and visible light image adaptive fusion system of the present invention can refer to the parameters and steps in the embodiment of the infrared and visible light image adaptive fusion method above, and will not be repeated here.

[0093] A storage medium according to an embodiment of the present invention stores instructions. When a computer reads the instructions, the computer executes any one of the above-mentioned methods for adaptively fusing infrared and visible light images.

[0094] An electronic device according to an embodiment of the present invention includes a processor and the above-mentioned storage medium, and the processor executes instructions in the storage medium.

[0095] Those skilled in the art will appreciate that the present invention may be implemented as a system, method or computer program product.

[0096] Therefore, the present disclosure may be embodied in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present disclosure may be embodied in the form of a computer program product embodied in one or more computer-readable media, wherein the computer-readable media contains computer-readable program code.

[0097] Any combination of one or more computer-readable media can be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0098] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for adaptive fusion of infrared and visible light images, characterized in that: include: Construct an encoder network, a cross-scale feature interaction network, an infrared and visible light adaptive fusion network, and a multi-scale decoding network, and train them; After inputting the infrared and visible light images to be fused into the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network, and the currently trained infrared and visible light adaptive fusion network, multi-scale fusion features corresponding to the infrared and visible light images to be fused are obtained, and the multi-scale fusion features corresponding to the infrared and visible light images to be fused are input into the currently trained multi-scale decoding network to obtain a fused image corresponding to the infrared and visible light images to be fused; The cross-scale feature interaction network is specifically used for: Position encoding is performed on each scale feature to obtain a scale feature after each position encoding, and feature extraction is performed on each scale feature after each position encoding to obtain an infrared feature and a visible light feature corresponding to each scale feature. Information interaction is performed between every two scale features of different scales after position encoding to obtain multiple information interaction features. The infrared feature and the visible light feature corresponding to each scale feature and the corresponding information interaction features are aggregated to obtain a cross-scale information interaction feature corresponding to each scale feature including the infrared feature and the visible light feature. The infrared and visible light adaptive fusion network is used for: The sum of the infrared feature and the visible light feature in each cross-scale information interaction feature is used as the guiding feature. The adaptive weight corresponding to each cross-scale information interaction feature is extracted and calculated based on the global average pooling feature and the global maximum pooling feature of each guiding feature. According to the adaptive weight corresponding to each cross-scale information interaction feature, the weights of the infrared feature and the visible light feature in the infrared and visible light adaptive fusion network are adaptively recalibrated. Each calibrated infrared feature and each visible light feature are added to obtain a multi-scale fusion feature.

2. The method for adaptive fusion of infrared and visible light images according to claim 1, characterized in that: The process of training the encoder network, cross-scale feature interaction network, infrared and visible light adaptive fusion network, and multi-scale decoding network includes: Based on a dataset including a plurality of infrared and visible light source image samples, the encoder network, the cross-scale feature interaction network, and the multi-scale decoding network are trained under a preset constraint condition, wherein the preset constraint condition is that the infrared and visible light source image samples input to the encoder network are the same as the output results of the multi-scale decoding network; The encoder network is used to extract multiple scale features of each infrared and visible light source image sample; The cross-scale feature interaction network is used to obtain cross-scale information interaction features corresponding to each scale feature, including infrared features and visible light features; The multi-scale decoding network is used to reconstruct the multiple cross-scale information interaction features corresponding to each infrared and visible light source image sample into infrared and visible light source images; The currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network, and the currently trained multi-scale decoding network are frozen, and the infrared and visible light adaptive fusion network is trained.

3. The method for adaptive fusion of infrared and visible light images according to claim 2, characterized in that: The encoder network includes four sequentially arranged combined convolution blocks, each of which includes two sequentially arranged convolution layers with convolution kernels of 3×3. The first convolution layer in each combined convolution block has a step size of 1 and the second convolution layer has a step size of 2. After the infrared and visible light source image samples are operated by the four sequentially arranged combined convolution blocks, the four scale features of the infrared and visible light source images are obtained. , ,in, represents the scale feature obtained after the infrared and visible light source images are operated by the first combined convolution block, express The scale feature obtained after the second combined convolution block operation, express The scale feature obtained after the third combined convolution block operation, express The scale features obtained after the fourth combined convolution block operation.

4. An adaptive fusion system for infrared and visible light images, characterized in that: Including building training modules and fusion modules; The training module is used to construct an encoder network, a cross-scale feature interaction network, an infrared and visible light adaptive fusion network, and a multi-scale decoding network, and perform training; The fusion module is used to: input the infrared and visible light images to be fused into the currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network, and the currently trained infrared and visible light adaptive fusion network, to obtain multi-scale fusion features corresponding to the infrared and visible light images to be fused, and input the multi-scale fusion features corresponding to the infrared and visible light images to be fused into the currently trained multi-scale decoding network to obtain a fused image corresponding to the infrared and visible light images to be fused; The cross-scale feature interaction network is specifically used for: Position encoding is performed on each scale feature to obtain a scale feature after each position encoding, and feature extraction is performed on each scale feature after each position encoding to obtain an infrared feature and a visible light feature corresponding to each scale feature. Information interaction is performed between every two scale features of different scales after position encoding to obtain multiple information interaction features. The infrared feature and the visible light feature corresponding to each scale feature and the corresponding information interaction features are aggregated to obtain a cross-scale information interaction feature corresponding to each scale feature including the infrared feature and the visible light feature. The infrared and visible light adaptive fusion network is used for: The sum of the infrared feature and the visible light feature in each cross-scale information interaction feature is used as the guiding feature. The adaptive weight corresponding to each cross-scale information interaction feature is extracted and calculated based on the global average pooling feature and the global maximum pooling feature of each guiding feature. According to the adaptive weight corresponding to each cross-scale information interaction feature, the weights of the infrared feature and the visible light feature in the infrared and visible light adaptive fusion network are adaptively recalibrated. Each calibrated infrared feature and each visible light feature are added to obtain a multi-scale fusion feature.

5. The infrared and visible light image adaptive fusion system according to claim 4, characterized in that: The encoder network includes four sequentially arranged combined convolution blocks, each of which includes two sequentially arranged convolution layers with convolution kernels of 3×3. The first convolution layer in each combined convolution block has a step size of 1 and the second convolution layer has a step size of 2. After the infrared and visible light source image samples are operated by the four sequentially arranged combined convolution blocks, the four scale features of the infrared and visible light source images are obtained. , ,in, represents the scale feature obtained after the infrared and visible light source images are operated by the first combined convolution block, express The scale feature obtained after the second combined convolution block operation, express The scale feature obtained after the third combined convolution block operation, express The scale features obtained after the fourth combined convolution block operation.

6. The infrared and visible light image adaptive fusion system according to claim 5, characterized in that: The process of constructing the training module to train the encoder network, the cross-scale feature interaction network, the infrared and visible light adaptive fusion network, and the multi-scale decoding network includes: Based on a dataset including a plurality of infrared and visible light source image samples, the encoder network, the cross-scale feature interaction network, and the multi-scale decoding network are trained under a preset constraint condition, wherein the preset constraint condition is that the infrared and visible light source image samples input to the encoder network are the same as the output results of the multi-scale decoding network; The encoder network is used to extract multiple scale features of each infrared and visible light source image sample; The cross-scale feature interaction network is used to obtain cross-scale information interaction features corresponding to each scale feature, including infrared features and visible light features; The multi-scale decoding network is used to reconstruct the multiple cross-scale information interaction features corresponding to each infrared and visible light source image sample into infrared and visible light source images; The currently trained multi-scale encoder network, the currently trained cross-scale feature interaction network, and the currently trained multi-scale decoding network are frozen, and the infrared and visible light adaptive fusion network is trained.

7. A storage medium, characterized in that: The storage medium stores instructions, and when a computer reads the instructions, the computer is enabled to execute the method for adaptive fusion of infrared and visible light images according to any one of claims 1 to 3.

8. An electronic device, characterized in that: The device comprises a processor and the storage medium according to claim 7, wherein the processor executes instructions in the storage medium.

Images