Generation method and device of fused image, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a fused image generation method and device, a storage medium and electronic equipment, and the method comprises the steps: inputting a source image into a content perception module, enabling the content perception module to carry out the enhancement of the source image, and outputting the enhanced feature information of an enhanced image, the source image comprises an infrared light image and a visible light image, and the enhanced image comprises an enhanced infrared light image and an enhanced visible light image; the enhanced feature information is input to a multi-stage contrast learning auto-encoder, so that the multi-stage contrast learning auto-encoder compares and reconstructs the enhanced image, and the reconstructed feature information of a reconstructed image is output, and the reconstructed image comprises a reconstructed infrared light image and a reconstructed visible light image; and inputting the reconstructed feature information into a fusion module, so that the fusion module generates a fused image according to the reconstructed feature information.

Description

Technical Field

[0001] The present application relates to the field of image processing, and more specifically, to a method and device for generating a fused image, a storage medium, and an electronic device. Background Art

[0002] With the continuous advancement of science and technology, image processing technology has been widely used in various fields, especially in the fields of military, medicine, industry and security monitoring. As an important part of it, infrared and visible light image fusion technology aims to effectively fuse image information of different bands to obtain richer and more accurate image information. Infrared images can effectively identify targets under various complex lighting conditions, especially in low light or even no light conditions, while visible light images can provide rich color and detail information. However, traditional image fusion methods mainly rely on mathematical transformations and manual analysis such as multi-scale transformation, sparse representation and saliency detection. These methods usually cannot fully consider the mutual information interaction between source images, resulting in incomplete feature extraction or information redundancy.

[0003] In recent years, with the rapid development of deep learning technology, image fusion methods based on deep learning have gradually emerged. These methods use the powerful feature extraction capabilities of deep neural networks to learn the feature representation of source images and achieve more efficient image fusion effects. Specifically, deep learning methods extract and fuse features of source images by constructing models such as convolutional neural networks, autoencoders, or generative adversarial networks. These methods can not only extract high-level semantic features of images, but also adaptively adjust the fusion strategy during the fusion process, thereby improving the quality of the fused image.

[0004] Convolutional neural network (CNN) is a typical deep learning model, which is widely used in the field of image processing. In image fusion, CNN extracts and fuses the features of the source image by constructing multiple layers of convolutional layers and pooling layers. Specifically, CNN first performs convolution operation on the source image to extract the low-level features of the image, then gradually extracts the high-level semantic features of the image through multiple layers of convolutional layers, and finally fuses them in the high-level feature space to obtain a fused image. The application of CNN in image fusion mainly includes fusion based on feature maps and fusion based on image generation. The fusion method based on feature maps extracts the feature maps of the source image and fuses them in the feature map space, thereby retaining the main feature information of the source image. The fusion method based on image generation constructs a generative adversarial network to generate a fused image during the adversarial training process, thereby improving the quality and naturalness of the fused image.

[0005] Auto-encoder is an unsupervised learning model that extracts and reconstructs image features by constructing encoders and decoders. In image fusion, the auto-encoder encodes the source image to extract the low-dimensional feature representation of the image, then fuses it in the low-dimensional feature space, and finally reconstructs the fused image through the decoder. The application of auto-encoders in image fusion mainly includes sparse auto-encoders, variational auto-encoders, and stacked auto-encoders. Sparse auto-encoders enhance the robustness of feature extraction by introducing sparse regularization terms. Variational auto-encoders can generate more realistic fused images by introducing variational inference. Stacked auto-encoders gradually extract high-level features of images by constructing multiple layers of auto-encoders, thereby improving the quality of fused images.

[0006] Generative Adversarial Network (GAN) is a deep generative model that generates realistic images in an adversarial training process by constructing a generator and a discriminator. In image fusion, GAN extracts and fuses features of source images by constructing a generator and a discriminator. Specifically, the generator generates a fused image by learning the feature representation of the source image; the discriminator guides the generator to optimize the generation strategy by judging the authenticity of the generated image, thereby improving the quality of the fused image. The application of GAN in image fusion mainly includes conditional GAN, cycle GAN and multimodal GAN. Conditional GAN ​​realizes image fusion under specific conditions by introducing conditional information; cycle GAN improves the stability and consistency of the fused image by introducing cycle consistency constraints; multimodal GAN ​​realizes the fusion of multiple modal images by constructing a multimodal generator.

[0007] Although deep learning-based image fusion methods have achieved significant performance improvements, these methods still have some challenges and shortcomings. First, deep learning methods require a large amount of labeled data during training, while in practical applications, it is often difficult and costly to obtain labeled data.

[0008] In the existing technology, the image fusion method based on deep learning has high computational complexity when processing images and is difficult to meet the requirements of real-time processing. No effective solution has been proposed yet.

[0009] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0010] The embodiments of the present application provide a method and device for generating a fused image, a storage medium, and an electronic device, so as to at least solve the problem in the prior art that the image fusion method based on deep learning has high computational complexity when processing images and is difficult to meet the requirements of real-time processing.

[0011] According to one embodiment of the present application, a method for generating a fused image is provided, comprising: inputting a source image into a content perception module so that the content perception module enhances the source image and outputs enhanced feature information of the enhanced image, wherein the source image comprises: an infrared light image and a visible light image, and the enhanced image comprises: an enhanced infrared light image and an enhanced visible light image; inputting the enhanced feature information into a multi-level contrastive learning autoencoder so that the multi-level contrastive learning autoencoder compares and reconstructs the enhanced image and outputs reconstructed feature information of the reconstructed image, wherein the reconstructed image comprises: a reconstructed infrared light image and a visible light image; inputting the reconstructed feature information into a fusion module so that the fusion module generates a fused image according to the reconstructed feature information.

[0012] In an exemplary embodiment, a source image is input into a content-aware module so that the content-aware module enhances the source image and outputs enhanced feature information of the enhanced image, including: determining the first feature information of the infrared light image and the first feature information of the visible light image through a convolution block in the content-aware module; determining the second feature information of the infrared light image and the second feature information of the visible light image through batch normalization and a nonlinear activation function; determining the enhanced feature information of the infrared light image according to the source feature information of the infrared light image, the first feature information of the infrared light image and the second feature information of the infrared light image, and determining the enhanced feature information of the visible light image according to the source feature information of the visible light image, the first feature information of the visible light image and the second feature information of the visible light image.

[0013] In an exemplary embodiment, the enhanced feature information is input into a multi-level contrastive learning autoencoder so that the multi-level contrastive learning autoencoder compares and reconstructs the enhanced image and outputs the reconstructed feature information of the reconstructed image, including: performing a convolution operation on the enhanced feature information through the multi-level contrastive learning autoencoder to generate third feature information; inputting the third feature information into the residual block of the multi-level contrastive learning autoencoder so that the residual block generates first reconstructed feature information based on the third feature information; and comparing and optimizing the first reconstructed feature information according to the feature information of the negative sample and the enhanced image feature to determine the reconstructed feature information of the reconstructed image.

[0014] In an exemplary embodiment, before inputting the enhanced feature information into a multi-level contrastive learning autoencoder, the method further includes: determining a first loss function of the multi-level contrastive learning autoencoder, wherein the first loss function includes: a perceptual loss function and a contrast loss function; optimizing parameters in the multi-level contrastive learning autoencoder according to the first loss function until the first loss function converges.

[0015] In an exemplary embodiment, the reconstructed feature information is input into a fusion module so that the fusion module generates a fused image based on the reconstructed feature information, including: decoding the reconstructed feature information of the infrared light image and the reconstructed feature information of the visible light image to generate preliminary fused feature information; iteratively fusing the preliminary fused feature information to determine fused feature information; and performing a convolution operation on the fused feature information to generate the fused image.

[0016] In an exemplary embodiment, the preliminary fused feature information is iteratively fused to determine the fused feature information, including: performing a fusion step: when A=1, decoding the preliminary fused feature information to determine the Ath fused feature information; adjusting the Ath fused feature information based on a target fusion mechanism to determine the A+1th fused feature information; when A≠1, decoding A-1th fused feature information to determine the Ath fused feature information; adjusting the Ath fused feature information based on a target fusion mechanism to determine the A+1th fused feature information; looping through the fusion step until the number of iterations is reached, and using the fused feature information generated in the last iteration as the fused feature information.

[0017] According to another embodiment of the present application, a device for generating a fused image is provided, including: a content perception module, used to receive a source image, enhance the source image, and output enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image; a multi-level contrast learning autoencoder, used to receive the enhanced feature information, contrast and reconstruct the enhanced image, and output reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image; a fusion module, used to receive the reconstructed feature information, and generate a fused image according to the reconstructed feature information.

[0018] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0019] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0020] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0021] Through this application, the source image is input into the content perception module so that the content perception module enhances the source image and outputs the enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image; the enhanced feature information is input into the multi-level contrast learning autoencoder so that the multi-level contrast learning autoencoder compares and reconstructs the enhanced image and outputs the reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image; the reconstructed feature information is input into the fusion module so that the fusion module generates a fused image according to the reconstructed feature information. That is, in the embodiment of the present application, the effective fusion of the infrared light image and the visible light image is achieved through the collaborative work of the content perception module, the multi-level contrast learning autoencoder and the fusion module. The content perception module can enhance the details and texture of the image, and the multi-level contrast learning autoencoder further optimizes the image features and improves the quality of the fused image through contrast learning and reconstruction operations. The fusion module adopts an iterative fusion strategy to generate clearer and more detailed fused images without the need for deep learning. Therefore, it can solve the problem that the image fusion method based on deep learning has high computational complexity when processing images and is difficult to meet the requirements of real-time processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0023] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0024] Figure 1 It is a hardware structure block diagram of a server device of a method for generating a fused image according to an embodiment of the present application;

[0025] Figure 2 is a flow chart of a method for generating a fused image according to an embodiment of the present application;

[0026] Figure 3 is a schematic diagram of a method for generating a fused image according to an embodiment of the present application;

[0027] Figure 4 It is a structural block diagram of a device for generating a fused image according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0030] The method embodiments provided in the embodiments of the present application can be executed in a server device or a similar computing device. Taking running on a server device as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a server device of a method for generating a fused image according to an embodiment of the present application. Figure 1 As shown, the server device may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the server device may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above server device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for generating a fused image in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the server device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0032] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the server device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0033] In this embodiment, a method for generating a fused image is provided. Figure 2 is a flow chart of a method for generating a fused image according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps:

[0034] Step S202, inputting the source image into a content perception module, so that the content perception module enhances the source image and outputs enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image;

[0035] Step S204, inputting the enhanced feature information into a multi-level contrast learning autoencoder, so that the multi-level contrast learning autoencoder compares and reconstructs the enhanced image, and outputs reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image;

[0036] Step S206: input the reconstructed feature information into a fusion module, so that the fusion module generates a fused image according to the reconstructed feature information.

[0037] Through the above steps, the source image is input into the content perception module, so that the content perception module enhances the source image and outputs the enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image; the enhanced feature information is input into the multi-level contrast learning autoencoder, so that the multi-level contrast learning autoencoder compares and reconstructs the enhanced image, and outputs the reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image; the reconstructed feature information is input into the fusion module, so that the fusion module generates a fused image according to the reconstructed feature information. That is, in the embodiment of the present application, the effective fusion of the infrared light image and the visible light image is achieved through the collaborative work of the content perception module, the multi-level contrast learning autoencoder and the fusion module. The content perception module can enhance the details and texture of the image, and the multi-level contrast learning autoencoder further optimizes the image features and improves the quality of the fused image through contrast learning and reconstruction operations. The fusion module adopts an iterative fusion strategy to generate clearer and more detailed fused images without the need for deep learning. Therefore, it can solve the problem that the image fusion method based on deep learning has high computational complexity when processing images and is difficult to meet the requirements of real-time processing.

[0038] The execution subject of the above steps may be a terminal, etc., but is not limited thereto.

[0039] In an exemplary embodiment, a source image is input into a content-aware module so that the content-aware module enhances the source image and outputs enhanced feature information of the enhanced image, including: determining the first feature information of the infrared light image and the first feature information of the visible light image through a convolution block in the content-aware module; determining the second feature information of the infrared light image and the second feature information of the visible light image through batch normalization and a nonlinear activation function; determining the enhanced feature information of the infrared light image according to the source feature information of the infrared light image, the first feature information of the infrared light image and the second feature information of the infrared light image, and determining the enhanced feature information of the visible light image according to the source feature information of the visible light image, the first feature information of the visible light image and the second feature information of the visible light image.

[0040] In the embodiment of the present application, a specific operation flow of a content-aware enhancement module (CEM) when processing infrared and visible light images is provided, that is, through multi-level feature extraction and enhancement, higher quality feature information is provided for subsequent image fusion. Specifically:

[0041] 1) Convolution block feature extraction: First, the source image (infrared image and visible light image) is input into the convolution block in CEM. The convolution block is a powerful tool for extracting local features of images in deep learning. In the embodiment of the present application, the role of the convolution block is to determine the first feature information of the source image, which is usually the low-level features of the image, such as edges, textures, and local structures. Through the convolution operation, CEM can capture the basic components of the image and lay the foundation for subsequent processing.

[0042] 2) Batch Normalization and Non-Linear Activation: After the initial convolution feature extraction, CEM further determines the second feature information of the source image through batch normalization and non-linear activation functions (such as ReLU or Leaky ReLU). The role of batch normalization is to standardize the output of each layer of the network and reduce internal covariate shift, thereby accelerating the training process and improving the stability of the model. The non-linear activation function introduces non-linear transformations, enabling the network to learn more complex feature representations, which is crucial for extracting high-level semantic features of the image.

[0043] 3) Feature information integration: CEM integrates the source feature information, the first feature information, and the second feature information of the source image to determine the enhanced feature information. The source feature information refers to the features extracted directly from the source image, while the first and second feature information are obtained through operations such as convolution blocks and batch normalization. The integration process is achieved through addition operations, and the Sigmoid function is used for final processing. The role of the Sigmoid function here is to map the feature information to the range of 0 to 1, which helps with feature fusion and image enhancement, ensuring that the enhanced image not only retains the main features of the original image, but also improves the brightness and detail clarity of the image.

[0044] 4) Enhanced image output: The enhanced infrared and visible light images will be output, and their feature information is richer and clearer, providing high-quality input for the subsequent multi-level contrastive learning autoencoder (MCLA) and fusion network (FN).

[0045] The CEM in the embodiment of the present application provides more abundant and accurate feature information for infrared and visible light image fusion through multi-step feature extraction and enhancement, which is a key step in achieving high-quality image fusion. Through such processing, information redundancy can be effectively reduced, image contrast and details can be enhanced, and the foundation for the efficient operation of the subsequent fusion network can be laid.

[0046] In an exemplary embodiment, the enhanced feature information is input into a multi-level contrastive learning autoencoder so that the multi-level contrastive learning autoencoder compares and reconstructs the enhanced image and outputs the reconstructed feature information of the reconstructed image, including: performing a convolution operation on the enhanced feature information through the multi-level contrastive learning autoencoder to generate third feature information; inputting the third feature information into the residual block of the multi-level contrastive learning autoencoder so that the residual block generates first reconstructed feature information based on the third feature information; and comparing and optimizing the first reconstructed feature information according to the feature information of the negative sample and the enhanced image feature to determine the reconstructed feature information of the reconstructed image.

[0047] In the embodiments of the present application, a workflow of a multi-level contrastive learning autoencoder (MCLA) is provided for processing enhanced feature information, and how to optimize and reconstruct feature information through a contrastive learning mechanism. Specifically:

[0048] First, the enhanced feature information extracted from the content-aware enhancement module (CEM) is input into the MCLA, where the enhanced feature information refers to the feature representation after adaptively enhancing the brightness information and texture details of the original infrared and visible light images.

[0049] After receiving the enhanced feature information, MCLA first generates the third feature information through convolution operation. The third feature information is the intermediate representation after the preliminary feature extraction, which is prepared for further reconstruction and contrastive learning.

[0050] The third feature information is fed into the residual block of MCLA. Among them, the residual block is a structure used in deep learning to solve the gradient vanishing problem and improve the learning efficiency of the network. The residual block directly passes the input feature information to the output of the block by adding a jump connection, and superimposes it with the transformed feature information inside the block. In this way, the residual block can generate the first reconstructed feature information. It should be noted that this step is performed in the feature space, and the purpose is to reconstruct the features of the enhanced image.

[0051] Next, MCLA optimizes the first reconstructed feature information using a contrastive loss function. The contrastive loss function determines the reconstructed feature information by comparing the feature information of the reconstructed image with the feature information of the original enhanced image, as well as with the feature information of the negative sample. Negative samples are generated by an adaptive negative sample synthesizer (ANS) to increase the difficulty and depth of contrastive learning, ensuring that the model can distinguish between positive and negative samples, thereby learning more discriminative feature representations.

[0052] Through the above comparison and optimization process, MCLA determines the reconstructed feature information of the reconstructed image. This is the result obtained through optimization iteration, which preserves and restores the features of the enhanced image as much as possible, while reducing redundant information and improving the fusion quality and naturalness of the image.

[0053] In the embodiment of the present application, MCLA optimizes the reconstruction feature information through a contrastive learning mechanism, thereby ensuring that the model can effectively learn the feature representation of the source image in an unsupervised learning environment, and verify and enhance the fusion ability of the model through the reconstruction process. The feature extraction and fusion capabilities of the model are enhanced, without relying on a large amount of labeled data, reducing training costs, and improving the quality of the fused image.

[0054] In an exemplary embodiment, before inputting the enhanced feature information into the multi-level contrastive learning autoencoder, it also includes: determining a first loss function of the multi-level contrastive learning autoencoder, wherein the first loss function includes: a perceptual loss function and a contrast loss function; optimizing the parameters in the multi-level contrastive learning autoencoder according to the first loss function until the first loss function converges.

[0055] It should be noted that the perceptual loss function can make the features learned by the model better match human visual perception. Usually, the perceptual loss calculates the difference between the source image and the reconstructed image in the intermediate layer feature map of the pre-trained deep neural network (such as the VGG network), rather than directly comparing the pixel-level differences. By minimizing the perceptual loss, the model can learn the high-level features of the source image and generate a fused image that is more in line with visual perception.

[0056] The contrastive loss function can distinguish positive samples from negative samples in the feature space, making the features of positive samples more similar while the features of negative samples remain different. In the context of image fusion, positive samples refer to enhanced image features, while negative samples are generated by an adaptive negative sample synthesizer. By minimizing the contrastive loss, the model can learn the mutual information interaction between source images, thereby more effectively separating and fusing image information of different modalities during feature extraction.

[0057] The optimization goal of the loss function is to enable the model to generate high-quality reconstructed images while preserving the features of the source image. The perceptual loss ensures that the reconstructed image is visually similar to the source image, while the contrast loss further improves the model's feature differentiation ability, making the features of the reconstructed image more distinct and accurate. The optimization process continues until the first loss function converges, that is, the value of the loss function no longer changes significantly, and the model has stably learned the feature representation of the source image.

[0058] Convergence is usually determined based on the changing trends of training loss and validation loss. If both are stable or fluctuate only within a small range, the model can be considered to have converged. In addition, a threshold can be set. When the change in the loss function is less than this threshold, the training can be terminated and the model can be considered to have converged.

[0059] In the embodiment of the present application, by defining and optimizing the first loss function, it is ensured that the multi-level contrastive learning autoencoder can effectively extract and fuse the features of infrared and visible light images while maintaining the main visual features of the source image, and the fused image finally generated is more visually and informationally rich and accurate. Through self-supervised learning, the dependence on a large amount of labeled data is avoided, the training cost is reduced, and the generalization ability of the model is improved.

[0060] In an exemplary embodiment, the reconstructed feature information is input into a fusion module so that the fusion module generates a fused image based on the reconstructed feature information, including: decoding the reconstructed feature information of the infrared light image and the reconstructed feature information of the visible light image to generate preliminary fused feature information; iteratively fusing the preliminary fused feature information to determine fused feature information; and performing a convolution operation on the fused feature information to generate the fused image.

[0061] The embodiment of the present application provides a specific step of how a fusion network (FN) processes the reconstructed feature information extracted by a multi-level contrastive learning autoencoder (MCLA) to generate a fused image. Specifically:

[0062] The reconstructed feature information of the infrared image and the visible light image after MCLA processing is input into the decoder of FN. The function of the decoder is to reversely transform these compressed feature information to restore their spatial structure and details and form preliminary fused feature information.

[0063] The initial fused feature information is further processed through the iterative fusion mechanism in FN, with the goal of deeply integrating the features of infrared and visible light images to overcome the problem of insufficient information fusion that may exist in single decoding. The final fused feature information is subjected to the convolution operation of FN to generate the final fused image.

[0064] The embodiment of the present application generates a high-quality fusion result by making full use of the information of the infrared image and the visible light image.

[0065] In an exemplary embodiment, the preliminary fused feature information is iteratively fused to determine the fused feature information, including: performing a fusion step: when A=1, decoding the preliminary fused feature information to determine the Ath fused feature information; adjusting the Ath fused feature information based on a target fusion mechanism to determine the A+1th fused feature information; when A≠1, decoding A-1th fused feature information to determine the Ath fused feature information; adjusting the Ath fused feature information based on a target fusion mechanism to determine the A+1th fused feature information; looping through the fusion step until the number of iterations is reached, and using the fused feature information generated in the last iteration as the fused feature information.

[0066] The embodiment of the present application provides a process of iteratively fusing preliminary fused feature information in a fusion network (FN) to gradually refine the feature information and finally determine the fused feature information. The details are as follows:

[0067] In the initial case of A=1, FN decodes these preliminary fused feature information, that is, through the reverse convolution operation, it preliminarily reconstructs the feature space and generates the first fused feature information (A=1).

[0068] In each iteration, FN adjusts the fused feature information generated in the previous round based on the target fusion mechanism to determine the fused feature information (A+1) for the next iteration. This adjustment can be weighted summation of features through learned weights, or integration of features through a specific network structure (such as Transformer).

[0069] In the case of A≠1, that is, starting from the second round of iteration, FN will decode the fused feature information generated in the previous round (A-1) to reconstruct the feature space again and generate the fused feature information of the Ath round. In each round of iteration, FN will perform decoding and adjustment based on the target fusion mechanism, gradually refining and optimizing the fused feature information until the preset number of iterations k is reached.

[0070] In the embodiment of the present application, features are gradually optimized from coarse to fine, so that detailed information in the source image can be captured more accurately and the quality of the fused image can be improved.

[0071] In order to better understand the process of the above-mentioned fused image generation method, the implementation method flow of the above-mentioned fused image generation is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0072] In this embodiment, a self-supervised multi-level contrast autoencoder framework (SS-MCAE) is provided to achieve the fusion of infrared and visible light images. The framework achieves efficient fusion of infrared and visible light images by designing a coarse-to-fine feature extraction network (CFEN) and a fusion network (FN). CFEN contains a content-aware enhancement module (CEM) and a multi-level contrastive learning autoencoder (MCLA). Figure 3 is a schematic diagram of a method for generating a fused image according to an embodiment of the present application, such as Figure 3 As shown, the details are as follows:

[0073] 1. Content-aware Enhancement Module (CEM): CEM adaptively enhances the brightness information and texture details of the source image through a convolutional neural network. According to the Retinex theory, any image can be represented as the product of a reflected image and an incident image. CEM retains most of the reflection information in the source image through image-level and feature-level residual connections to ensure enhanced performance while reducing computational complexity. The formula is as follows: eni_ir,vi =I ir,vi +φ(I ir,vi );

[0074] Among them, φ(·) represents the reflection information extracted by the convolution block, I ir is the infrared light source image, I vi is the visible light source image, I eni_ir is the enhanced infrared light source image, I eni_vi is the enhanced visible light source image. After the source image undergoes a preliminary convolution operation, the first layer of features is obtained. It is expressed as: φ1=Conv(I ir,vi );

[0075] Then, through batch normalization and nonlinear activation function, the second layer features are obtained Representation: φ2 = LeakyReLU(BatchNorm(Conv(φ1)));

[0076] Finally, the second layer features are added to the first layer features and processed by the Sigmoid function to obtain the enhanced image: eni_ir,vi =I ir,vi +Sigmoid(φ1+φ2).

[0077] 2. Multi-level contrastive learning autoencoder (MCLA): MCLA consists of four convolutional blocks and four residual blocks, and adopts a two-layer residual structure to ensure the effectiveness of global feature extraction. In the contrastive autoencoding process, MCLA reconstructs the enhanced image into a reconstructed image and extracts paired features from it. First, the enhanced image undergoes a preliminary convolution operation to obtain the preliminary feature F0 representation: F0 = Conv(I eni_ir,vi ) Then, after multiple residual blocks, features are extracted layer by layer. The calculation process of each residual block is as follows:

[0078] F i+1 =F i +Conv(BatchNorm(LeakyReLU(Conv(F i ))));

[0079] After the last layer of convolution, the final feature representation is obtained:

[0080] F final =Conv(F last ).

[0081] MCLA introduces a contrastive learning mechanism in the feature extraction process of each layer and uses a contrastive loss function to optimize the feature extraction process. Specifically, challenging samples are constructed through a sample generator to improve the effect of contrastive learning. Negative samples are generated by an adaptive negative sample synthesizer (ANS) with the following formula:

[0082]

[0083] Among them, G i (X i ) represents the optimal output of layer (i) obtained from ANS, ⊙ represents the connection operation, g i is a randomly generated noise vector.

[0084] 3. Sample Generator (SG): SG consists of MCLA, Projection Head (PH) and Adaptive Negative Sample Synthesizer (ANS). PH is a two-layer MLP network used to embed sampled features into a high-dimensional representation space. ANS adaptively generates the optimal output by considering the characteristics of the input features. The features of the enhanced image and the reconstructed image are projected layer by layer into the high-dimensional space for contrastive learning. In each contrastive learning, the features of the reconstructed image are used as queries, the features of the enhanced image are used as positive samples, and the negative samples are generated by ANS. In order to ensure the comprehensiveness of feature extraction and the high quality of the fused image, innovative perceptual loss functions (Perceptual Loss) and contrastive loss functions (Contrastive Loss) are designed to maintain the original features of the source image and guide image reconstruction. The perceptual loss function formula is as follows:

[0085]

[0086] The source image is represented as I ir ,I vi , the enhanced image is represented as

[0087] The contrast loss function formula is as follows:

[0088]

[0089] in, To query features, is the positive sample feature, is the negative sample feature, λ is the weight parameter, and m represents the number of blocks in the image sampled at each layer.

[0090] 4. Fusion Network (FN): The fusion network obtains a fused image through a convolution block and a Transformer layer. The last layer of features extracted by MCLA is used as the input of FN. Through iterative decoding and reconstruction of features, a fused image containing multiple layers of features is finally generated. FN first decodes the last layer of features and preliminarily fuses the features. The formula is as follows:

[0091]

[0092] Then, through layer-by-layer iteration, the features are further fused, and the formula is as follows:

[0093]

[0094] Among them, i is the layer index and k is the number of iterations. After multiple iterations of fusion, the feature information is integrated through the Transformer layer. The formula is as follows:

[0095] δ F = Trans(δ final F);

[0096] Finally, the final fused image is generated through the convolution layer, and the formula is as follows:

[0097] I f =Tanh(Conv(δ F )).

[0098] When training FN, structural similarity loss (SSIM) and gradient difference loss are used to retain the edge information of the source image. The SSIM loss function formula is as follows:

[0099]

[0100] The gradient difference loss formula is as follows:

[0101]

[0102] The final total loss function is:

[0103] Among them, β is the weight parameter.

[0104] The embodiment of the present application can achieve efficient feature extraction and fusion by introducing a self-supervised learning mechanism without a large amount of labeled data. The effect and efficiency of infrared and visible light image fusion are improved through innovative network structure design and loss function optimization.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0106] In this embodiment, a device for generating a fused image is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0107] Figure 4 is a structural block diagram of a device for generating a fused image according to an embodiment of the present application, such as Figure 4 As shown, the device comprises:

[0108] The content perception module 42 is used to receive a source image, enhance the source image, and output enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image;

[0109] A multi-level contrast learning autoencoder 44 is used to receive the enhanced feature information, contrast and reconstruct the enhanced image, and output the reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image;

[0110] The fusion module 46 is used to receive the reconstructed feature information and generate a fused image according to the reconstructed feature information.

[0111] Through the above-mentioned device, the source image is input into the content perception module so that the content perception module enhances the source image and outputs the enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image; the enhanced feature information is input into the multi-level contrast learning autoencoder so that the multi-level contrast learning autoencoder compares and reconstructs the enhanced image and outputs the reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image; the reconstructed feature information is input into the fusion module so that the fusion module generates a fused image according to the reconstructed feature information. That is, in the embodiment of the present application, the effective fusion of the infrared light image and the visible light image is achieved through the collaborative work of the content perception module, the multi-level contrast learning autoencoder and the fusion module. The content perception module can enhance the details and texture of the image, and the multi-level contrast learning autoencoder further optimizes the image features and improves the quality of the fused image through contrast learning and reconstruction operations. The fusion module adopts an iterative fusion strategy to generate clearer and more detailed fused images without the need for deep learning. Therefore, it can solve the problem that the image fusion method based on deep learning has high computational complexity when processing images and is difficult to meet the requirements of real-time processing.

[0112] In an exemplary embodiment, a content perception module is used to determine the first feature information of the infrared light image and the first feature information of the visible light image through a convolution block in the content perception module; determine the second feature information of the infrared light image and the second feature information of the visible light image through batch normalization and a nonlinear activation function; determine enhanced feature information of the infrared light image based on source feature information of the infrared light image, the first feature information of the infrared light image and the second feature information of the infrared light image, and determine enhanced feature information of the visible light image based on the source feature information of the visible light image, the first feature information of the visible light image and the second feature information of the visible light image.

[0113] In an exemplary embodiment, a multi-level contrastive learning autoencoder is used to perform a convolution operation on the enhanced feature information through the multi-level contrastive learning autoencoder to generate third feature information; input the third feature information into the residual block of the multi-level contrastive learning autoencoder so that the residual block generates first reconstructed feature information based on the third feature information; and compare and optimize the first reconstructed feature information according to the feature information of the negative sample and the enhanced image feature to determine the reconstructed feature information of the reconstructed image.

[0114] In an exemplary embodiment, a multi-level contrastive learning autoencoder is used to determine a first loss function of the multi-level contrastive learning autoencoder, wherein the first loss function includes: a perceptual loss function and a contrast loss function; and the parameters in the multi-level contrastive learning autoencoder are optimized according to the first loss function until the first loss function converges.

[0115] In an exemplary embodiment, a fusion module is used to decode the reconstructed feature information of the infrared light image and the reconstructed feature information of the visible light image to generate preliminary fused feature information; iteratively fuse the preliminary fused feature information to determine fused feature information; and perform a convolution operation on the fused feature information to generate the fused image.

[0116] In an exemplary embodiment, the fusion module is used to perform the fusion step: when A=1, the preliminary fusion feature information is decoded to determine the Ath fusion feature information; the Ath fusion feature information is adjusted based on the target fusion mechanism to determine the A+1th fusion feature information; when A≠1, the A-1 fusion feature information is decoded to determine the Ath fusion feature information; the Ath fusion feature information is adjusted based on the target fusion mechanism to determine the A+1th fusion feature information; the fusion step is executed repeatedly until the number of iterations is reached, and the fusion feature information generated by the last iteration is used as the fusion feature information. It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0117] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0118] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0119] S1, inputting a source image into a content perception module, so that the content perception module enhances the source image and outputs enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image;

[0120] S2, inputting the enhanced feature information into a multi-level contrast learning autoencoder, so that the multi-level contrast learning autoencoder compares and reconstructs the enhanced image, and outputs reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image;

[0121] S3, inputting the reconstructed feature information into a fusion module, so that the fusion module generates a fused image according to the reconstructed feature information.

[0122] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0123] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0124] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0125] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0126] S1, inputting a source image into a content perception module, so that the content perception module enhances the source image and outputs enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image;

[0127] S2, inputting the enhanced feature information into a multi-level contrast learning autoencoder, so that the multi-level contrast learning autoencoder compares and reconstructs the enhanced image, and outputs reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image;

[0128] S3, inputting the reconstructed feature information into a fusion module, so that the fusion module generates a fused image according to the reconstructed feature information.

[0129] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0130] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0131] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0132] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0133] S1, inputting a source image into a content perception module, so that the content perception module enhances the source image and outputs enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image;

[0134] S2, inputting the enhanced feature information into a multi-level contrast learning autoencoder, so that the multi-level contrast learning autoencoder compares and reconstructs the enhanced image, and outputs reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image;

[0135] S3, inputting the reconstructed feature information into a fusion module, so that the fusion module generates a fused image according to the reconstructed feature information.

[0136] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0137] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0138] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for generating a fused image, characterized in that: include: Inputting a source image into a content perception module so that the content perception module enhances the source image and outputs enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image; Inputting the enhanced feature information into a multi-level contrast learning autoencoder so that the multi-level contrast learning autoencoder compares and reconstructs the enhanced image and outputs reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image; The reconstructed feature information is input into a fusion module, so that the fusion module generates a fused image according to the reconstructed feature information.

2. The method according to claim 1, characterized in that Inputting the source image to the content perception module so that the content perception module enhances the source image and outputs enhanced feature information of the enhanced image, including: Determining first feature information of the infrared light image and first feature information of the visible light image through a convolution block in the content perception module; Determining the second feature information of the infrared light image and determining the second feature information of the visible light image through batch normalization and a nonlinear activation function; The enhanced characteristic information of the infrared light image is determined according to the source characteristic information of the infrared light image, the first characteristic information of the infrared light image and the second characteristic information of the infrared light image, and the enhanced characteristic information of the visible light image is determined according to the source characteristic information of the visible light image, the first characteristic information of the visible light image and the second characteristic information of the visible light image.

3. The method according to claim 1, characterized in that: Inputting the enhanced feature information into a multi-stage contrastive learning autoencoder so that the multi-stage contrastive learning autoencoder compares and reconstructs the enhanced image and outputs the reconstructed feature information of the reconstructed image, including: Performing a convolution operation on the enhanced feature information through the multi-level contrastive learning autoencoder to generate third feature information; Inputting the third feature information into a residual block of the multi-level contrastive learning autoencoder so that the residual block generates first reconstructed feature information based on the third feature information; The first reconstructed feature information is compared and optimized according to the feature information of the negative sample and the enhanced image feature to determine the reconstructed feature information of the reconstructed image.

4. The method according to claim 1, characterized in that Before inputting the enhanced feature information into the multi-stage contrastive learning autoencoder, the method further comprises: Determine a first loss function of the multi-level contrastive learning autoencoder, wherein the first loss function includes: a perceptual loss function and a contrastive loss function; Parameters in the multi-level contrastive learning autoencoder are optimized according to the first loss function until the first loss function converges.

5. The method according to claim 1, characterized in that: Inputting the reconstructed feature information into a fusion module so that the fusion module generates a fused image according to the reconstructed feature information, comprising: Decoding the reconstructed feature information of the infrared light image and the reconstructed feature information of the visible light image to generate preliminary fusion feature information; Iteratively fusing the preliminary fused feature information to determine fused feature information; A convolution operation is performed on the fused feature information to generate the fused image.

6. The method according to claim 5, characterized in that Iteratively fusing the preliminary fused feature information to determine fused feature information, including: Performing a fusion step: when A=1, decoding the preliminary fusion feature information to determine A-th fusion feature information; adjusting the A-th fusion feature information based on a target fusion mechanism to determine A+1-th fusion feature information; In the case where A≠1, A-1 fused feature information is decoded to determine A-th fused feature information; the A-th fused feature information is adjusted based on the target fusion mechanism to determine A+1-th fused feature information; The fusion step is executed in a loop until the number of iterations is reached, and the fusion feature information generated by the last iteration is used as the fusion feature information.

7. A device for generating a fused image, characterized in that: include: A content perception module, configured to receive a source image, enhance the source image, and output enhanced feature information of the enhanced image, wherein the source image includes: an infrared light image and a visible light image, and the enhanced image includes: an enhanced infrared light image and an enhanced visible light image; A multi-level contrast learning autoencoder, used for receiving the enhanced feature information, contrasting and reconstructing the enhanced image, and outputting reconstructed feature information of the reconstructed image, wherein the reconstructed image includes: a reconstructed infrared light image and a visible light image; The fusion module is used to receive the reconstructed feature information and generate a fused image according to the reconstructed feature information.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method of any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images